Data acquisition method and system for multiple data sources

Abstract

The invention relates to the technical field of data acquisition, discloses a multi-data-source-oriented data acquisition method and system, and aims to solve the problems of complexity and challenging of data acquisition in a multi-data-source environment. The method comprises the following three main steps: 1, constructing a data source feature extraction model and a data quality evaluation model which are used for extracting data features from a plurality of heterogeneous data sources and comprehensively evaluating data quality; 2, a data acquisition decision model is constructed, a data acquisition process is modeled as a Markov decision process, and intelligence and adaptivity of data acquisition decision are realized; and thirdly, real-time data acquisition is carried out based on the trained model, and a decision result is automatically executed, so that the accuracy and efficiency of data acquisition are improved. According to the method, through a systematic feature extraction and data quality evaluation mechanism and an intelligent data acquisition decision process, data isomerism, dynamics and uncertainty in a multi-data-source environment are effectively dealt with, and high-quality data support is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of data collection, and in particular to a data collection method and system for multiple data sources. Background Art

[0002] In today's information age, data has become a key factor in corporate decision-making, business optimization and innovative development. With the rapid development of big data technology, the sources of data are becoming increasingly diverse, including but not limited to sensor networks, social media, corporate databases, log files, etc. These multi-source data contain rich information, but also bring complexity and challenges to data collection.

[0003] Existing data collection technologies usually lack systematic feature extraction and data quality assessment mechanisms. Feature extraction is the basis for understanding the nature, content and potential value of data sources, while data quality assessment is the key to ensuring the accuracy, completeness and consistency of collected data. The lack of these mechanisms may result in the collected data failing to meet the needs of subsequent analysis or application, or even introduce errors or deviations, affecting the accuracy of decision-making. The data collection decision-making process often relies on manual experience or simple rule settings, lacking intelligence and adaptability. In a changing data environment, this static decision-making method is difficult to respond to changes in data quality in real time, and cannot dynamically adjust the collection strategy to optimize resource allocation and collection efficiency. Summary of the invention

[0004] The purpose of the present invention is to provide a data collection method and system for multiple data sources to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a data collection method for multiple data sources, the method comprising: Step A: constructing a data source feature extraction model and a data quality assessment model, wherein the data source feature extraction model is used to extract data features from multiple data sources, and the data quality assessment model performs data quality assessment based on the output of the data source feature extraction model; Step B: Construct a data collection decision model and model the data collection process as a Markov decision process for making data collection decisions based on the output of the data quality assessment model; Step C: Perform real-time data collection based on the trained data source feature extraction model, data quality assessment model and data collection decision model, and automatically execute the decision results.

[0006] Preferably, the data source feature extraction model is constructed by using an algorithm combining a self-attention mechanism with a graph convolutional network (GCN), and its network structure at least includes: Input layer: used to receive training data of the model; Hidden layer: contains at least one self-attention mechanism layer for extracting correlation features between data, followed by at least one GCN hidden layer. Each GCN hidden layer contains several GCN units to capture the structural information of the data source. Output layer: Receives the output of the hidden layer and maps the network output to the extracted feature vector through a fully connected layer.

[0007] Preferably, the step of training the data source feature extraction model includes: Initialize the weights and biases of the self-attention mechanism and GCN network using the Xavier initialization method; The cross entropy loss function is used as the loss function to measure the difference between the model extracted features and the real features; The back propagation algorithm and RMSprop optimization algorithm are used to calculate the gradient according to the loss function, and the weights and bias items of the self-attention mechanism and the GCN network are updated. The network is iteratively trained until the loss function converges. Save the trained model as the data source feature extraction model.

[0008] Preferably, the data quality assessment model is constructed using a gradient boosted decision tree (GBDT) algorithm, and its training steps include: Collect training data, including historical data source characteristics, historical data collection efficiency, historical data quality indicators, and corresponding actual data usage effects; The historical data source characteristics, historical data collection efficiency and historical data quality indicators in the training data are used as input features, and the corresponding data usage effects are used as labels to construct the training sample set of the GBDT algorithm; The GBDT algorithm is used to train the training sample set, and the evaluation accuracy is improved by constructing multiple decision trees and integrating their output results; During the training process, the number of trees, learning rate, and maximum depth of the GBDT algorithm are optimized by grid search.

[0009] Preferably, the data acquisition decision model is constructed using a deep deterministic policy gradient DDPG algorithm.

[0010] Preferably, the specific definition of the data acquisition decision model includes: State space: defined as the combination of data quality indicators of multiple data sources, data source availability status, and data acquisition system status; Action space: defined as the set of executable data source selection, collection frequency adjustment, and data preprocessing operations; Reward function: Calculate rewards or penalties based on the data collection efficiency and data quality evaluation results. Give positive rewards to operations that improve collection efficiency and data quality, and give penalties to operations that reduce collection efficiency and data quality. Output strategy: It is defined as the method of selecting the best action in the current state, and the decision actions generated are continuously optimized through deep reinforcement learning algorithms.

[0011] Preferably, the data quality indicators include data integrity, accuracy, consistency and timeliness; the data source availability status includes the online status, response time and data throughput of the data source; and the data acquisition system status includes system load, storage capacity and processing power.

[0012] Preferably, the executable data source selection, collection frequency adjustment and data preprocessing operation set includes: selecting a data source with high data quality, adjusting the frequency of data collection, performing data cleaning and formatting operations, and data compression and encryption processing.

[0013] Preferably, the implementation method of the reward function includes: Obtain real-time data collection efficiency indicators; compare the real-time data collection efficiency indicators with the preset collection efficiency targets, and calculate the collection efficiency deviation; Obtain real-time data quality assessment results; compare the current data quality indicators output by the data quality assessment model with the preset quality standards or historical quality data, and calculate the data quality deviation; Rewards or penalties are calculated based on the collection efficiency deviation and data quality deviation. The specific formula is: R = γ * (E_target - E_current) - δ * (Q_target - Q_current) Among them, R represents the reward value, E_target and E_current represent the target acquisition efficiency and the current acquisition efficiency respectively, Q_target and Q_current represent the target data quality and the current data quality respectively, γ and δ are the weight coefficients of the acquisition efficiency deviation and the data quality deviation respectively, and γ, δ>0.

[0014] Preferably, a data collection system for multiple data sources comprises: A data source feature extraction module is used to extract data features from multiple data sources; A data quality assessment module, connected to the data source feature extraction module, performs data quality assessment based on the output of the data source feature extraction module; A data collection decision module, connected to the data quality assessment module, models the data collection process as a Markov decision process, and is used to make data collection decisions based on the output of the data quality assessment module; The execution module is connected to the data acquisition decision module and is used to perform real-time data acquisition based on the trained data source feature extraction model, data quality assessment model and data acquisition decision model, and automatically execute the decision results.

[0015] Compared with the prior art, the present invention has the following beneficial effects: By constructing a data source feature extraction model, the present invention can automatically extract data features from multiple heterogeneous data sources and fully understand the nature, content and potential value of the data source. The introduction of the data quality assessment model ensures the accuracy, completeness and consistency of the collected data, and effectively avoids decision-making biases caused by data quality issues. The combined use of these two models greatly improves the accuracy and efficiency of data collection, and provides a high-quality data foundation for subsequent data analysis, mining and application.

[0016] The present invention models the data collection process as a Markov decision process and constructs a data collection decision model. This model can adjust the collection strategy in real time and optimize resource allocation according to the output of the data quality assessment model. This intelligent decision-making method enables the data collection process to dynamically respond to changes in the data environment and automatically adjust the collection strategy, thereby improving the flexibility and adaptability of data collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a working principle diagram of the data collection method for multiple data sources according to the present invention; Figure 2 A schematic diagram of the construction of a data quality assessment model; Figure 3 Schematic diagram of the reward function design. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] See also Figure 1-3 The present invention provides a technical solution: a data collection method for multiple data sources, the method comprising: Step A: Build a data source feature extraction model and a data quality assessment model 1. Construction of data source feature extraction model: Determine the types of data features that need to be extracted from multiple data sources. These features may include the format, structure, content, update frequency, source reliability, etc. of the data. Select or design appropriate feature extraction algorithms, such as statistical analysis, machine learning algorithms, text mining techniques, etc., to automatically extract these features from various data sources. Implement a feature extraction model to process the input multi-source data and output a data feature vector containing the extracted features.

[0020] 2. Construction of data quality assessment model: Design data quality assessment indicators such as accuracy, completeness, consistency, timeliness, etc. based on the output of the data source feature extraction model. Select or develop data quality assessment algorithms that can quantitatively score or classify data quality based on data feature vectors and evaluation indicators. Implement a data quality assessment model that receives data feature vectors as input and outputs data quality assessment results.

[0021] Step B: Build a data collection decision model 1. Modeling of Markov decision process: The data collection process is considered as a Markov decision process, where the state represents the current data collection state (such as the collected data source, data quality, etc.), the action represents the possible collection decision (such as continue collection, stop collection, adjust collection strategy, etc.), and the reward represents the benefit or cost brought by the collection decision. Define the state transition probability and reward function to describe the change of state and the effect of decision during the data collection process.

[0022] 3. Construction of data collection decision model: Based on the Markov decision process, select or design a suitable decision algorithm, such as reinforcement learning algorithm, dynamic programming algorithm, etc. Implement a data collection decision model that can select the optimal collection decision based on the current data collection status and data quality assessment results.

[0023] Step C: Real-time data collection and decision execution 1. Model training and verification: Use historical data or simulated data to train data source feature extraction models, data quality assessment models, and data acquisition decision models.

[0024] The model is verified and optimized through cross-validation, accuracy evaluation and other methods to ensure the performance and stability of the model.

[0025] 2. Real-time data collection and decision execution: The trained model is deployed to the data acquisition system, and the system receives multi-source data input in real time.

[0026] The data source feature extraction model extracts features from the input data and outputs a data feature vector.

[0027] The data quality assessment model performs data quality assessment based on the data feature vector and outputs the assessment results.

[0028] The data collection decision model makes collection decisions based on the data quality assessment results and the current collection status.

[0029] The system automatically executes collection decisions, such as continuing collection, stopping collection, or adjusting collection strategies to achieve efficient and accurate data collection.

[0030] The present invention will be further described below in conjunction with Examples 1 to 4: Example

[0031] The data source feature extraction model is constructed using an algorithm that combines the self-attention mechanism with the graph convolutional network (GCN). The following describes in detail the specific implementation of the model and its training process.

[0032] 1. Network structure design Input layer: An input layer designed to receive training data for the model. The training data includes data samples from multiple data sources, each of which can be represented as a feature vector or a data node in a graph structure.

[0033] Hidden layer: contains at least one self-attention mechanism layer. The self-attention mechanism layer is used to extract the correlation features between data. By calculating the attention weights between data samples, it highlights important features and suppresses irrelevant features. The self-attention mechanism layer is followed by at least one GCN hidden layer. Each GCN hidden layer contains several GCN units, which capture the structural information of the data source through the product of the adjacency matrix and the feature matrix to realize the convolution operation of graph structure data.

[0034] Output layer: Receives the output of the hidden layer and maps the network output to the extracted feature vector through a fully connected layer. The feature vector is used to represent the key features of the data source for subsequent data quality assessment and data collection decision-making.

[0035] 2. Model training process Initialize network parameters: Use the Xavier initialization method to initialize the weights and biases of the self-attention mechanism and the GCN network. The Xavier initialization method helps keep the variance of the input and output consistent and prevents the network from experiencing gradient vanishing or exploding problems in the early stages of training.

[0036] Define the loss function: Use the cross entropy loss function as the loss function. The cross entropy loss function is used to measure the difference between the features extracted by the model and the real features, and optimize the parameters of the model by minimizing the loss function.

[0037] Optimization algorithm selection: The back propagation algorithm and RMSprop optimization algorithm are used for model training. The back propagation algorithm is used to calculate the gradient of the loss function with respect to the network parameters, and the RMSprop optimization algorithm updates the network parameters according to the gradient to achieve iterative optimization of the model.

[0038] Iterative training network: Input the training data into the model and calculate the output feature vector of the model through forward propagation. Calculate the cross entropy loss between the output feature vector and the true feature vector. Calculate the gradient according to the loss function, and use the RMSprop optimization algorithm to update the weights and bias terms of the self-attention mechanism and the GCN network. Repeat the above process and iteratively train the network until the loss function converges or the preset training rounds are reached.

[0039] Save the trained model: After training is completed, save the trained model as a data source feature extraction model. This model can be used in real-time data collection to extract data features from multiple data sources, providing support for subsequent data quality assessment and data collection decisions.

[0040] Taking social media data as an example, assuming that it is necessary to collect content data posted by users from multiple social media platforms and extract key features therein for subsequent analysis, the data source feature extraction model of the present invention can be used for feature extraction.

[0041] Input layer: Receives user content data from multiple social media platforms, such as text, pictures, videos, etc.

[0042] Hidden layer: First, the self-attention mechanism layer is used to extract the correlation features between user content, such as similar topics, hot topics, etc. Then, the GCN hidden layer is used to capture the social relationship structure information between users, such as attention relationship, interaction frequency, etc.

[0043] Output layer: Outputs the extracted feature vector, including key features of user content (such as topics, sentiment tendencies) and social relationship features (such as influence, activity). Example

[0044] The data quality assessment model is constructed using the gradient boosted decision tree (GBDT) algorithm. The steps for constructing and training the data quality assessment model include: Step 1. Collect training data: Collect training data that includes historical data source characteristics, historical data collection efficiency, historical data quality indicators, and corresponding actual data usage effects.

[0045] Historical data source characteristics may include data format, structure, content type, source reliability, etc.

[0046] The efficiency of historical data collection can be expressed as the amount of data collected per unit time or the time consumed in the collection process.

[0047] Historical data quality indicators may include data accuracy, completeness, consistency, timeliness, etc.

[0048] The actual data usage effect can be the business results or decision-making effects achieved through data analysis, mining or application, such as increased sales, reduced costs, improved prediction accuracy, etc.

[0049] Step 2. Construct a training sample set: The historical data source characteristics, historical data collection efficiency and historical data quality indicators in the collected training data are used as input features.

[0050] The corresponding data usage effect is used as a label to indicate the quality of the data or the quality of the data usage effect.

[0051] Construct a training sample set for the GBDT algorithm, where each sample contains an input feature vector and a corresponding label.

[0052] Step 3. Train the GBDT model: The GBDT algorithm is used to train the training sample set.

[0053] The GBDT algorithm improves evaluation accuracy by constructing multiple decision trees and integrating their output results. Each decision tree is trained based on a subset of the training data, and the error of the previous tree is corrected by continuously iteratively training the new tree.

[0054] During the training process, the GBDT algorithm gradually optimizes the parameters of the model to minimize the prediction error or loss function.

[0055] Step 4. Optimize GBDT algorithm parameters: The parameters such as the number of trees, learning rate and maximum depth of the GBDT algorithm are optimized by grid search method.

[0056] Grid search is an exhaustive search method that tries all possible parameter combinations within a given parameter range and selects the parameter combination that optimizes the model performance.

[0057] The number of trees determines the total number of decision trees in the GBDT model. Too many trees may lead to overfitting, while too few trees may not fully capture the information in the data.

[0058] The learning rate controls the weight of each tree's contribution to the model. A learning rate that is too large may cause the model to quickly converge to a local optimal solution in the early stages of training, while a learning rate that is too small may make the model training process too slow.

[0059] The maximum depth limits the maximum depth of a decision tree. A tree that is too deep may lead to an overly complex model and overfitting, while a tree that is too shallow may not fully capture the complex relationships in the data.

[0060] Step 5. Model training completion and application: After training and optimization, a data quality assessment model is obtained. This model can be used in real-time data collection to assess the quality of the collected data.

[0061] By inputting data characteristics (such as data source characteristics, collection efficiency, quality indicators, etc.), the model can output data quality scores or classification results to provide a basis for subsequent data collection decisions.

[0062] Taking the user behavior data of an e-commerce platform as an example, it is assumed that it is necessary to collect user browsing, clicking, purchasing and other behavior data from multiple data sources, and evaluate the quality of these data for subsequent user portrait construction and precision marketing. The data quality evaluation model of the present invention can be used to perform data quality evaluation.

[0063] Collect training data: Collect historical user behavior data, including data source characteristics (such as data source, data format, etc.), collection efficiency (such as collection time, amount of collected data, etc.), data quality indicators (such as accuracy, completeness, etc.) and corresponding data usage effects (such as accuracy of user portraits, conversion rate of marketing activities, etc.).

[0064] Construct a training sample set: Use the features of historical user behavior data as input features and the data usage effects as labels to construct a training sample set for the GBDT algorithm.

[0065] Training GBDT model: Use GBDT algorithm to train the training sample set, and improve the evaluation accuracy by building multiple decision trees and integrating their output results. During the training process, the number of trees, learning rate, maximum depth and other parameters of GBDT algorithm are optimized through grid search method.

[0066] Application model: During the real-time data collection process, the trained data quality assessment model is used to assess the quality of the collected user behavior data. Based on the assessment results, it can be decided whether to continue collecting data from the data source, or to clean and preprocess the data to improve data quality. Example

[0067] The data acquisition decision model is constructed using the Deep Deterministic Policy Gradient (DDPG) algorithm. The specific definition, components and implementation of the model are described in detail below.

[0068] Definition of data collection decision model: State space: defined as the combination of data quality indicators of multiple data sources, data source availability status, and data acquisition system status.

[0069] Data quality indicators include data completeness, accuracy, consistency, and timeliness, which are used to measure the quality level of data.

[0070] The data source availability status includes the online status, response time, and data throughput of the data source, which reflect the availability and performance of the data source.

[0071] The data acquisition system status includes system load, storage capacity, and processing power, which represent the current operating status of the data acquisition system.

[0072] The action space is defined as the set of executable data source selection, acquisition frequency adjustment, and data preprocessing operations.

[0073] The actions that can be performed include selecting data sources with high data quality to ensure that the collected data is of high quality.

[0074] Adjust the frequency of data collection and dynamically adjust the collection frequency to optimize collection efficiency based on the real-time performance of the data source and system load.

[0075] Perform data cleaning and formatting operations to remove noise and errors in the data and make the data conform to the predetermined format and standards.

[0076] Data compression and encryption are used to reduce data storage and transmission costs and protect data privacy and security.

[0077] The reward function calculates rewards or penalties based on the data collection efficiency and data quality evaluation results.

[0078] Positive rewards are given for actions that improve collection efficiency and data quality to encourage the model to take these beneficial actions.

[0079] Penalize operations that reduce acquisition efficiency and data quality to discourage the model from taking these unfavorable operations.

[0080] The reward function is designed to guide the model to find the best balance between acquisition efficiency and data quality.

[0081] The output strategy is defined as the method of selecting the best action in the current state. The generated decision actions are continuously optimized through the deep reinforcement learning algorithm (DDPG algorithm). The DDPG algorithm combines the advantages of deep learning and reinforcement learning and can handle problems in high-dimensional state space and continuous action space.

[0082] During the training process, the model gradually learns what actions to take in different states to maximize long-term rewards through continuous trial and exploration.

[0083] Specific implementation method State representation: Quantify data quality indicators, data source availability status, and data acquisition system status into specific values ​​or vectors.

[0084] For example, data quality indicators can be quantified through specific indicators such as accuracy, completeness, and consistency; the data source availability status can be quantified through indicators such as online time ratio, average response time, and throughput; the data collection system status can be quantified through indicators such as CPU utilization, memory utilization, and remaining storage capacity.

[0085] Action execution: Select the optimal action based on the output strategy, and perform corresponding data source selection, collection frequency adjustment, and data preprocessing operations.

[0086] For example, when the model determines that the data quality of a certain data source is high, the data source can be selected for collection; when the system load is high, the collection frequency can be reduced to reduce system pressure; when the data contains noise and errors, data cleaning and formatting operations can be performed.

[0087] Reward calculation: Rewards or penalties are calculated based on data collection efficiency and data quality assessment results.

[0088] For example, you can set a positive reward when the collection efficiency increases by a certain percentage, and a positive reward when the data quality assessment score improves; otherwise, a penalty will be given.

[0089] Model training: The DDPG algorithm is used to train the data acquisition decision model.

[0090] During the training process, the model gradually learns what actions to take in different states to maximize long-term rewards by continuously interacting with the environment (i.e. performing actions and observing state changes and rewards).

[0091] Through continuous iterative training, the model gradually optimizes its output strategy and improves the accuracy and efficiency of data collection decisions.

[0092] Taking the intelligent transportation system as an example, assuming that it is necessary to collect traffic data from multiple traffic data sources (such as traffic cameras, vehicle sensors, weather service stations, etc.) and use them for real-time traffic condition monitoring and prediction, the data collection decision model of the present invention can be used to make data collection decisions.

[0093] State space: includes indicators of completeness, accuracy, consistency, and timeliness of traffic data sources, online status, response time, and data throughput of data sources, as well as load, storage capacity, and processing power of data collection systems.

[0094] Action space: includes selecting traffic data sources with high data quality, adjusting the frequency of data collection (such as dynamically adjusting the collection frequency according to traffic congestion), performing data cleaning and formatting operations (such as removing noise data and unifying data formats), and data compression and encryption processing.

[0095] Reward function: Calculate rewards or penalties based on the data collection efficiency and data quality evaluation results. For example, positive rewards are given for timely and accurate collection of key traffic data, and penalties are given for collecting wrong or outdated data.

[0096] Output strategy: The generated decision actions are continuously optimized through the DDPG algorithm. For example, during peak traffic hours, data sources with short response time and high data throughput are selected for collection to improve data collection efficiency and accuracy.

[0097] Embodiment 4: The reward function is used to calculate rewards or penalties based on the real-time data collection efficiency and data quality evaluation results. This embodiment is used to describe the implementation of the reward function in detail, specifically including: During the data collection process, the efficiency indicators of the data collection system are monitored and recorded in real time.

[0098] Collection efficiency indicators may include the amount of data collected per unit time, the success rate of data collection, the delay time of data collection, etc.

[0099] These indicators are obtained through the monitoring module or log system of the data acquisition system and used for subsequent reward calculations.

[0100] Compare the real-time data collection efficiency index with the preset collection efficiency target.

[0101] The preset collection efficiency target can be set based on the performance requirements of the data collection system, business needs or historical data.

[0102] Calculate the collection efficiency deviation, which is the difference between the target collection efficiency and the current collection efficiency (E_target - E_current).

[0103] The data quality assessment model is used to perform real-time quality assessment on the collected data.

[0104] The data quality assessment model outputs quality indicators of current data, such as data completeness, accuracy, consistency, and timeliness.

[0105] These quality indicators are used to measure whether the collected data meets the predetermined quality standards or business requirements.

[0106] Compare the current data quality indicators output by the data quality assessment model with the preset quality standards or historical quality data.

[0107] Pre-set quality criteria can be set based on business rules, data usage requirements, or industry standards.

[0108] Calculate the data quality deviation, which is the difference between the target data quality and the current data quality (Q_target - Q_current).

[0109] Rewards or penalties are calculated based on the collection efficiency deviation and data quality deviation. The specific formula is: R = γ * (E_target - E_current) - δ * (Q_target - Q_current) Among them, R represents the reward value, E_target and E_current represent the target acquisition efficiency and the current acquisition efficiency respectively, Q_target and Q_current represent the target data quality and the current data quality respectively, γ and δ are the weight coefficients of the acquisition efficiency deviation and the data quality deviation respectively, and γ, δ>0.

[0110] These weight coefficients can be set based on business needs, system performance, or experimental adjustments to balance the relative importance of collection efficiency and data quality in reward calculations.

[0111] The calculated reward value R is used to guide the action selection of the data acquisition decision model.

[0112] A positive reward value indicates that the current action improves acquisition efficiency or data quality, encouraging the model to continue taking similar actions.

[0113] A negative reward value (i.e., penalty) indicates that the current action reduces acquisition efficiency or data quality, prompting the model to avoid taking similar actions.

[0114] By continuously optimizing the reward function and adjusting the weight coefficient, the data collection decision model can be guided to find the best balance between collection efficiency and data quality.

[0115] Taking the intelligent logistics system as an example, assuming that logistics data needs to be collected from multiple logistics data sources (such as warehouse management systems, transportation tracking systems, customer feedback systems, etc.) and used for real-time logistics monitoring and decision support, the reward function of the present invention can be used to optimize data collection decisions.

[0116] Real-time data collection efficiency indicators: including the amount of logistics data collected per unit time, the success rate of data collection, the delay time of data collection, etc.

[0117] Preset collection efficiency targets: set according to the performance requirements and business needs of the logistics system, such as the amount of logistics data collected per hour should meet certain standards, the success rate of data collection should be higher than a certain threshold, etc.

[0118] Real-time data quality assessment results: The collected logistics data is evaluated in real time through the data quality assessment model, and quality indicators such as data completeness, accuracy, consistency and timeliness are output.

[0119] Preset quality standards: set according to the rules and requirements of logistics business, such as the accuracy of logistics data should reach more than 99%, the timeliness of data should meet the requirements of real-time updates, etc.

[0120] Reward calculation: Based on the real-time data collection efficiency index and quality assessment results, the reward value is calculated using the reward function formula. For example, when the collection efficiency is improved and the data quality meets the standard, a positive reward is given; when the collection efficiency is reduced or the data quality does not meet the standard, a penalty is given.

[0121] Application of reward function: The calculated reward value is used to guide the action selection of the data collection decision model, such as selecting a data source with high data quality, adjusting the frequency of data collection, performing data cleaning and formatting operations, etc., to optimize the data collection process and improve data quality.

[0122] The present invention also includes a data collection system for multiple data sources, the system comprising: A data source feature extraction module is used to extract data features from multiple data sources; A data quality assessment module, connected to the data source feature extraction module, performs data quality assessment based on the output of the data source feature extraction module; A data collection decision module, connected to the data quality assessment module, models the data collection process as a Markov decision process, and is used to make data collection decisions based on the output of the data quality assessment module; The execution module is connected to the data acquisition decision module and is used to perform real-time data acquisition based on the trained data source feature extraction model, data quality assessment model and data acquisition decision model, and automatically execute the decision results.

[0123] The implementation of the system refers to the above-mentioned embodiment and will not be described in detail in the specification.

[0124] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0125] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data collection method for multiple data sources, characterized in that: The method comprises: Step A: constructing a data source feature extraction model and a data quality assessment model, wherein the data source feature extraction model is used to extract data features from multiple data sources, and the data quality assessment model performs data quality assessment based on the output of the data source feature extraction model; Step B: Construct a data collection decision model and model the data collection process as a Markov decision process for making data collection decisions based on the output of the data quality assessment model; Step C: Perform real-time data collection based on the trained data source feature extraction model, data quality assessment model and data collection decision model, and automatically execute the decision results.

2. A data collection method for multiple data sources according to claim 1, characterized in that: The data source feature extraction model is constructed by using an algorithm combining a self-attention mechanism with a graph convolutional network (GCN), and its network structure includes at least: Input layer: used to receive training data of the model; Hidden layer: contains at least one self-attention mechanism layer for extracting correlation features between data, followed by at least one GCN hidden layer. Each GCN hidden layer contains several GCN units to capture the structural information of the data source. Output layer: Receives the output of the hidden layer and maps the network output to the extracted feature vector through a fully connected layer.

3. The data collection method for multiple data sources according to claim 2, characterized in that: The steps to train a data source feature extraction model include: Initialize the weights and biases of the self-attention mechanism and GCN network using the Xavier initialization method; The cross entropy loss function is used as the loss function to measure the difference between the model extracted features and the real features; The back propagation algorithm and RMSprop optimization algorithm are used to calculate the gradient according to the loss function, and the weights and bias items of the self-attention mechanism and the GCN network are updated. The network is iteratively trained until the loss function converges. Save the trained model as the data source feature extraction model.

4. The data collection method for multiple data sources according to claim 1, characterized in that: The data quality assessment model is constructed using the gradient boosting decision tree (GBDT) algorithm, and its training steps include: Collect training data, including historical data source characteristics, historical data collection efficiency, historical data quality indicators, and corresponding actual data usage effects; The historical data source characteristics, historical data collection efficiency and historical data quality indicators in the training data are used as input features, and the corresponding data usage effects are used as labels to construct the training sample set of the GBDT algorithm; The GBDT algorithm is used to train the training sample set, and the evaluation accuracy is improved by constructing multiple decision trees and integrating their output results; During the training process, the number of trees, learning rate, and maximum depth of the GBDT algorithm are optimized by grid search.

5. The data collection method for multiple data sources according to claim 1, characterized in that: The data acquisition decision model is constructed using the deep deterministic policy gradient DDPG algorithm.

6. A data collection method for multiple data sources according to claim 5, characterized in that: The specific definition of the data acquisition decision model includes: State space: defined as the combination of data quality indicators of multiple data sources, data source availability status, and data acquisition system status; Action space: defined as the set of executable data source selection, collection frequency adjustment, and data preprocessing operations; Reward function: Calculate rewards or penalties based on the data collection efficiency and data quality evaluation results. Give positive rewards to operations that improve collection efficiency and data quality, and give penalties to operations that reduce collection efficiency and data quality. Output strategy: It is defined as the method of selecting the best action in the current state, and the decision actions generated are continuously optimized through deep reinforcement learning algorithms.

7. The data collection method for multiple data sources according to claim 6, characterized in that: The data quality indicators include data integrity, accuracy, consistency and timeliness; the data source availability status includes the online status, response time and data throughput of the data source; the data acquisition system status includes system load, storage capacity and processing power.

8. The data collection method for multiple data sources according to claim 6, characterized in that: The executable data source selection, collection frequency adjustment and data preprocessing operation set includes: selecting a data source with high data quality, adjusting the frequency of data collection, performing data cleaning and formatting operations, and data compression and encryption processing.

9. The data collection method for multiple data sources according to claim 6, characterized in that: The implementation method of the reward function includes: Obtain real-time data collection efficiency indicators; compare the real-time data collection efficiency indicators with the preset collection efficiency targets, and calculate the collection efficiency deviation; Obtain real-time data quality assessment results; compare the current data quality indicators output by the data quality assessment model with the preset quality standards or historical quality data, and calculate the data quality deviation; Rewards or penalties are calculated based on the collection efficiency deviation and data quality deviation. The specific formula is: R = γ * (E_target - E_current) - δ * (Q_target - Q_current) Where R represents the reward value, E_target and E_current represent the target acquisition efficiency and the current acquisition efficiency respectively, Q_target and Q_current represent the target data quality and the current data quality respectively, γ and δ are the weight coefficients of the acquisition efficiency deviation and the data quality deviation respectively, and γ, δ > 0.

10. A data collection system for multiple data sources, characterized in that: The system comprises: A data source feature extraction module is used to extract data features from multiple data sources; A data quality assessment module, connected to the data source feature extraction module, performs data quality assessment based on the output of the data source feature extraction module; A data collection decision module, connected to the data quality assessment module, models the data collection process as a Markov decision process, and is used to make data collection decisions based on the output of the data quality assessment module; The execution module is connected to the data acquisition decision module and is used to perform real-time data acquisition based on the trained data source feature extraction model, data quality assessment model and data acquisition decision model, and automatically execute the decision results.

Images