An artificial intelligence-based multi-source data acquisition processing method and system

Abstract

The application relates to the technical field of data processing, and discloses a multi-source data acquisition processing method and system based on artificial intelligence, which comprises the following steps: preferential priority of data of a data source is evaluated based on a data multi-dimensional feature evaluation system and an LSTM network model, an association result of data in an acquired data set is determined based on an association rule algorithm, whether the association result passes an association verification is judged according to a recurrence frequency, the association result is directionally modified and a data association identifier is generated, the association result is supplemented based on the data association identifier and a graph theory method, data in the acquired data set that does not exist in association is classified and a classification result is determined by adopting a three-level classification system, a data degree score of the associated data set and the classification result is determined based on a result of the preferential priority evaluation, the integrity of storage is verified according to a hash value, and a storage position of data storage is adjusted according to a calling access amount. The application ensures real-time performance, integrity and accuracy of multi-source data acquisition processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a multi-source data acquisition processing method and system based on artificial intelligence. BACKGROUND

[0002] With the deep development of digital economy, multi-source data has become the basis for decision analysis and business optimization in fields such as finance, medical treatment, industrial manufacturing, and smart city. For example, the financial field needs to integrate user transaction data, credit data, and market data to achieve risk control. Multi-source data has a wide range of sources, and the data volume is growing explosively, which puts higher requirements on the real-time, completeness, and accuracy of data acquisition and processing.

[0003] Chinese Patent Publication No. CN111143651B discloses a data acquisition and analysis system for new media integrated operation management, which includes a resource management module, a data acquisition module, a data storage module, a data processing module, a data analysis module, a data management module, and an application service module. The data analysis module is used for data extraction, conversion, cleaning, and loading processing, and provides data inspection rule definition, data quality report, planned task management, and abnormal management functions to timely define data processing schemes and realize massive data loading. As can be seen, in the process of data acquisition and processing, there is a lack of mining and dynamic verification capability for implicit associated data, which cannot identify potential association patterns between data. Moreover, for the process of acquisition and processing of massive data, there is a lack of verification mechanism for checking completeness, which is difficult to guarantee the integrity of the data processing process and cannot meet the data retrieval demand.

[0004] Therefore, it is necessary to design a multi-source data acquisition and processing method and system based on artificial intelligence to solve the problems in the current technology. SUMMARY

[0005] In view of this, the present application provides a multi-source data acquisition and processing method and system based on artificial intelligence, which aims to solve the above problems.

[0006] In one aspect, the present application provides a multi-source data acquisition and processing method based on artificial intelligence, which includes:

[0007] Based on the data multi-dimensional feature evaluation system and the LSTM network model, the priority of the data of the data source is evaluated, the data of the data source is acquired based on the result of the priority evaluation, and the acquired data set is determined;

[0008] determine an association result of data in the collected data set based on an association rule algorithm, determine a recurrence frequency of the association result in the collected data set, determine whether to pass an association verification according to the recurrence frequency, when it is determined that the association verification is not passed, modify the association result in a targeted manner and generate a data association identifier, supplement the association result based on the data association identifier and a graph theory method, and determine an association data set;

[0009] classify data in the collected data set that does not exist in association using a three-level classification system to determine a classification result, determine a data degree score based on the result of the priority evaluation of the association data set and the classification result, and determine an encryption mode according to the data degree score;

[0010] store based on the encryption mode, verify the integrity of the storage according to a hash value, and adjust the storage location of the data storage according to the calling access amount.

[0011] Further, when the data in the data source is evaluated based on the data multi-dimensional feature evaluation system and the LSTM network model, it includes:

[0012] The data multi-dimensional feature evaluation system includes timeliness, update frequency and collection cost, obtains a historical data set of the data multi-dimensional feature evaluation system, and divides the historical data set into a training set and a test set;

[0013] Based on the grid search, the initial neural network model is established and constructed, the LSTM network model is trained using the training set, and the test set is substituted into the trained initial neural network model to determine the prediction accuracy;

[0014] When the prediction accuracy is greater than or equal to a prediction accuracy threshold, the trained initial neural network model is determined as the LSTM network model, and the data in the data source is substituted into the LSTM network model to determine the priority of each data in the data source.

[0015] Further, when the association result of the data in the collected data set is determined based on the association rule algorithm, and the recurrence frequency of the association result in the collected data set is determined, it includes:

[0016] According to the Eclat algorithm, a plurality of candidate item sets are generated, the support of the candidate item set is determined to generate a frequent item set, all association rules are generated in all frequent item sets, and the confidence of each association rule is determined, the association rules are filtered based on the support and the confidence, and the association result is determined;

[0017] The collected data set is divided into at least three collected data subsets, the data of the association result is extracted and the recurrence frequency in all collected data subsets is determined.

[0018] Furthermore, when determining whether the association verification passes based on the recurrence frequency, the process includes:

[0019] When all recurrence frequencies are greater than or equal to 0.7, the association verification is deemed successful.

[0020] If the recurrence frequency is less than 0.7, the association verification is deemed unsuccessful.

[0021] Furthermore, when the association verification fails, the process of modifying the association result and generating a data association identifier includes:

[0022] Based on all the association results, determine the attribute association, spatiotemporal association, and logical association;

[0023] For the data associated with the aforementioned attributes, targeted modification will replace the attribute association with a random attribute;

[0024] Directed modification of the logically related data would disrupt its causal condition;

[0025] For the spatiotemporally correlated data, targeted modification will disturb its corresponding timestamp;

[0026] The data association identifier is a data authenticity identifier, a data falsification identifier, and a mixed association identifier;

[0027] If any associated result disappears after the targeted modification, then the data authenticity identifier is generated for that associated result.

[0028] If no associated result disappears after the targeted modification, then a false data identifier is generated for that associated result;

[0029] If, after the directional modification, the recurrence frequency decreases and no associated results disappear, then the hybrid association identifier is generated for that associated result.

[0030] Furthermore, when supplementing the association results based on the data association identifier and graph theory method to determine the associated dataset, the process includes:

[0031] Extract the data that generated the true data identifier, and use a density clustering algorithm on the remaining data in the collected dataset to identify isolated data that are separate from the data that generated the hybrid association identifier;

[0032] Based on graph theory, a path search of up to 3 hops is performed on the isolated data to determine the association between the isolated data and the data that generates the hybrid association identifier. The data that generates the hybrid association identifier and the isolated data that are associated are extracted and constructed with the data that generates the true identifier of the data to form the association dataset.

[0033] Furthermore, when classifying unrelated data in the collected dataset using a three-level classification system to determine the classification result, the following steps are included:

[0034] The remaining data in the collected dataset are used to construct a non-associative dataset;

[0035] The three-level classification system categorizes data sources by type, data type, and data transmission.

[0036] The data source categories include client-side data sources, device-side data sources, and public service-side data sources;

[0037] The data types include text data and image data;

[0038] The data transmission includes batch transmission, incremental transmission, and full transmission;

[0039] The non-related datasets are classified according to the three-level classification system to determine the classification structure dataset.

[0040] Furthermore, when determining a data quality score for the associated dataset and classification results based on the priority assessment results, and determining the encryption method according to the data quality score, the process includes:

[0041] Obtain the priority and data size of each data point in the classification structure dataset, and determine the data severity score based on the priority and data size;

[0042] Obtain the priority corresponding to each data point with a correlation result in the associated dataset, take the highest priority as the correlation priority, and take the total size of the data points with correlation results as the associated data size. Determine the data degree score based on the correlation priority and the associated data size.

[0043] The data severity score is compared with the data severity score threshold;

[0044] When the data quality score is less than the data quality score threshold, then the data in the classification structure dataset, or the data in the association dataset that has an association result, is subject to single encryption.

[0045] When the data quality score is greater than or equal to the data quality score threshold, then the data in the classification structure dataset, or the data in the association dataset that has an association result, is subjected to compound encryption.

[0046] Furthermore, when storing data based on the encryption method, verifying the integrity of the storage according to the hash value, and adjusting the storage location of the data according to the number of accesses, the process includes:

[0047] Determine the first hash value of the data in the classified structure dataset and the associated dataset that has undergone the single or compound encryption;

[0048] The data in the classification structure dataset that is encrypted is stored in the first memory;

[0049] The data that has undergone the compound encryption in the classification structure dataset and the data that has undergone the single encryption in the associated dataset are stored in the second memory;

[0050] The data of the associated dataset that has undergone the composite encryption is stored in a third storage device;

[0051] Determine the second hash value of the data after data storage is completed in the classification structure dataset and the association dataset, and compare it with the corresponding first hash value;

[0052] When the second hash value and the first hash value are equal, it is determined that the data is stored completely.

[0053] If the second hash value and the first hash value are not equal, it is determined that the data is not stored completely and is then stored again.

[0054] After all data in the classification structure dataset and the association dataset has been stored, the access volume of each stored data in the first memory, the second memory and the third memory is determined, and the average access volume of the corresponding memory is determined.

[0055] When the number of calls exceeds the average number of calls, the storage location of the stored data is upgraded by one level.

[0056] Compared with existing technologies, the advantages of this invention are as follows: By combining a multi-dimensional data feature evaluation system with an LSTM network model to prioritize data from data sources, the priority of each data point is accurately identified, avoiding blind collection of massive amounts of data and effectively reducing resource consumption during data collection. Based on association rule algorithms, data association results are determined and verified using recurrence frequency. This not only uncovers potential implicit association patterns between data but also ensures the accuracy of association results through a verification mechanism. For association results that fail verification, targeted modifications are used to generate data association identifiers, supplemented by graph theory methods, further improving the relationships between data and avoiding the risks of failing to identify potential associations or lacking dynamic verification, thus providing comprehensive data support. A three-level classification system is used for unrelated data, improving data retrieval efficiency and meeting the need for rapid retrieval. Simultaneously, data severity scores are determined based on priority evaluation results, and differentiated encryption methods are matched according to these scores, avoiding a "one-size-fits-all" encryption strategy and ensuring the security and adaptability of multi-source data collection and processing. By verifying data integrity through hash values, a reliable integrity verification mechanism is established, effectively preventing data from being tampered with during storage. The storage location is adjusted according to the number of accesses, ensuring the integrity and efficient access of multi-source data processing.

[0057] On the other hand, this application also provides an artificial intelligence-based multi-source data acquisition and processing system for applying the above-mentioned artificial intelligence-based multi-source data acquisition and processing method, including:

[0058] The data acquisition module is configured to prioritize the data from the data source based on a multidimensional feature evaluation system and an LSTM network model, and to acquire the data from the data source based on the priority evaluation results to determine the acquisition dataset.

[0059] The data analysis module is configured to determine the association results of the data in the collected dataset based on the association rule algorithm, and determine the recurrence frequency of the association results in the collected dataset. Based on the recurrence frequency, it determines whether the association verification is passed. When it is determined that the association verification is not passed, the association results are modified in a targeted manner and a data association identifier is generated. Based on the data association identifier and graph theory method, the association results are supplemented to determine the associated dataset.

[0060] The first data processing module is configured to classify the unrelated data in the collected dataset using a three-level classification system to determine the classification result, determine the data degree score of the related dataset and the classification result based on the priority assessment result, and determine the encryption method according to the data degree score;

[0061] The second data processing module is configured to store data based on the encryption method, verify the integrity of the storage based on the hash value, and adjust the storage location of the data according to the number of accesses.

[0062] It is understandable that the above-mentioned multi-source data acquisition and processing method and system based on artificial intelligence have the same beneficial effects, and will not be elaborated further here. Attached Figure Description

[0063] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0064] Figure 1 A flowchart illustrating a multi-source data acquisition and processing method based on artificial intelligence, provided in an embodiment of the present invention;

[0065] Figure 2 This is a functional block diagram of a multi-source data acquisition and processing system based on artificial intelligence, provided as an embodiment of the present invention. Detailed Implementation

[0066] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0067] See Figure 1 As shown in some embodiments of this application, a multi-source data acquisition and processing method based on artificial intelligence includes:

[0068] S100: Prioritize the data from the data source based on the data multidimensional feature evaluation system and LSTM network model, collect the data from the data source based on the priority evaluation results, and determine the collection dataset;

[0069] S200: Based on the association rule algorithm, determine the association results of the data in the collected dataset and determine the recurrence frequency of the association results in the collected dataset. Determine whether the association verification passes based on the recurrence frequency. If the association verification fails, modify the association results in a targeted manner and generate data association identifiers. Supplement the association results based on the data association identifiers and graph theory methods to determine the associated dataset.

[0070] S300: For data that is not related in the collected dataset, a three-level classification system is used to classify and determine the classification results. Based on the priority assessment results, a data quality score is determined for related datasets and classification results, and an encryption method is determined based on the data quality score.

[0071] S400: Stores data using encryption, verifies the integrity of the stored data based on hash values, and adjusts the storage location of the data based on the number of accesses.

[0072] Specifically, in the multi-source data acquisition phase, the data from the data sources is first prioritized using a multi-dimensional data feature evaluation system and an LSTM network model. Data acquisition is then completed based on the evaluation results. The multi-dimensional data feature evaluation system comprehensively assesses the data from multiple dimensions, including timeliness and acquisition cost. The LSTM network model, leveraging its advantages in processing time-series data, captures the changing patterns of the data source across different time dimensions. The combined approach determines the extent of data acquisition from the data source. Given the wide range of sources and explosive growth in data volume, and the inherent differences between data sources, blindly acquiring the entire dataset would lead to resource waste and low efficiency. Prioritization using the multi-dimensional data feature evaluation system and the LSTM network model accurately identifies the priorities of different data from the data source, thereby improving the targeting and efficiency of data acquisition. After acquisition, the collected data is merged into a dataset. Association rule algorithms are used to analyze the association results within the dataset, and the recurrence frequency of these association results is statistically analyzed. This recurrence frequency is then used as the criterion for determining whether the association verification is successful. Association rule algorithms can uncover potential relationships between data from massive datasets, such as the implicit relationship between user consumption data and credit risk data in the financial field. However, the discovered association results alone are insufficient to ensure their reliability. Verification through recurrence frequency is crucial. If the association result repeatedly appears in the collected dataset, it indicates stability and authenticity, thus passing the association verification. If the recurrence frequency is low, it may indicate that other data are associated with the result, leading to failure. Association rule algorithms solve the problem of implicit association mining, and recurrence frequency verification ensures the authenticity of the association results. When association verification fails, the association results are modified in a targeted manner, and data association identifiers are generated. Targeted modifications and data association identifiers can mark and correct unreliable association results. Graph theory methods can construct data association networks to supplement missing association nodes and links, effectively addressing the issues of insufficient implicit association identification and incomplete association results, further ensuring the correlation between data, and thus constructing an association dataset. For data that is not related in the collected dataset, a three-level classification system is used for classification. The three-level classification system divides unrelated data according to hierarchical standards such as the data source category and data transmission. Unrelated data is usually in a messy state and lacks a unified classification logic. Directly processing the data will increase the difficulty of processing. However, the three-level classification system can sort out the disordered and unrelated data into a structured classification result, improve the reliability of data classification, and lay the foundation for subsequent storage and analysis.Based on the previous priority assessment of data in the data source, a data quality score is performed on the associated dataset and classification results. The data quality score takes into account factors such as data priority and data quality, and then determines the corresponding encryption method based on the data quality score. If a uniform encryption method is used for all data, either insufficient encryption will lead to the leakage of sensitive data, or excessive encryption will increase the computational burden and reduce processing efficiency. Matching encryption methods by data quality score can ensure data security while taking into account processing efficiency.

[0073] Understandably, after encrypting and storing data using a defined encryption method, the integrity of the stored data is verified through hash values. Each encrypted data generates a unique hash value, calculated from the data content using a hash algorithm. During storage, encrypted data may experience integrity issues due to hardware failures, human tampering, or other factors. Once the data content is modified (including additions, deletions, and alterations), the corresponding hash value changes. Since hash values ​​are unique and irreversible, they can monitor the integrity status of encrypted data in real time and accurately. By comparing the encrypted hash value with the stored hash value, it's possible to quickly determine if the encrypted data is stored completely. Access volume reflects the access frequency of the encrypted data after storage. Higher access volume indicates a higher frequency of access and greater demand for the data. Therefore, it's migrated to a more reliable storage location with higher backup frequency to ensure data security. This optimizes resource allocation for different types of data, improving the retrieval efficiency of high-frequency data while reducing overall storage costs, effectively addressing the retrieval needs of massive amounts of data.

[0074] In some embodiments of this application, when prioritizing data from a data source based on a multidimensional feature evaluation system and an LSTM network model, the process includes: the multidimensional feature evaluation system includes timeliness, update frequency, and acquisition cost; acquiring historical datasets of the multidimensional feature evaluation system and dividing the historical datasets into training and testing sets; finding and establishing parameters and constructing an initial neural network model based on grid search; training the LSTM network model using the training set; substituting the testing set into the trained initial neural network model to determine the prediction accuracy; when the prediction accuracy is greater than or equal to a prediction accuracy threshold, determining the trained initial neural network model as an LSTM network model; and substituting the data from the data source into the LSTM network model to determine the priority of each piece of data from the data source.

[0075] Specifically, the multidimensional feature evaluation system for data includes three dimensions: timeliness, update frequency, and acquisition cost. These three dimensions directly determine the value and acquisition cost of data in the data source. Timely data can support real-time decision-making capabilities, while update frequency reflects the unknown nature of the data, and acquisition cost relates to the rationality of resource investment. The combination of these three can comprehensively measure the value of data in the data source. Based on this, historical datasets under this system are obtained. Historical datasets include the generation time of each data point in the data source over different time periods, the effective lifespan of the data content, response records supporting decision-making in historical application scenarios, update interval, resource consumption, and corresponding priority labels. These datasets are divided into training and testing sets, typically in a 3:2 ratio, to ensure the generalization ability of the model. The training set is used to allow the model to learn the patterns of data features and priorities, while the testing set is used to test the model's generalization ability. Grid search can traverse parameter combinations in the parameter space to select parameters that are suitable for the data features, thereby constructing an initial neural network model. The initial neural network model contains multiple layers and different types of neurons and activation functions, aiming to capture complex relationships in the data. An initial neural network model is trained using a training set. During training, the model attempts to learn patterns and relationships in the data to improve its prediction or classification capabilities. The test set is then substituted into the trained initial neural network model. Prediction accuracy is used to measure the model's performance, with a preferred threshold of 0.85. When the prediction accuracy is greater than or equal to the threshold, the model's performance is considered satisfactory, and the trained initial neural network model is then designated as an LSTM network model. If the prediction accuracy threshold is not met, training continues, and parameters such as the learning rate and recall are adjusted to ensure that the prediction accuracy is greater than or equal to the threshold. By continuously training the parameters of the initial neural network model, the determined LSTM network model can accurately output the priority of each data point, further improving the targeting and efficiency of multi-source data collection and laying the foundation for overall data processing.

[0076] In some embodiments of this application, when determining the association results of data in the collected dataset based on the association rule algorithm and determining the recurrence frequency of the association results in the collected dataset, the process includes: generating several candidate itemsets according to the Eclat algorithm, determining frequent itemsets based on the support of the candidate itemsets, generating all association rules in all frequent itemsets, determining the confidence of each association rule, filtering association rules based on support and confidence, determining the association results, dividing the collected dataset into at least three subsets of collected data, extracting the data of the association results, and determining the recurrence frequency in all subsets of collected data.

[0077] In some embodiments of this application, when determining whether the association verification is passed based on the recurrence frequency, the method includes: when all recurrence frequencies are greater than or equal to 0.7, the association verification is passed; when there is a recurrence frequency less than 0.7, the association verification is not passed.

[0078] Specifically, the Eclat algorithm generates several candidate itemsets from the collected dataset. Frequent itemsets are then determined based on their support. Subsequently, all association rules are generated from these frequent itemsets, and the confidence score of each rule is calculated. Finally, association rules are selected based on support and confidence to determine the association outcome. The Eclat algorithm efficiently handles complex itemset relationships in multi-source datasets, avoiding the efficiency bottleneck of traditional algorithms with massive datasets. Generating candidate itemsets initially filters for potentially associated data combinations, while support measures the probability of a candidate itemset appearing in the collected dataset. Only frequent itemsets with sufficient support are retained, thus eliminating occasionally occurring invalid itemsets. After generating association rules, the confidence score is determined. Confidence reflects the reliability of the association rule, i.e., the probability that another associated data item will appear when one data item appears. Combining support and confidence for selection ensures the reliability of the association rules and avoids rule bias caused by relying on a single indicator. Support and confidence are dynamically set based on the amount of data in the collected dataset; this embodiment does not impose specific limitations. After determining the correlation between the data, the collected dataset is backed up. The backed-up dataset contains the same data as the original dataset. The backup is only for verifying the correlation results. Except for the backup dataset used for verification, all subsequent datasets used are the original dataset. When dividing the backed-up dataset into at least three subsets, ensure that each subset contains different data samples, and that the data distribution of each subset is consistent with the original dataset (e.g., if the original dataset contains financial transactions, user behavior, and risk assessment, each subset should also contain these three types of data). This ensures coverage of different data distribution characteristics and reduces the impact of local biases on the verification of the correlation results. For the association results selected through support and confidence tests, the frequency of occurrence of the association results is statistically analyzed in each subset of collected data. First, within a single subset, all data points contained in the association result are checked to see if they appear simultaneously. For example, if the association result is "user consumption data A and credit risk data B", the number of times A and B appear simultaneously in the subset is counted. This number is then divided by the total number of data sets in the subset, i.e., the total number of all data combinations in the subset, to obtain the single frequency of the association result in that subset. This process is repeated to obtain the single frequency of the association result across all subsets of collected data. These single frequencies collectively constitute the recurrence frequency of the association result. The single frequency reflects the probability of the association result occurring in a local subset of collected data, while averaging multiple single frequencies to determine the recurrence frequency comprehensively reflects the stability of the association result across different data segments.

[0079] It is understandable that multi-source data inevitably contains a small amount of noisy data, such as collection errors and temporary anomalies. If the recurrence frequency threshold is set too high, the noisy data may cause the originally true association results to be misjudged. If the recurrence frequency threshold is set too low, occasional invalid associations will be included in the pass range. Setting the recurrence frequency threshold to 0.7 ensures that the association results exist in the vast majority (over 70%) of the collected data subsets, thus possessing a certain degree of stability. This ensures the reliability of the association results that pass the association verification, lays a reliable foundation for subsequent targeted modifications and the supplementation of graph theory methods, and further improves the accuracy of multi-source data association mining.

[0080] In some embodiments of this application, when the association verification fails, the association result is modified in a targeted manner and a data association identifier is generated. This includes: determining attribute association, spatiotemporal association, and logical association based on all association results; for attribute-related data, the targeted modification replaces the attribute association with a random attribute; for logically related data, the targeted modification disrupts its causal conditions; for spatiotemporally related data, the targeted modification disturbs its corresponding timestamp; and the data association identifier is a data authenticity identifier, a data falsification identifier, or a mixed association identifier. If an association result disappears after the targeted modification, a data authenticity identifier is generated for that association result; if no association result disappears after the targeted modification, a data falsification identifier is generated for that association result; and if the recurrence frequency decreases after the targeted modification and no association result disappears, a mixed association identifier is generated for that association result.

[0081] Specifically, before making targeted modifications, the three types of associations—attribute associations, spatiotemporal associations, and logical associations—are first identified based on all association results. Attribute associations refer to the associations formed between data due to common attribute characteristics, such as the association between a user's occupation attribute and consumption amount attribute in financial data. Spatiotemporal associations refer to the associations formed between data due to time synchronization and spatial consistency, such as the association between traffic camera data and traffic flow data on the same road segment in a smart city. Logical associations refer to the associations formed between data due to logical relationships such as causality and deduction, such as the causal association between symptom data and diagnosis conclusion data in medical data. Targeted modifications are implemented for different association types. For attribute-related data, the attribute association is replaced with random attributes to sever the original attribute dependencies between data. If the association result is generated due to the existence of attribute features, the modified association result will be affected, thereby verifying whether the attribute-related data is a true association. For logically related data, its causal conditions are disrupted. For example, if the original logic of symptom A diagnosing B is modified so that symptom A cannot be deduced to diagnosis B, the disruption of the condition can be used to test whether the association result truly depends on the logic, avoiding misjudging non-causal accidental associations as valid associations. For spatiotemporal related data, its corresponding timestamps are perturbed. For example, the originally synchronized camera data time and traffic flow data time are changed to be asynchronous. Since spatiotemporal associations depend on the consistency of time or space, perturbing the timestamps can verify whether spatiotemporal synchronization is the key to the existence of the association, eliminating false associations formed by coincidence of time. After making corresponding targeted modifications to each association result, if any association result disappears, a data authenticity identifier is generated for that association result, indicating that the association result does indeed depend on the modified attribute, logic, or spatiotemporal conditions and is a real association result. If no association result disappears after targeted modification, a data false identifier is generated for that association result, indicating that the association result is unrelated to the modified factor, and even targeted modification does not affect the association result; it may be a false association result formed by chance. If the recurrence frequency decreases after targeted modification, and no association result disappears, a mixed association identifier is generated for that association result. Since the recurrence frequency will also change accordingly after targeted modification, if the recurrence frequency decreases after targeted modification, and this decrease is much greater than the decrease in the recurrence frequency of a false association, but no association result disappears, a mixed association identifier is generated, indicating that the association result partially depends on the modified factor, leading to a decrease in the recurrence frequency, and partially depends on other unmodified factors; it belongs to a mixed association of partial components and accidental components of the association result, and there may be other data related to this association result. By selectively modifying the data, the associations of falsely labeled data are removed, while the associations of truely labeled data are retained. For associations of mixed-label data, data that may be associated with them is further explored. This reduces data processing bias caused by false associations and improves the accuracy of supplementing the associated dataset.

[0082] In some embodiments of this application, when supplementing the association results based on data association identifiers and graph theory methods to determine the associated dataset, the process includes: extracting the data that generates the true data identifiers, applying a density clustering algorithm to the remaining data in the collected dataset to determine isolated data that is separate from the data that generates the mixed association identifiers, performing a path search of up to 3 hops on the isolated data based on graph theory methods to determine the association between the isolated data and the data that generates the mixed association identifiers, extracting the data that generates the mixed association identifiers and the isolated data that are associated, and constructing an associated dataset with the data that generates the true data identifiers.

[0083] Specifically, the association results corresponding to the true data identifiers are genuine associations that have been verified through targeted modifications and do not require further association supplementation. Therefore, these are extracted and directly included in the association dataset. For the remaining data in the collected dataset (including data whose association results were broken due to the generation of false data identifiers), a density clustering algorithm is used to identify isolated data outside the data that generated mixed association identifiers. The density clustering algorithm distinguishes between dense regions and isolated points based on the density of data points. Data that generated mixed association identifiers, due to some genuine associations, will form relatively dense data clusters, while isolated data, due to missed associations or unclear association characteristics during collection, will be outside these relatively dense clusters. Density clustering does not require pre-setting cluster shapes to identify isolated data outside the mixed association identifier data. Moreover, isolated data is not invalid noise but may be valid data whose associations were not discovered during the collection process, such as potential associations between missed symptom data and medical history data in medical data. A graph theory method is used to search for paths within three hops on isolated data to determine the association between isolated data and data that generated mixed association identifiers. Graph theory methods construct a network model of data nodes and connecting edges to uncover implicit links between data. Limiting the number of hops to three is because excessive hops lead to overly long links. Three hops cover short-range implicit links between isolated data and data with mixed association markers, such as two-hop links like isolated user behavior data → user preference data → mixed-association consumption data, while avoiding invalid associations caused by excessive hops. Isolated data and data with mixed association markers may have undetected implicit links. Graph theory-based path search can connect isolated data and data with mixed association markers through the link relationships between nodes, thus supplementing any missed associations. Extracting the data that generates mixed association markers and the isolated data with existing associations, and combining them with the data that generates the actual data markers, constructs a relational dataset. This improves the completeness of the relational dataset, thus aligning with the overall relationship profile of multi-source data and reducing processing bias caused by missing associations.

[0084] In some embodiments of this application, when classifying unrelated data in the collected dataset using a three-level classification system to determine the classification result, the following steps are taken: the remaining data in the collected dataset is constructed into an unrelated dataset; the three-level classification system includes classification based on data source category, data type, and data transmission; the data source category includes client data source, device data source, and public service data source; the data type includes text data and image data; and the data transmission includes batch transmission, incremental transmission, and full transmission; and the unrelated dataset is classified according to the three-level classification system to determine the classification structure dataset.

[0085] Specifically, unrelated data within the collected dataset is constructed as a non-associative dataset. Data in a non-associative dataset lacks explicit relationships and is categorized using a three-level classification system. The first level is based on data source type: client-side data sources, device-side data sources, and public service-side data sources. Client-side data sources include data uploaded by users' mobile apps; device-side data sources include data collected by sensors; and public service-side data sources include publicly available data from public service platforms. The second level is based on data type: text data and image data, further refining the non-associative dataset. The third level is based on data transmission: batch transmission, incremental transmission, and full transmission. Different transmission methods correspond to different data update rhythms. This classification optimizes resource allocation for data processing, avoids waste of processing resources due to data disorder, and allows for rapid location of data based on data source type, data type, and data transmission when searching and retrieving data, thereby improving data retrieval efficiency.

[0086] In some embodiments of this application, when determining the data degree score for the associated dataset and classification results based on the priority assessment results, and determining the encryption method based on the data degree score, the method includes: obtaining the priority and data size corresponding to each data in the classification structure dataset; determining the data degree score based on the priority and data size; obtaining the priority corresponding to each data with associated results in the associated dataset, taking the highest priority as the association priority, and taking the sum of the data sizes of the data with associated results as the associated data size; determining the data degree score based on the association priority and associated data size; comparing the data degree score with a data degree score threshold; when the data degree score is less than the data degree score threshold, then performing single encryption on the data in the classification structure dataset, or the data with associated results in the associated dataset; when the data degree score is greater than or equal to the data degree score threshold, then performing compound encryption on the data in the classification structure dataset, or the data with associated results in the associated dataset.

[0087] Specifically, since there is no correlation between data points in a categorical dataset, the data quality score is determined based on the priority and size of each data point. The score is determined by a weighted sum of priority and size; higher priority or larger data points have higher weights, with second-level priority having a greater weight than first-level priority. In contrast, in an association dataset, each correlated data point is composed of multiple data points. The highest priority among these is used as the association priority, and the total size of the correlated data points is used as the association data size. For example, if each correlated data point in an association dataset is a combination of first-level and second-level priorities, with first-level priority data at 5MB and second-level priority data at 3MB, then the association priority is second-level, and the association data size is 8MB. Similarly, the data quality score is determined by a weighted sum of association priority and association data size. It's important to note that whether the score is determined by priority and size or association priority and association data size, the weighted sum is only a weighted sum of numbers, not a weighted sum of units. A higher data quality score indicates that the data or associated data is more important. The data quality score is compared to a data quality score threshold, which is determined based on the average data quality scores of all data in the classification structure dataset and the association dataset. When the data quality score is less than the data quality score threshold, the data in the classification structure dataset or the data in the association dataset that shows association results is subject to single encryption, which includes one of AES-128, AES-256, and ECC. When the data quality score is greater than or equal to the data quality score threshold, the data in the classification structure dataset or the data in the association dataset that shows association results is subject to compound encryption, which includes at least two of AES-128, AES-256, and ECC.

[0088] In some embodiments of this application, when storing data based on encryption, verifying the integrity of the storage based on hash values, and adjusting the storage location of the data based on the number of accesses, the process includes: determining the first hash value of the data in the classification structure dataset and the association dataset that has undergone single or multiple encryption; storing the data in the classification structure dataset that has undergone single encryption into a first memory; storing the data in the classification structure dataset that has undergone multiple encryption, and storing the data in the association dataset that has undergone single encryption into a second memory; storing the data in the association dataset that has undergone multiple encryption into a third memory; determining the second hash value of the data in the classification structure dataset and the association dataset after the data has been stored, and comparing it with the corresponding first hash value; when the second hash value and the first hash value are equal, it is determined that the data is stored completely; when the second hash value and the first hash value are not equal, it is determined that the data is stored incompletely, and the data is re-stored; after all the data in the classification structure dataset and the association dataset has been stored, determining the number of accesses for each stored data in the first memory, the second memory, and the third memory, and determining the average number of accesses for the corresponding memory; when the number of accesses is greater than the average number of accesses, the storage location of the stored data is upgraded by one level.

[0089] Specifically, the priority of the first, second, and third storage devices increases sequentially, and their backup frequencies also increase accordingly. The first hash value is a unique identifier generated by a specific hash algorithm for the encrypted data content. It represents the actual encrypted result of the data. The first hash value ensures that each piece of encrypted data has its own unique baseline; even if the data content is slightly altered or lost, the first hash value will change. Data in the categorical structure dataset that is individually encrypted is stored in the first storage device. Since the categorical structure dataset itself consists of non-associative data and is individually encrypted, its security requirements are relatively low. However, data in the categorical structure dataset that is compounded with encryption and data in the associative dataset that is individually encrypted require relatively higher security and are therefore stored in the second storage device. Data in the associative dataset that is compounded with encryption, and which also has correlation results and is relatively large, requires the highest level of security. The tiered third storage unit ensures precise matching of storage resources to data needs, while reducing wasteful data usage. It determines the second hash value of the encrypted data stored in the first, second, and third storage units and compares it with the corresponding first hash value. If they match, the storage is complete; otherwise, it's incomplete and needs to be re-stored. This prevents data loss or tampering due to hardware failure, network interference, or human error. After all encrypted data is stored, the access frequency of each stored data unit in the first, second, and third storage units is counted. Access frequency reflects the degree to which the data is accessed; higher access frequency indicates high-frequency data, requiring a higher storage tier. For example, if the data is stored in the first storage unit, it is migrated to the second storage unit; if it's in the third storage unit, it remains there without migration. Higher storage tiers result in faster backup frequencies, ensuring data storage security and enabling dynamic optimization of data resource allocation, thus improving the targeting and efficiency of multi-source data acquisition and processing.

[0090] In summary, the beneficial effects of this invention are as follows: By combining a multi-dimensional feature evaluation system with an LSTM network model to prioritize data from data sources, the priority of each data point is accurately identified, avoiding blind collection of massive amounts of data and effectively reducing resource consumption in data collection. Based on association rule algorithms, data association results are determined and verified using recurrence frequency. This not only uncovers potential implicit association patterns between data but also ensures the accuracy of association results through a verification mechanism. For association results that fail verification, targeted modifications are used to generate data association identifiers, supplemented by graph theory methods, further improving the relationships between data and avoiding the risks of failing to identify potential associations or lacking dynamic verification, thus providing comprehensive data support. A three-level classification system is used for unrelated data, improving data retrieval efficiency and meeting the needs of rapid retrieval. Simultaneously, data severity scores are determined based on priority evaluation results, and differentiated encryption methods are matched according to these scores, avoiding a "one-size-fits-all" encryption strategy and ensuring the security and adaptability of multi-source data collection and processing. By verifying data integrity through hash values, a reliable integrity verification mechanism is established, effectively preventing data from being tampered with during storage. The storage location is adjusted according to the number of accesses, ensuring the integrity and efficient access of multi-source data processing.

[0091] In another preferred embodiment based on the above embodiments, see [reference] Figure 2 As shown, this embodiment provides an artificial intelligence-based multi-source data acquisition and processing system for applying the above-described artificial intelligence-based multi-source data acquisition and processing method, including:

[0092] The data acquisition module is configured to prioritize data from the data source based on a multidimensional feature evaluation system and an LSTM network model, and then acquire data from the data source based on the priority evaluation results to determine the acquisition dataset.

[0093] The data analysis module is configured to determine the association results of data in the collected dataset based on the association rule algorithm, and determine the recurrence frequency of the association results in the collected dataset. Based on the recurrence frequency, it determines whether the association verification passes. When the association verification fails, the association results are modified in a targeted manner and a data association identifier is generated. Based on the data association identifier and graph theory methods, the association results are supplemented to determine the associated dataset.

[0094] The first data processing module is configured to classify unrelated data in the collected dataset using a three-level classification system to determine the classification results, determine the data quality score for related datasets and classification results based on the priority assessment results, and determine the encryption method based on the data quality score.

[0095] The second data processing module is configured to store data using encryption, verify the integrity of the storage based on hash values, and adjust the storage location of the data based on the number of accesses.

[0096] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program goods. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program goods according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multi-source data acquisition and processing method based on artificial intelligence, characterized in that, include: The data from the data source is prioritized based on a multidimensional feature evaluation system and an LSTM network model. Based on the priority evaluation results, the data from the data source is collected to determine the collection dataset. The association results of the data in the collected dataset are determined based on the association rule algorithm, and the recurrence frequency of the association results in the collected dataset is determined. The recurrence frequency is used to determine whether the association verification is passed. If the association verification is not passed, the association results are modified in a targeted manner and a data association identifier is generated. The association results are supplemented based on the data association identifier and graph theory method to determine the associated dataset. For data in the collected dataset that are not related, a three-level classification system is used to classify and determine the classification results. Based on the priority assessment results, a data quality score is determined for the related datasets and classification results, and an encryption method is determined based on the data quality score. The storage is performed based on the encryption method, the integrity of the storage is verified based on the hash value, and the storage location of the data is adjusted according to the number of accesses. When the association verification fails, the process of modifying the association result and generating a data association identifier includes: Based on all the association results, determine the attribute association, spatiotemporal association, and logical association; For the data associated with the aforementioned attributes, targeted modification will replace the attribute association with a random attribute; Directed modification of the logically related data would disrupt its causal condition; For the spatiotemporally correlated data, targeted modification will disturb its corresponding timestamp; The data association identifier is a data authenticity identifier, a data falsification identifier, and a mixed association identifier; If any associated result disappears after the targeted modification, then the data authenticity identifier is generated for that associated result. If no associated result disappears after the targeted modification, then a false data identifier is generated for that associated result; If, after the targeted modification, the recurrence frequency decreases and no associated results disappear, then the hybrid association identifier is generated for that association result. When supplementing the association results based on the data association identifier and graph theory method to determine the associated dataset, the following steps are included: Extract the data that generated the true data identifier, and use a density clustering algorithm on the remaining data in the collected dataset to identify isolated data that are separate from the data that generated the hybrid association identifier; Based on graph theory, a path search of up to 3 hops is performed on the isolated data to determine the association between the isolated data and the data that generates the hybrid association identifier. The data that generates the hybrid association identifier and the isolated data that are associated are extracted and constructed with the data that generates the true identifier of the data to form the association dataset.

2. The multi-source data acquisition and processing method based on artificial intelligence according to claim 1, characterized in that, When prioritizing data from data sources based on a multidimensional data feature evaluation system and an LSTM network model, the following are included: The data multidimensional feature evaluation system includes timeliness, update frequency and collection cost. The historical dataset of the data multidimensional feature evaluation system is obtained and the historical dataset is divided into training set and test set. The parameters are found and an initial neural network model is constructed based on grid search. The LSTM network model is trained using the training set, and the test set is substituted into the trained initial neural network model to determine the prediction accuracy. When the prediction accuracy is greater than or equal to the prediction accuracy threshold, the initial neural network model after training is determined as the LSTM network model, and the data from the data source is substituted into the LSTM network model to determine the priority of each data from the data source.

3. The multi-source data acquisition and processing method based on artificial intelligence according to claim 2, characterized in that, When determining the association results of data in the collected dataset based on the association rule algorithm, and determining the recurrence frequency of the association results in the collected dataset, the following are included: Several candidate item sets are generated according to the Eclat algorithm. Frequent itemsets are determined based on the support of the candidate item sets. All association rules are generated in all frequent itemsets. The confidence of each association rule is determined. Association rules are filtered based on the support and confidence. The association result is determined. The collected dataset is divided into at least three subsets, the data of the association results are extracted, and the recurrence frequency in all subsets is determined.

4. The multi-source data acquisition and processing method based on artificial intelligence according to claim 3, characterized in that, When determining whether the association verification passes based on the recurrence frequency, the following steps are included: When all recurrence frequencies are greater than or equal to 0.7, the association verification is deemed successful. If the recurrence frequency is less than 0.7, the association verification is deemed unsuccessful.

5. The multi-source data acquisition and processing method based on artificial intelligence according to claim 4, characterized in that, When classifying unrelated data in the collected dataset using a three-level classification system to determine the classification result, the following are included: The remaining data in the collected dataset are used to construct a non-associative dataset; The three-level classification system includes data source category, data type, and data transmission; The data source categories include client-side data sources, device-side data sources, and public service-side data sources; The data types include text data and image data; The data transmission includes batch transmission, incremental transmission, and full transmission; The non-related datasets are classified according to the three-level classification system to determine the classification structure dataset.

6. The multi-source data acquisition and processing method based on artificial intelligence according to claim 5, characterized in that, When determining a data quality score for the associated dataset and classification results based on the priority assessment results, and determining the encryption method based on the data quality score, the process includes: Obtain the priority and data size of each data point in the classification structure dataset, and determine the data severity score based on the priority and data size; Obtain the priority corresponding to each data point with a correlation result in the associated dataset, take the highest priority as the correlation priority, and take the total size of the data points with correlation results as the associated data size. Determine the data degree score based on the correlation priority and the associated data size. The data severity score is compared with the data severity score threshold; When the data quality score is less than the data quality score threshold, then the data in the classification structure dataset, or the data in the association dataset that has an association result, is subject to single encryption. When the data quality score is greater than or equal to the data quality score threshold, then the data in the classification structure dataset, or the data in the association dataset that has an association result, is subjected to compound encryption.

7. The multi-source data acquisition and processing method based on artificial intelligence according to claim 6, characterized in that, When storing data based on the encryption method, verifying the integrity of the storage according to the hash value, and adjusting the storage location of the data according to the number of accesses, the process includes: Determine the first hash value of the data in the classified structure dataset and the associated dataset that has undergone the single or compound encryption; The data in the classification structure dataset that is encrypted is stored in the first memory; The data that has undergone the compound encryption in the classification structure dataset and the data that has undergone the single encryption in the associated dataset are stored in the second memory; The data of the associated dataset that has undergone the composite encryption is stored in a third storage device; Determine the second hash value of the data after data storage is completed in the classification structure dataset and the association dataset, and compare it with the corresponding first hash value; When the second hash value and the first hash value are equal, it is determined that the data is stored completely. If the second hash value and the first hash value are not equal, it is determined that the data is not stored completely and is then stored again. After all data in the classification structure dataset and the association dataset has been stored, the access volume of each stored data in the first memory, the second memory and the third memory is determined, and the average access volume of the corresponding memory is determined. When the number of calls exceeds the average number of calls, the storage location of the stored data is upgraded by one level.

8. A multi-source data acquisition and processing system based on artificial intelligence, used to apply the multi-source data acquisition and processing method based on artificial intelligence as described in any one of claims 1-7, characterized in that, include: The data acquisition module is configured to prioritize the data from the data source based on a multidimensional feature evaluation system and an LSTM network model, and to acquire the data from the data source based on the priority evaluation results to determine the acquisition dataset. The data analysis module is configured to determine the association results of the data in the collected dataset based on the association rule algorithm, and determine the recurrence frequency of the association results in the collected dataset. Based on the recurrence frequency, it determines whether the association verification is passed. When it is determined that the association verification is not passed, the association results are modified in a targeted manner and a data association identifier is generated. Based on the data association identifier and graph theory method, the association results are supplemented to determine the associated dataset. The first data processing module is configured to classify the unrelated data in the collected dataset using a three-level classification system to determine the classification result, determine the data degree score of the related dataset and the classification result based on the priority assessment result, and determine the encryption method according to the data degree score; The second data processing module is configured to store data based on the encryption method, verify the integrity of the storage based on the hash value, and adjust the storage location of the data according to the number of accesses. When the association verification fails, the process of modifying the association result and generating a data association identifier includes: Based on all the association results, determine the attribute association, spatiotemporal association, and logical association; For the data associated with the aforementioned attributes, targeted modification will replace the attribute association with a random attribute; Directed modification of the logically related data would disrupt its causal condition; For the spatiotemporally correlated data, targeted modification will disturb its corresponding timestamp; The data association identifier is a data authenticity identifier, a data falsification identifier, and a mixed association identifier; If any associated result disappears after the targeted modification, then the data authenticity identifier is generated for that associated result. If no associated result disappears after the targeted modification, then a false data identifier is generated for that associated result; If, after the targeted modification, the recurrence frequency decreases and no associated results disappear, then the hybrid association identifier is generated for that association result. When supplementing the association results based on the data association identifier and graph theory method to determine the associated dataset, the following steps are included: Extract the data that generated the true data identifier, and use a density clustering algorithm on the remaining data in the collected dataset to identify isolated data that are separate from the data that generated the hybrid association identifier; Based on graph theory, a path search of up to 3 hops is performed on the isolated data to determine the association between the isolated data and the data that generates the hybrid association identifier. The data that generates the hybrid association identifier and the isolated data that are associated are extracted and constructed with the data that generates the true identifier of the data to form the association dataset.

Images