Intelligent big data acquisition method and system

Abstract

The invention discloses an intelligent big data acquisition method and system, and relates to the technical field of data acquisition, and the method comprises the steps: deploying a monitored environment condition; calculating the optimal data acquisition frequency according to the current environment condition; acquiring various performance data of the data source based on the data acquisition frequency; and inputting the performance data into a pre-constructed big data tamper-proof model, and feeding back a big data tamper-proof result. According to the method, the problems existing in a traditional method are solved, and powerful support is provided for digital transformation in various fields.

Description

Technical Field

[0001] The present invention relates to the technical field of data collection, and in particular to an intelligent big data collection method and system. Background Art

[0002] With the rapid development of information technology, big data has become an indispensable part of all walks of life. However, in the process of data collection and processing, there are many challenges, such as huge data volume, diverse data sources, uneven data quality, and potential data tampering risks. Traditional big data collection methods often use fixed frequencies or manually set rules to collect data. This method is difficult to adapt to the dynamically changing network environment and technical conditions, and it is easy to waste resources or miss key information. In addition, the lack of an effective anti-tampering mechanism makes it impossible to fully guarantee the authenticity and integrity of the data.

[0003] In view of this, it is necessary to improve the existing big data collection methods. Summary of the invention

[0004] The present invention aims to solve one of the technical problems in the prior art to at least a certain extent, and proposes an intelligent big data collection method and system.

[0005] To achieve the above objectives, in a first aspect, the present application provides an intelligent big data collection method, comprising: Environmental conditions for deployment monitoring; Calculate the optimal data collection frequency based on current environmental conditions; Collect various performance data of data sources based on data collection frequency; Input various performance data into the pre-built big data anti-tampering model and feedback the big data anti-tampering results.

[0006] Furthermore, the environmental conditions include: simultaneous data requests, bandwidth, and query complexity.

[0007] Further, the calculating of the optimal data collection frequency according to the current environmental conditions includes: ; Where A represents the maximum acquisition frequency based on the data source performance indicator; k is the adjustment coefficient of the delay on the acquisition frequency; λ is the proportional coefficient of simultaneous data requests and bandwidth to delay; n(t) is the number of simultaneous data requests; b(t) is the transmission capacity, B is the adjustment coefficient of the load on the acquisition frequency; μ is the proportional coefficient of query complexity and data item size to data load; q(t) is the complexity of the query operation; s(t) is the volume of a single data point; ξ is the proportional coefficient of the cumulative effect; r(s) is the successful acquisition ratio; θ is the adjustment coefficient of the error rate on the cumulative effect; e(s) is the failed acquisition ratio.

[0008] Furthermore, the big data anti-tampering model includes: Data extraction layer, data feature mapping layer, data context understanding layer, decision fusion layer, and tampering location layer; The data extraction layer generates a vector representation of each word or character in the performance data and converts it into a vector sequence of fixed length; The data feature mapping layer processes the output of the data extraction layer to further capture the hierarchical structure and semantic information in the data; The data context understanding layer enhances data understanding and extracts effective features; The decision fusion layer combines information from the data feature mapping layer and the data context understanding layer to determine the final classification result; The tampering location layer locates specific abnormal data when detecting that the data has been tampered with.

[0009] Furthermore, the data extraction layer uses the Word2Vec model architecture to generate vector representations of each word or character in the performance data and converts them into a vector sequence of fixed length.

[0010] Furthermore, the data feature mapping layer includes: S1: Through two convolutional layers, the convolution kernel size is 3x3, the number of convolution kernels is 64, and the word embedding matrix is ​​convolved to obtain the WH64 feature map, where W represents the data sequence length and H represents the embedding dimension; S2: Two more convolutional layers are used, with a kernel size of 5x5 and 64 kernels, to further extract deeper features and obtain the feature map of WH128; S3: Connect the outputs of the two convolutional layers constructed in step S2 to obtain the feature map of WH256 and construct a three-layer pooling layer; S4: Get three output results of different dimensions, where: The first method is to perform a maximum pooling operation on each feature map to obtain a 256-dimensional output result; The second method is to divide each feature map into 4 blocks, and obtain four outputs for each block, and finally obtain a 4*256-dimensional output; The third method is to divide each feature map into 16 blocks, each block gets 16 outputs, and finally gets a 16*256-dimensional output; S5: All outputs are expanded and concatenated into one-dimensional data of 21*256 dimensions, and finally connected to the softmax layer for classification prediction to generate the result of whether the data has been tampered with.

[0011] Furthermore, the data context understanding layer includes: a bidirectional LSTM part, a unidirectional LSTM part and introduces an attention mechanism; The bidirectional LSTM part is used to capture the previous and next dependencies in the data and enhance the understanding of the data semantics; The unidirectional LSTM part mines the information on the time series to ensure that the model can understand the linear structure of the data; The weight distribution of features in the data is obtained by introducing the attention mechanism.

[0012] Furthermore, the decision fusion layer combines the features extracted from the data content by the data feature mapping layer and the context information understood by the data context understanding layer, and uses weighted averaging to perform decision fusion.

[0013] Furthermore, the tampering location layer identifies the probability of tampering by analyzing the weight distribution of the attention mechanism and outputs the specific location information of the tampering.

[0014] In a second aspect, the present application further provides an intelligent big data collection system, which executes an intelligent big data collection method, including: Setting module: deploy monitoring environment conditions; Calculation module: calculates the optimal data collection frequency according to current environmental conditions; Data collection module: collects various performance data of data sources based on data collection frequency; Data monitoring module: Input various performance data into the pre-built big data anti-tampering model and feedback the big data anti-tampering results.

[0015] Beneficial effects of the invention: Compared with the prior art, the present invention provides an intelligent big data collection method and system, which has the following beneficial effects: (1) By monitoring environmental conditions in real time and using adaptive algorithms to dynamically adjust the data collection frequency, unnecessary data transmission and storage are avoided, bandwidth consumption and storage costs are significantly reduced, and the overall efficiency of the system is improved.

[0016] (2) Intelligent adjustment of the collection frequency enables the system to respond quickly when needed, ensuring that no important data updates or events are missed, thereby improving the effectiveness of decision support.

[0017] (3) The integrated big data anti-tampering model can effectively identify and prevent unauthorized data modification and protect the authenticity and integrity of the data. This is particularly important for application scenarios that rely on high-quality data for analysis and decision-making.

[0018] (4) Based on accurate, complete and unaltered data sources, subsequent data analysis work can obtain more reliable results, thereby providing more valuable insights for enterprises and individuals.

[0019] (5) Intelligent data collection methods with a high degree of automation reduce the need for human intervention, reduce operational complexity, and help reduce the probability of human errors.

[0020] (6) The present invention is flexible in design and can be customized according to the specific needs of different industries and application scenarios, and is applicable to various types of data sources and business processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of the steps of the method of the present invention; Figure 2 is a functional block diagram of the system of the present invention; DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Example 1, please refer to Figure 1 As shown, the present application provides an intelligent big data collection method, comprising: Environmental conditions for deployment monitoring; Calculate the optimal data collection frequency based on current environmental conditions; Collect various performance data of data sources based on data collection frequency; Input various performance data into the pre-built big data anti-tampering model and feedback the big data anti-tampering results.

[0024] In the specific implementation process, environmental conditions include: simultaneous data requests, bandwidth, and query complexity.

[0025] Specifically, various performance data from multiple data sources include: different sources (such as web crawlers, sensors, collection devices, social media APIs, file uploads, etc.) obtain different types of data; In the specific implementation process, the optimal data collection frequency is calculated according to the current environmental conditions, including: ; Where A represents the maximum acquisition frequency based on the data source performance indicator (representing the highest frequency that the system can achieve under the optimal state); k is the adjustment coefficient of the delay on the acquisition frequency (determines the degree of influence of the delay on the frequency); λ is the ratio of simultaneous data requests and bandwidth to delay; n(t) is the number of simultaneous data requests; b(t) is the transmission capacity, B is the adjustment coefficient of the load on the acquisition frequency (used to measure the impact of the load on the frequency); μ is the ratio of query complexity and data item size to data load (indicates the impact of query difficulty and data volume on load); q(t) is the complexity of the query operation; s(t) is the volume of a single data point; ξ is the ratio of cumulative effect (used to evaluate the impact of successful acquisition rate and error rate over time); r(s) is the successful acquisition ratio; θ is the adjustment coefficient of the error rate on the cumulative effect (used to adjust the impact of the error rate on the overall effect); e(s) is the failed acquisition ratio, that is, the error rate at time s.

[0026] Specifically, it also includes: introducing historical data analysis and machine learning algorithms to predict future load trends and adjust the collection frequency in advance to cope with potential changes. By continuously collecting data performance in actual operation, the parameters of the adaptive algorithm are optimized to ensure that it is always in the best state.

[0027] Specifically, the latest performance data is obtained from the designated data source according to the determined optimal frequency. Appropriate collection tools and techniques are selected according to different data types, such as Web Scraping, database query, API call, etc. The collected data is cleaned, formatted and standardized to ensure that all data meets unified standards for subsequent analysis.

[0028] In the specific implementation process, the big data anti-tampering model includes: Data extraction layer, data feature mapping layer, data context understanding layer, decision fusion layer, and tampering location layer; The data extraction layer generates vector representations of each word or character in the performance data and converts them into a vector sequence of fixed length; The data feature mapping layer processes the output of the data extraction layer to further capture the hierarchical structure and semantic information in the data; The data context understanding layer enhances data understanding and extracts effective features; The decision fusion layer combines information from the data feature mapping layer and the data context understanding layer to determine the final classification result; The tampering location layer locates specific abnormal data when detecting that the data has been tampered with.

[0029] In the specific implementation process, the data extraction layer uses the Word2Vec model architecture to generate vector representations of each word or character in the performance data and converts them into a fixed-length vector sequence.

[0030] In the specific implementation process, the data feature mapping layer includes: S1: Through two convolutional layers, the convolution kernel size is 3x3, the number of convolution kernels is 64, and the word embedding matrix is ​​convolved to obtain the WH64 feature map, where W represents the data sequence length and H represents the embedding dimension; S2: Two more convolutional layers are used, with a kernel size of 5x5 and 64 kernels, to further extract deeper features and obtain the feature map of WH128. S3: Connect the outputs of the two convolutional layers constructed in step S2 to obtain the feature map of WH256 and construct a three-layer pooling layer; S4: Get three output results of different dimensions, where: The first method is to perform a maximum pooling operation on each feature map to obtain a 256-dimensional output result; The second method is to divide each feature map into 4 blocks, and obtain four outputs for each block, and finally obtain a 4*256-dimensional output; The third method is to divide each feature map into 16 blocks, each block gets 16 outputs, and finally gets a 16*256-dimensional output; S5: All outputs are expanded and concatenated into one-dimensional data of 21*256 dimensions, and finally connected to the softmax layer for classification prediction to generate the result of whether the data has been tampered with.

[0031] In the specific implementation process, the data context understanding layer includes: bidirectional LSTM part, unidirectional LSTM part and the introduction of attention mechanism; The bidirectional LSTM part is used to capture the previous and next dependencies in the data and enhance the understanding of the data semantics; The unidirectional LSTM part mines the information on the time series to ensure that the model can understand the linear structure of the data; The weight distribution of features in the data is obtained by introducing the attention mechanism.

[0032] In the specific implementation process, the decision fusion layer combines the features extracted from the data content by the data feature mapping layer and the context information understood by the data context understanding layer, and uses weighted average to perform decision fusion.

[0033] In the specific implementation process, the tampering location layer identifies the probability of tampering by analyzing the weight distribution of the attention mechanism and outputs the specific location information of the tampering.

[0034] Specifically, the data extraction layer is responsible for converting the original data into a form that can be processed by the machine learning algorithm. For the characters in the data, a pre-trained language model (such as BERT, Word2Vec, GloVe, etc.) is used to generate a vector representation.

[0035] Specifically, the data feature mapping layer further abstracts the output of the data extraction layer through a deep neural network (such as a convolutional neural network (CNN)) to capture deeper structural and semantic information in the data. Specifically, it uses multiple convolutional layers to identify local patterns; applies pooling layers to reduce dimensions while keeping important features unchanged; and also includes residual connections and batch normalization in the specific solution. Finally, it outputs high-level feature representations, which should be able to better reflect the true content and intent of the data.

[0036] Specifically, the data context understanding layer aims to enhance the understanding of data context, ensuring that the model understands not only the meaning of individual words, but also their roles in sentences or paragraphs. Use a bidirectional long short-term memory network framework or a transformer architecture to capture bidirectional dependencies. Introduce an attention mechanism so that the model can focus on a specific part of the data, thereby more accurately understanding the overall meaning of the data. Output a rich feature representation with contextual information.

[0037] Specifically, the decision fusion layer combines information from the data feature mapping layer and the data context understanding layer to determine the final classification result, and uses ensemble learning methods such as voting and weighted average to combine the prediction results from different sources and comprehensively evaluate the importance of multiple models or features. Output is a binary classification result (yes / no) to determine whether the data has been tampered with.

[0038] Specifically, if data tampering is detected, the tampering location layer indicates which parts are affected. This includes: using attention weights or analyzing the activation of each layer within the model to determine the most likely location to be tampered with. Compare the differences between the original data and the suspected tampered data to find inconsistencies, and finally provide specific location information of the tampering, which may be one or more sentences or specific words (such as: the flow of device No. 1; another example: temperature).

[0039] Specifically, each level of the big data anti-tampering model of the present invention has clear functional definitions, which is convenient for independent development, testing and optimization. Through multi-level feature extraction and context understanding, the model's ability to resist different types of tampering behaviors is improved. In particular, at the tampering location layer, a certain degree of explainability is provided to help users understand why certain parts are considered to have been tampered with.

[0040] Example 2, please refer to Figure 2 As shown, the present application also provides an intelligent big data collection system, which executes an intelligent big data collection method, including: Setting module: deploy monitoring environment conditions; Calculation module: calculates the optimal data collection frequency according to current environmental conditions; Data collection module: collects various performance data of data sources based on data collection frequency; Data monitoring module: Input various performance data into the pre-built big data anti-tampering model and feedback the big data anti-tampering results.

[0041] Specifically, in order to meet the above challenges, intelligent big data collection methods have emerged. The present invention can not only adaptively adjust the data collection frequency according to the real-time monitoring results to ensure efficient use of resources and timely capture of important events, but also integrate advanced data anti-tampering models to improve data security and reliability. By deploying a comprehensive monitoring system and applying complex mathematical models and machine learning algorithms, intelligent big data collection methods make it possible to achieve accurate and secure data management.

[0042] In the embodiments provided in the present application, it should be understood that the disclosed system or method can be implemented in other ways. The embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of systems, modules and units can be electrical, mechanical or other forms.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An intelligent big data collection method, characterized in that: include: Environmental conditions for deployment monitoring; Calculate the optimal data collection frequency based on current environmental conditions; Collect various performance data of data sources based on data collection frequency; Input various performance data into the pre-built big data anti-tampering model and feedback the big data anti-tampering results.

2. The intelligent big data collection method according to claim 1, characterized in that: The environmental conditions include: simultaneous data requests, bandwidth, and query complexity.

3. The intelligent big data collection method according to claim 1, characterized in that: The calculating the optimal data collection frequency according to the current environmental conditions includes: ； Where A represents the maximum acquisition frequency based on the data source performance indicator; k is the adjustment coefficient of the delay on the acquisition frequency; λ is the proportional coefficient of simultaneous data requests and bandwidth to delay; n(t) is the number of simultaneous data requests; b(t) is the transmission capacity, B is the adjustment coefficient of the load on the acquisition frequency; μ is the proportional coefficient of query complexity and data item size to data load; q(t) is the complexity of the query operation; s(t) is the volume of a single data point; ξ is the proportional coefficient of the cumulative effect; r(s) is the successful acquisition ratio; θ is the adjustment coefficient of the error rate on the cumulative effect; e(s) is the failed acquisition ratio.

4. The intelligent big data collection method according to claim 1, characterized in that: The big data anti-tampering model includes: Data extraction layer, data feature mapping layer, data context understanding layer, decision fusion layer, and tampering location layer; The data extraction layer generates vector representations of each word or character in the performance data and converts them into a vector sequence of fixed length; The data feature mapping layer processes the output of the data extraction layer to further capture the hierarchical structure and semantic information in the data; The data context understanding layer enhances data understanding and extracts effective features; The decision fusion layer combines information from the data feature mapping layer and the data context understanding layer to determine the final classification result; The tampering location layer locates specific abnormal data when detecting that the data has been tampered with.

5. The intelligent big data collection method according to claim 4, characterized in that: The data extraction layer uses the Word2Vec model architecture to generate vector representations of each word or character in the performance data and converts them into a fixed-length vector sequence.

6. The intelligent big data collection method according to claim 4, characterized in that: The data feature mapping layer includes: S1: Through two convolutional layers, the convolution kernel size is 3x3, the number of convolution kernels is 64, and the word embedding matrix is ​​convolved to obtain the WH64 feature map, where W represents the data sequence length and H represents the embedding dimension; S2: Two more convolutional layers are used, with a kernel size of 5x5 and 64 kernels, to further extract deeper features and obtain the feature map of WH128. S3: Connect the outputs of the two convolutional layers constructed in step S2 to obtain the feature map of WH256 and construct a three-layer pooling layer; S4: Get three output results of different dimensions, where: The first method is to perform a maximum pooling operation on each feature map to obtain a 256-dimensional output result; The second method is to divide each feature map into 4 blocks, and obtain four outputs for each block, and finally obtain a 4*256-dimensional output; The third method is to divide each feature map into 16 blocks, each block gets 16 outputs, and finally gets a 16*256-dimensional output; S5: All outputs are expanded and concatenated into one-dimensional data of 21*256 dimensions, and finally connected to the softmax layer for classification prediction to generate the result of whether the data has been tampered with.

7. The intelligent big data collection method according to claim 4, characterized in that: The data context understanding layer includes: a bidirectional LSTM part, a unidirectional LSTM part and introduces an attention mechanism; The bidirectional LSTM part is used to capture the previous and next dependencies in the data and enhance the understanding of the data semantics; The unidirectional LSTM part mines the information on the time series to ensure that the model can understand the linear structure of the data; The weight distribution of features in the data is obtained by introducing the attention mechanism.

8. The intelligent big data collection method according to claim 4, characterized in that: The decision fusion layer combines the features extracted from the data content by the data feature mapping layer and the context information understood by the data context understanding layer, and uses weighted average to perform decision fusion.

9. The intelligent big data collection method according to claim 7, characterized in that: The tampering location layer identifies the probability of tampering by analyzing the weight distribution of the attention mechanism and outputs the specific location information of the tampering.

10. An intelligent big data acquisition system, characterized in that: Executing an intelligent big data collection method as described in any one of claims 1 to 9, comprising: Setting module: deploy monitoring environment conditions; Calculation module: calculates the optimal data collection frequency according to current environmental conditions; Data collection module: collects various performance data of data sources based on data collection frequency; Data monitoring module: Input various performance data into the pre-built big data anti-tampering model and feedback the big data anti-tampering results.

Images