Big data collection method and system based on cloud computing

Abstract

The present invention discloses a big data collection method and system based on cloud computing, which relates to the field of cloud computing and big data processing technology, including monitoring and collecting data from cloud data sources and performing data preprocessing; analyzing the similarity of the preprocessed data, fusing the clustered similarity data; performing anomaly detection and processing on the fused data; storing and managing the data. The present invention dynamically adjusts the frequency of data collection by constructing an adaptive algorithm, extracts data features for fusion, and then performs anomaly processing on the data, effectively improving the speed and quality of data collection, being able to more effectively cope with the rapid growth of data volume and the diversity of data types, ensuring the high availability and accuracy of data, integrating valuable information in a large amount of raw data, reducing the amount of data to be processed later, and improving data processing efficiency, making data analysis more accurate and efficient, and providing a more reliable data foundation for decision support.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing and big data processing, and in particular to a big data collection method and system based on cloud computing. Background Art

[0002] With the rapid development of cloud computing and big data technologies, data collection and processing technologies have been widely used in many fields, from business intelligence analysis to social science research, to industrial Internet and intelligent manufacturing, all of which have demonstrated the powerful capabilities of these technologies. Cloud computing provides a flexible and scalable computing resource sharing model, which greatly promotes the development and application of big data technology. Big data collection is the first link in the big data technology chain, and its quality and efficiency are directly related to the effectiveness of subsequent data analysis and application. Existing big data collection methods and systems mostly rely on traditional data centers. These methods often encounter inflexible resource allocation, low data processing efficiency and data security issues when processing massive data generated at a high speed. Summary of the invention

[0003] In view of the problems existing in the above-mentioned existing big data collection methods based on cloud computing, the present invention is proposed.

[0004] Therefore, the problem to be solved by the present invention is that most of the existing big data collection methods and systems rely on traditional data centers. These methods often encounter inflexible resource allocation, low data processing efficiency and data security issues when processing massive data generated at a high speed.

[0005] To solve the above technical problems, the present invention provides the following technical solutions: a big data collection method based on cloud computing, which includes monitoring and collecting data from cloud data sources and performing data preprocessing; analyzing the similarity of the preprocessed data and fusing the clustered similarity data; performing anomaly detection and processing on the fused data; and storing and managing the data.

[0006] As a preferred solution of the cloud computing-based big data collection method of the present invention, wherein: the cloud data source monitoring and data collection includes the following steps:

[0007] Deploy Amazon CloudWatch monitoring service on the cloud platform to monitor the status and performance indicators of data sources in real time;

[0008] Register data sources on the cloud platform and configure monitoring parameters based on the data source type and business requirements;

[0009] According to the monitoring results, the frequency of data collection is dynamically adjusted using an adaptive algorithm. The formula is as follows:

[0010] ;

[0011] In the formula, is the maximum collection frequency based on the data source performance indicators. is the adjustment coefficient of the delay on the acquisition frequency, is the ratio of the number of concurrent accesses and bandwidth to latency, is the number of concurrent accesses at time t, is the available bandwidth at time t, is the adjustment coefficient of the load on the acquisition frequency, is the ratio of query complexity and data item size to data load, is the query complexity at time t, is the average size of each data item at time t, is the proportionality coefficient of the cumulative effect, is the successful data acquisition rate at time s, is the adjustment coefficient of the error rate on the cumulative effect, is the error rate at time s;

[0012] A fixed number of data points are collected each time, which is 2n+1, where n is a positive integer;

[0013] After each data collection is completed, the 2n+1th data point is marked as the reference point.

[0014] As a preferred solution of the cloud computing-based big data collection method of the present invention, the data preprocessing refers to cleaning the collected data, including removing irrelevant data, correcting erroneous data and processing missing values;

[0015] The data were converted into a unified format and normalized using the Z-score method.

[0016] As a preferred solution of the cloud computing-based big data collection method of the present invention, the similarity analysis of the pre-processed data includes the following steps:

[0017] Receive preprocessed data streams in real time, extract key features from the data according to data characteristics and business requirements, and construct feature vectors;

[0018] The cosine similarity is used to calculate the similarity between the feature vectors of the data points, and all data points are compared pairwise;

[0019] Use the K-means algorithm to cluster similar data points, update the cluster center and member list, and classify the newly received data points into the corresponding cluster based on their similarity.

[0020] As a preferred solution of the cloud computing-based big data collection method of the present invention, the fusion of clustered similarity data includes the following steps:

[0021] Select the reference point in each cluster after clustering as the benchmark for fusion;

[0022] Define cluster C to consist of n data points, each data point The eigenvector of:

[0023] ;

[0024] In the formula, is the eigenvector of the ith data point in the cluster, It is a data point The value of the first feature, It is a data point The value on the second feature, It is a data point The value of the ath feature;

[0025] The feature vectors of the data in each cluster are fused, and the formula is as follows:

[0026] ;

[0027] In the formula, is the feature vector of the fused cluster C, is the total number of data points in cluster C, It is a data point The weights in the fusion process, It is a data point The value of the first feature, It is a data point The value on the second feature, It is a data point The value of the ath feature;

[0028] Define the fused feature vector set X as:

[0029] ;

[0030] In the formula, Represents the first eigenvector in the fused eigenvector set, Represents the second eigenvector in the fused eigenvector set, Represents the nth feature vector in the fused feature vector set;

[0031] Mark the fused feature vector as the new reference point.

[0032] As a preferred solution of the cloud computing-based big data collection method of the present invention, the anomaly detection and processing of the fused data includes the following steps:

[0033] Defining Data Points The Z-score in X is:

[0034] ;

[0035] In the formula, For data points The Z-score, is the i-th data point in the feature vector X, is the mean value of X, is the standard deviation of X;

[0036] Use the anomaly detection algorithm to analyze the fused feature vector and identify the location of the abnormal data points, which is expressed as:

[0037] ;

[0038] In the formula, For data points The anomaly score, For data points The Z-score, is the normalization constant, is a parameter that adjusts the steepness of the curve. is the threshold parameter, is the integration variable, is the total number of data points in the fused feature vector set, refers to the jth data point in the feature vector X;

[0039] For the identified abnormal data points, split the data into two parts according to their position in the time series, analyze the front and back data respectively, and define the replacement strategy;

[0040] The replacement strategy is to use linear interpolation to replace abnormal values ​​for trend data and use the average value within the same period to replace periodic data;

[0041] After anomaly processing, the data is evaluated for quality, and detailed records of anomaly detection, processing, and evaluation are stored in logs.

[0042] As a preferred solution of the cloud computing-based big data collection method of the present invention, wherein: the storing and managing of data refers to storing the processed data in a distributed database of the cloud platform;

[0043] Encrypt data and perform access control, and set user access rights;

[0044] Create indexes for frequently accessed data and back up and archive data regularly;

[0045] Monitor database performance and status in real time.

[0046] Another object of the present invention is to provide a big data acquisition system based on cloud computing, which comprises:

[0047] Data collection module, used to monitor cloud data sources, automatically collect data from different data sources and pre-process the data;

[0048] The data similarity analysis and fusion module is used to analyze the similarity of preprocessed data, cluster similar data, and fuse the clustered data;

[0049] Anomaly detection and processing module, used to detect and process anomalies on the fused data;

[0050] The data storage management module is used to store the processed data in a distributed database and manage it effectively.

[0051] A computer device comprises: a memory and a processor; the memory stores a computer program, and the processor implements the steps of a big data collection method based on cloud computing when executing the computer program.

[0052] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a big data collection method based on cloud computing.

[0053] The beneficial effects of the present invention are as follows: the present invention dynamically adjusts the frequency of data collection by constructing an adaptive algorithm, extracts data features for fusion and then performs data exception processing, thereby effectively improving the speed and quality of data collection, being able to more effectively cope with the rapid growth of data volume and the diversity of data types, ensuring the high availability and accuracy of data, integrating valuable information in a large amount of raw data, reducing the amount of data for subsequent processing, improving data processing efficiency, making data analysis more accurate and efficient, and providing a more reliable data foundation for decision support. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0055] Figure 1The figure is a flowchart of a big data collection method based on cloud computing.

[0056] Figure 2 Schematic diagram of the implementation of the big data collection method based on cloud computing.

[0057] Figure 3 This is a structural diagram of a big data acquisition system based on cloud computing. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0059] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0060] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0061] Example 1

[0062] Reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides a big data collection method based on cloud computing. The big data collection method based on cloud computing includes the following steps:

[0063] S1, monitor and collect data from cloud data sources and perform data preprocessing;

[0064] Specifically, monitoring and collecting data from cloud data sources includes the following steps:

[0065] Deploy Amazon CloudWatch monitoring service on the cloud platform to monitor the status and performance indicators of data sources in real time;

[0066] Register data sources on the cloud platform and configure monitoring parameters based on the data source type and business requirements;

[0067] According to the monitoring results, the frequency of data collection is dynamically adjusted using an adaptive algorithm. The formula is as follows:

[0068] ;

[0069] In the formula, It is the upper limit value determined based on the maximum acquisition frequency of data source performance indicators, and is determined based on system performance, network bandwidth, data source capacity and other factors. It is the adjustment coefficient of the effect of delay on the acquisition frequency, which is set through pre-performance testing and experience value. It is the ratio of concurrent access number and bandwidth to delay, which is obtained based on historical data analysis. is the number of concurrent accesses at time t, is the available bandwidth at time t, is the adjustment coefficient of the load on the acquisition frequency, It is the ratio of query complexity and data item size to data load, which is adjusted according to data type and processing power. is the query complexity at time t, is the average size of each data item at time t, is the proportionality coefficient of the cumulative effect, obtained through the analysis of historical data collection efficiency, is the successful data acquisition rate at time s, is the adjustment coefficient of the error rate on the cumulative effect, is the error rate at time s,

[0070] A fixed number of data points are collected each time, which is 2n+1, where n is a positive integer;

[0071] After each data collection is completed, the 2n+1th data point is marked as the reference point.

[0072] Amazon CloudWatch monitoring service is a monitoring service provided by Amazon Cloud Computing Service. It allows users to monitor AWS cloud resources and applications running on AWS. By deploying the monitoring service on the cloud platform, the status and performance indicators of the data source are monitored in real time, and the monitoring parameters are configured according to the type of data source and business needs, and the frequency of data collection is dynamically adjusted using adaptive algorithms, it not only improves the flexibility and real-time performance of data collection, but also intelligently adjusts the collection frequency according to the real-time network status and data source performance, reduces data delays and losses caused by resource competition and network congestion, and realizes efficient and flexible data collection, laying the foundation for high-quality data processing.

[0073] Furthermore, data preprocessing refers to cleaning the collected data, including removing irrelevant data, correcting erroneous data, and handling missing values;

[0074] Processing missing values ​​refers to using the mean to fill in the missing values ​​of the processing data;

[0075] The data were converted into a unified format and normalized using the Z-score method.

[0076] The collected data were cleaned, including removing irrelevant data, correcting erroneous data, and processing missing values. The data were then converted into a unified format and standardized using the Z-score, which significantly improved the quality and consistency of the data and provided an accurate and standardized data source for subsequent data analysis. The effective execution of data preprocessing reduced the interference of noise and outliers, ensuring the accuracy and reliability of data analysis.

[0077] S2, analyzing the similarity of the preprocessed data and fusing the clustered similarity data;

[0078] Specifically, analyzing the similarity of preprocessed data includes the following steps:

[0079] Receive preprocessed data streams in real time, extract key features from the data according to data characteristics and business requirements, and construct feature vectors;

[0080] The cosine similarity is used to calculate the similarity between the feature vectors of the data points, and all data points are compared pairwise;

[0081] Use the K-means algorithm to cluster similar data points, update the cluster center and member list, and classify the newly received data points into the corresponding cluster based on their similarity.

[0082] By receiving the preprocessed data stream in real time, extracting key features from it to construct feature vectors, using cosine similarity calculation and K-means algorithm for clustering, it not only optimizes the organizational structure of the data, so that data with similar characteristics are effectively clustered, but also greatly improves the efficiency of data processing and analysis. By clustering similar data, data analysis and mining can be performed more accurately to provide support for decision-making, while also reducing the complexity of data processing and the consumption of computing resources.

[0083] Furthermore, fusing the clustered similarity data includes the following steps:

[0084] Select the reference point in each cluster after clustering as the benchmark for fusion;

[0085] Define cluster C to consist of n data points, each data point The eigenvector of:

[0086] ;

[0087] In the formula, is the eigenvector of the ith data point in the cluster, It is a data point The value of the first feature, It is a data point The value on the second feature, It is a data point The value of the ath feature;

[0088] The feature vectors of the data in each cluster are fused, and the formula is as follows:

[0089] ;

[0090] In the formula, is the feature vector of the fused cluster C, is the total number of data points in cluster C, It is a data point The weights in the fusion process, It is a data point The value of the first feature, It is a data point The value on the second feature, It is a data point The value of the ath feature;

[0091] Define the fused feature vector set X as:

[0092] ;

[0093] In the formula, Represents the first eigenvector in the fused eigenvector set, Represents the second eigenvector in the fused eigenvector set, Represents the nth feature vector in the fused feature vector set;

[0094] Mark the fused feature vector as the new reference point.

[0095] The reference points in each cluster after clustering are selected as the fusion benchmark, and the feature vectors of the data in the cluster are weightedly fused. By accurately defining and executing data fusion strategies, efficient data integration is achieved. This fusion process not only retains the core information of the data, but also reduces the burden of data processing and storage. The new feature vector set obtained through fusion provides a more streamlined and high-quality data source for subsequent data analysis and application.

[0096] S3, perform anomaly detection and processing on the fused data;

[0097] Specifically, anomaly detection and processing of the fused data includes the following steps:

[0098] Defining Data Points The Z-score in X is:

[0099]

[0100] In the formula, For data points The Z-score, is the i-th data point in the feature vector X, is the mean value of X, is the standard deviation of X;

[0101] Use the anomaly detection algorithm to analyze the fused feature vector and identify the location of the abnormal data points, which is expressed as:

[0102] ;

[0103] In the formula, For data points The anomaly score, For data points The Z-score, is the normalization constant, is a parameter that adjusts the steepness of the curve. is the threshold parameter used to determine the switching point where the data point turns into a high sensitivity state. is the integral variable, representing the position in the Gaussian distribution, is the total number of data points in the fused feature vector set, refers to the jth data point in the feature vector X;

[0104] For the identified abnormal data points, split the data into two parts according to their position in the time series, analyze the front and back data respectively, and define the replacement strategy;

[0105] The replacement strategy is to use linear interpolation to replace abnormal values ​​for trend data and use the average value within the same period to replace periodic data;

[0106] After anomaly processing, the data is evaluated for quality, and detailed records of anomaly detection, processing, and evaluation are stored in logs.

[0107] By calculating the Z-score of each data point, the degree of deviation of each data point from the overall data set can be objectively evaluated. The fused feature vector is analyzed using an anomaly detection algorithm. This analysis process can identify the location of abnormal data points, thereby providing precise target positioning for subsequent processing. The targeted processing method not only retains the overall trend and periodic characteristics of the data, but also avoids the interference of outliers on subsequent analysis, thereby improving data accuracy and completeness. By evaluating the quality of the data and storing detailed records of anomaly detection, processing and evaluation in the log, it not only increases the transparency of the processing process, but also provides an important basis for subsequent data analysis and system optimization.

[0108] S4, store and manage data;

[0109] Specifically, storing and managing data means storing the processed data in a distributed database on the cloud platform;

[0110] Encrypt data and perform access control, and set user access rights;

[0111] Create indexes for frequently accessed data and back up and archive data regularly;

[0112] Monitor database performance and status in real time.

[0113] By storing the processed data in the distributed database of the cloud platform, the high availability and elastic expansion capabilities of cloud computing are fully utilized to ensure the efficiency and stability of the big data processing system. The data is encrypted and access control strategies are implemented to enhance data security, ensure the protection of sensitive information and meet compliance requirements. By creating indexes for frequently accessed data, the efficiency of data retrieval is optimized, the system response speed and user experience are improved. Regular data backup and archiving ensure data persistence and recoverability, and reduce the risk of data loss.

[0114] Example 2

[0115] Reference Figure 3 , which is the second embodiment of the present invention, and which is different from the previous embodiment, provides a big data acquisition system based on cloud computing, which includes:

[0116] Data collection module, used to monitor cloud data sources, automatically collect data from different data sources and pre-process the data;

[0117] The data similarity analysis and fusion module is used to analyze the similarity of preprocessed data, cluster similar data, and fuse the clustered data;

[0118] Anomaly detection and processing module, used to detect and process anomalies on the fused data;

[0119] The data storage management module is used to store the processed data in a distributed database and manage it effectively.

[0120] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0121] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0122] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0123] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit with a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit with a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

Claims

1. A big data collection method based on cloud computing, characterized in that: include, Monitor and collect data from cloud data sources and perform data preprocessing; Analyze the similarity of the preprocessed data and fuse the similar data after clustering; Perform anomaly detection and processing on the fused data; Store data and manage it; The cloud data source monitoring and data collection comprises the following steps: Deploy Amazon CloudWatch monitoring service on the cloud platform to monitor the status and performance indicators of data sources in real time; Register data sources on the cloud platform and configure monitoring parameters based on the data source type and business requirements; According to the monitoring results, the frequency of data collection is dynamically adjusted using an adaptive algorithm. The formula is as follows: Where A is the maximum collection frequency based on the data source performance indicator, root k is the adjustment coefficient of the delay on the collection frequency, λ is the proportional coefficient of the number of concurrent accesses and bandwidth to the delay, n(t) is the number of concurrent accesses at time t, b(t) is the available bandwidth at time t, B is the adjustment coefficient of the load on the collection frequency, μ is the proportional coefficient of the query complexity and data item size to the data load, q(t) is the query complexity at time t, s(t) is the average size of each data item at time t, ξ is the proportional coefficient of the cumulative effect, r(s) is the successful data collection rate at time s, θ is the adjustment coefficient of the error rate on the cumulative effect, and e(s) is the error rate at time s; A fixed number of data points are collected each time, which is 2n+1, where n is a positive integer; After each data collection is completed, the 2n+1th data point is marked as the reference point; The analysis of the similarity of the pre-processed data comprises the following steps: Receive preprocessed data streams in real time, extract key features from the data according to data characteristics and business requirements, and construct feature vectors; The cosine similarity is used to calculate the similarity between the feature vectors of the data points, and all data points are compared pairwise; Use the K-means algorithm to cluster similar data points, update the cluster center and member list, and classify the newly received data points into the corresponding cluster according to the similarity judgment results; The fusion of the clustered similarity data comprises the following steps: Select the reference point in each cluster after clustering as the benchmark for fusion; Define cluster C to consist of n data points, each data point D i The eigenvector of: V i =[f i1 ,f i2 ,…,f ia ] Where V i is the feature vector of the i-th data point in the cluster, f i1 is the data point D i The value on the first feature, f i2 is the data point D i The value of the second feature, f ia is the data point D i The value of the ath feature; The feature vectors of the data in each cluster are fused, and the formula is as follows: Where V C is the feature vector of cluster C after fusion, n is the total number of data points in cluster C, and w i is the data point D i The weight in the fusion process, f i1 is the data point D i The value on the first feature, f i2 is the data point D i The value of the second feature, f ia is the data point D i The value of the ath feature; Define the fused feature vector set X as: X={x1,x2,...,x q } In the formula, x1 represents the first feature vector in the fused feature vector set, x2 represents the second feature vector in the fused feature vector set, and x q Represents the nth feature vector in the fused feature vector set; Mark the fused feature vector as a new reference point; The anomaly detection and processing of the fused data includes the following steps: Define data point x i The Z-score in X is: In the formula, Z(x i ) is the data point x i The Z-score, x i is the i-th data point in the feature vector X, μ is the mean of X, and σ is the standard deviation of X; Use the anomaly detection algorithm to analyze the fused feature vector and identify the location of the abnormal data points, which is expressed as: In the formula, E(x i ) is the data point x i The anomaly score, Z(x i ) is the data point x i The Z-score of the feature vector after fusion is , C is the normalization constant, β is the parameter for adjusting the steepness of the curve, γ is the threshold parameter, c is the integral variable, m is the total number of data points in the fused feature vector set, and x j refers to the jth data point in the feature vector X; For the identified abnormal data points, split the data into two parts according to their position in the time series, analyze the front and back data respectively, and define the replacement strategy; The replacement strategy is to use linear interpolation to replace abnormal values ​​for trend data and use the average value within the same period to replace periodic data; After anomaly processing, the data is evaluated for quality, and detailed records of anomaly detection, processing, and evaluation are stored in logs.

2. The big data collection method based on cloud computing according to claim 1, characterized in that: The data preprocessing refers to cleaning the collected data, including removing irrelevant data, correcting erroneous data and processing missing values; The data were converted into a unified format and normalized using the Z-score method.

3. The big data collection method based on cloud computing as claimed in claim 2, characterized in that: Storing and managing data refers to storing the processed data in a distributed database of the cloud platform; Encrypt data and perform access control, and set user access rights; Create indexes for frequently accessed data and back up and archive data regularly; Monitor database performance and status in real time.

4. A cloud computing-based big data acquisition system according to any one of claims 1 to 3, characterized in that: include, Data collection module, used to monitor cloud data sources, automatically collect data from different data sources and pre-process the data; The data similarity analysis and fusion module is used to analyze the similarity of preprocessed data, cluster similar data, and fuse the clustered data; Anomaly detection and processing module, used to detect and process anomalies on the fused data; The data storage management module is used to store the processed data in a distributed database and manage it effectively.

5. A computer device comprising: Memory and processor; The memory stores a computer program, characterized in that: when the processor executes the computer program, the steps of the cloud computing-based big data collection method described in any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the cloud computing-based big data collection method described in any one of claims 1 to 3 are implemented.

Images