Big data information management system

Abstract

The invention provides a big data information management system, and relates to the field of big data information management, and the system comprises the following modules: a data collection module which collects data from a plurality of sources in real time and in batches by using data capture, web crawler and API interface technologies; the data storage module is used for storing mass data by adopting a distributed storage system and ensuring high availability and expandability of the data; the data processing module is used for processing and analyzing big data by utilizing a distributed computing framework and improving the processing efficiency through parallel computing; a sorting algorithm is used for data sorting. Through distributed storage and calculation, efficient data processing, advanced visualization, comprehensive data security, system operation and maintenance improvement and flexible data collection, efficient storage and management of mass data are realized, the data processing and analysis capability is improved, the data readability and security are enhanced, the system stability and reliability are ensured, and the application prospect is wide. And meanwhile, the flexibility and efficiency of data collection and integration are improved.

Description

Technical Field

[0001] The present invention relates to the field of big data information management, and in particular to a big data information management system. Background Art

[0002] With the surge in data volume, efficient and flexible data processing has become a core challenge. Traditional data storage systems, such as relational databases, organize data through predefined table structures and relationships, providing powerful data consistency and query capabilities. However, these systems often seem to be unable to cope with massive, unstructured or semi-structured data. Therefore, NoSQL databases came into being, which enable fast reading and writing of large-scale data by abandoning some relational constraints. At the same time, distributed file systems enhance availability and fault tolerance through decentralized data storage, providing new solutions for big data storage. In addition, the rise of big data processing frameworks has made it possible to efficiently analyze huge data sets; these technologies not only innovate in storage structure, but also develop mechanisms such as intelligent indexing and query optimization to further improve data retrieval efficiency; combined with artificial intelligence and machine learning, storage systems can achieve automatic tuning and move towards smarter data management.

[0003] For example, the Chinese invention patent (application number: CN202310344827.3) discloses a "bank customer information management system based on big data and distribution". Its specification discloses: the bank's core customer information management system mainly stores bank-level customer information. As the core customer information management system of the bank, it needs to provide peripheral systems with timely, complete, consistent, accurate, shared, and authoritative bank-level customer information to achieve cross-business line and cross-channel customer information sharing. At the same time, as the number of bank customers continues to increase, the customer information system is required to have big data processing capabilities; the above patents can prove the defects of the existing technology.

[0004] Therefore, we make improvements to this and propose a big data information management system. Summary of the invention

[0005] The purpose of the present invention is to provide powerful data consistency and query capabilities for existing traditional data storage systems, such as relational databases, by organizing data through predefined table structures and relationships; however, these systems often appear to be unable to cope with massive, unstructured or semi-structured data.

[0006] In order to achieve the above-mentioned purpose of the invention, the present invention provides a big data information management system to improve the above-mentioned problems.

[0007] The specific application is as follows: Includes the following modules: Data collection module: using data capture, web crawlers and API interface technology to collect data from multiple sources in real time and batch; Data storage module: uses a distributed storage system to store massive data to ensure high availability and scalability of data; Data processing module: Use distributed computing framework to process and analyze big data, improve processing efficiency through parallel computing; use sorting algorithms to sort data; use graph algorithms to process graph structured data; Data analysis and mining module: using supervised learning algorithms for classification and prediction; using unsupervised learning algorithms to discover potential patterns in data; using association rule mining to discover associations between data items; using optimization algorithms to solve complex optimization problems; using neural networks to process complex nonlinear relationships and high-dimensional data; Data visualization and interaction module: used to display data analysis results in a graphical manner and use an interactive analysis platform to provide real-time data analysis and visualization environment; Data security and privacy protection module: Encryption algorithms are used to encrypt data and protect data confidentiality; hash algorithms are used to verify data integrity; Diffie-Hellman key exchange algorithm is used for secure communication; data governance and management are used to manage data quality, metadata, and data lifecycle; System security and operation and maintenance module: security audit and monitoring mechanisms ensure system security; fault recovery and backup strategies ensure data and service continuity; performance optimization and capacity expansion mechanisms dynamically adjust resources according to system load.

[0008] As a preferred technical solution of this application, the distributed computing framework in the data processing module specifically adopts the following algorithm: MapReduce algorithm: f(inputdata)={(k 1 ,v 1 ),(k 2 ,v 2 ),…,(k n ,v n )}; Where (k 1 ,v 1 ),(k 2 ,v 2 ),…,(k n ,v n） It is a key-value pair generated by function f according to the input data, and each key k i are all associated with a corresponding value v i associated; Map phase: Divide the input data into multiple subtasks for processing and output key-value pairs; its function expression is: (key,value)pairs=f(inputdata); Where (key, value) pairs represents a set, list, or dictionary structure consisting of key-value pairs, f is a function that accepts some input and outputs a key-value pair, and inputdata is the input data of function f; Reduce phase: Group and aggregate the key-value pairs output in the Map phase to generate the final result; its mathematical model is expressed as: textReduce(Map(D))={g(k 1 {v 11 ,v 12 ,…}),g(k 2 ,{v 21 ,v 22 ,…})…}; Where {g(k 1 {v 11 ,v 12 ,…}),g(k 2 ,{v 21 ,v 22 ,…})…} is the key grouping; Combined operation: Combine the Map and Reduce operations to get the final result R: R = Reduce (Map (D)); Spark algorithm: used to represent distributed data sets; RDD transformations and operations are represented as; RDD=(T,P,F)(D); Where T is a function, P is a function or mapping, F is a function, and D is the initial dataset or data source.

[0009] As the preferred technical solution of this application, the data analysis and mining specifically includes the following algorithms: Decision Tree: Information entropy formula: H(D)=−∑ i=1 n p i log 2 (pi); Where H(D) represents the information entropy of data set D, p i Represents the proportion of samples of the i-th category in the data set D, log 2 represents the logarithm with base 2; Information Gain: Gain(A)=H(D)−∑v∈Values(A) ∣D v ∣ / ∣D∣H(D v ); Where Gain(A) represents the information gain obtained by partitioning on attribute A, H(D) is the information entropy of data set D, Values(A) represents the set of all possible values ​​of attribute A, and D v represents the subset of data set D whose attribute A has value v, |D v | and |D| represent the number of samples in the subset Dv and the entire dataset D, respectively. v | / |D| represents subset D v The proportion of H(D v） It is a subset D v The information entropy is calculated according to the information entropy formula; Variance formula: used to measure the degree of dispersion of data; the larger the variance, the more dispersed the data distribution; the smaller the variance, the more concentrated the data distribution; σ 2 =(1 / n)∑ i=1 n (x i −xˉ) 2 ; where σ 2 represents the variance of this set of numbers, n is the number of numbers in this set, x i represents the i-th data point in this group of numbers, xˉ represents the average value of this group of numbers, and x i −xˉ) 2 represents the square of the difference between each data point and the mean value ∑ i=1 n (x i −xˉ) 2 Represents the sum of the squares of the differences between all data points and the mean.

[0010] As the preferred technical solution of this application, the data visualization and interaction module specifically adopts the following algorithm: Support Vector Machines: Decision function: f(x)=w·x+b; Constraints: yi(w·xi+b)≥1,∀i; Where f(x) is the decision function of SVM, which is used to classify new data; yi is the category label of sample xi, w is the weight vector, and b is the bias term; Expectation-Maximization Algorithm: E-step formula: Q(θ|θ(t))=∑ z p(z∣X,θ(t))logp(X,z∣θ); M-step formula: θ(t+1)=argmaxθ Q(θ|θ(t)); where p(z|X,θ(t)) is the posterior probability distribution of the latent variable z given the observed data X and the current parameter estimate θ(t), logp(X,z|θ) is the log-likelihood function of the complete data, and argmax θ It means finding the value of θ that maximizes Q(θ|θ(t)).

[0011] As the preferred technical solution of this application, the data security and privacy protection module specifically adopts the following algorithm: The Huffman coding formula constructs a prefix code table based on the frequency of character occurrence, so that characters with high frequency use shorter codes, thereby achieving data compression; Diffie-Hellman key exchange algorithm: s=g a mod p; s=g b mod p; Where a and b are the private keys chosen by both parties, g is a public generator, and p is a large prime number; AES encryption algorithm: AES is a symmetric encryption algorithm that uses the same key for encryption and decryption. The core of the AES algorithm is the round function, and each round includes byte replacement, row shift, column confusion, and round key addition operations. Encryption process: C=EK(P,K); Where EK is a function that takes plaintext P and key K as input, and outputs ciphertext C; Decryption process: P=DK(C,K); Where DK is a function that takes ciphertext C and key K as input, and outputs plaintext P.

[0012] As the preferred technical solution of this application, the data capture technology is implemented in the following ways: Identify the type of data source that needs to be captured; determine the data fields and formats that need to be captured, and select appropriate tools and technologies based on the type of data source and access restrictions; for web page data, use web browsers, Python programming language, and Selenium data capture tools; for database data, use database connection tools and write SQL query statements; for API interface data, use Python's requests library HTTP request library to call the API interface; Write crawling scripts: Use Python programming and tools to write crawling scripts; the scripts need to implement the functions of sending requests, parsing responses, extracting data, and storing data; Verify and clean data: Verify the captured data to ensure its integrity and accuracy; use data cleaning to remove duplicate data, process missing values, and format data.

[0013] As the preferred technical solution of this application, the implementation of the web crawler technology includes: Use Python programming language and requests library to write crawler code; use multi-threading, multi-process and asynchronous request technology to improve crawler efficiency, control crawler crawling frequency and sequence, analyze the target website's anti-crawler strategy, and take corresponding countermeasures; Storing and exporting data: storing captured data in local files, databases, and other storage systems; using the Pandas library to perform data cleaning, deduplication, and formatting preprocessing operations, while also providing data export functions.

[0014] As the preferred technical solution of this application, the implementation of the API interface technology includes: Write code to call the API interface: Use Python's requests to write code to call the API interface; pass parameters and API keys in the request to obtain the required data; Processing API responses: Parsing the responses returned by the API interface, using the json library to parse the JSON data, and converting it into a data structure that is easy to process; Error handling and data validation: Implement error handling mechanisms and validate returned data to ensure data integrity and accuracy; Caching and current limiting: Store frequently accessed data locally and in memory to increase access speed and reduce pressure on the API interface; implement a current limiting mechanism to control the frequency of access to the API interface to avoid exceeding the usage restrictions of the API interface.

[0015] As a preferred technical solution of the present application, the specific implementation of the data storage module includes: Determine business requirements: Identify the business scenarios and data types that the data storage module needs to support, evaluate the growth rate and expected scale of data to determine the scalability requirements of the storage solution, and select storage technology based on business requirements and data characteristics; Database installation and configuration: Select the MongoDB database management system, install the database software, and perform basic configuration; create the database and table structure according to the data model.

[0016] As the preferred technical solution of the present application, the specific implementation method of the system security and operation and maintenance module is as follows: use the HikariCP connection pool management tool to manage database connections; use the ACID properties provided by the database to manage transactions to ensure data consistency and integrity; use Caffeine local cache and distributed cache Memcached to reduce database access pressure and improve query performance; encrypt and store sensitive data to ensure data security, and regularly back up database data to prevent data loss; use Prometheus database monitoring tool to monitor database performance in real time; and use Alertmanager alarm notification tool to send alarm notifications.

[0017] Compared with the prior art, the present invention has the following beneficial effects: In the scheme of this application: 1. In order to solve the problem of insufficient storage capacity faced by traditional information management systems in the prior art when storing big data, this application adopts distributed storage and data storage modules to achieve efficient storage of massive data in the system, ensure high availability and scalability of data, and take into account business needs and data characteristics, so that the storage solution is more suitable for practical applications, and improves storage efficiency and convenience of data management; 2. In order to solve the problem of low efficiency of traditional systems in processing big data in the prior art, this application realizes efficient processing and analysis of big data through the distributed computing and data analysis and mining modules set up, improves processing efficiency through parallel computing, and provides a wealth of data analysis and mining tools to help users discover potential patterns and laws in the data and provide support for decision-making; 3. The data security and privacy protection module uses encryption algorithm, hash algorithm and key exchange algorithm to ensure the confidentiality, integrity and secure communication of data, solving the problem of data security being difficult to ensure in the big data environment in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The structural diagram of the big data information management system provided for this application; Figure 2 Flow chart of the data capture technology for the big data information management system provided for this application; Figure 3 Flowchart of the web crawler technology for the big data information management system provided for this application; Figure 4 Flow chart of the big data information management system API interface technology provided for this application; Figure 5 Flow chart of the data storage module of the big data information management system provided in this application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0020] As described in the background technology, traditional data storage systems, such as relational databases, organize data through predefined table structures and relationships, providing powerful data consistency and query capabilities; however, when faced with massive, unstructured or semi-structured data, these systems often seem to be unable to cope with it.

[0021] In order to solve this technical problem, the present invention provides a big data information management system, which is applied to distributed data management.

[0022] Specifically, please refer to Figure 1 - Figure 5 ,The big data information management system specifically includes the following modules: Data collection module: using data capture, web crawlers and API interface technology to collect data from multiple sources in real time and batch; Data storage module: uses a distributed storage system to store massive data to ensure high availability and scalability of data; Data processing module: Use distributed computing framework to process and analyze big data, improve processing efficiency through parallel computing; use sorting algorithms to sort data; use graph algorithms to process graph structured data; Data analysis and mining module: using supervised learning algorithms for classification and prediction; using unsupervised learning algorithms to discover potential patterns in data; using association rule mining to discover associations between data items; using optimization algorithms to solve complex optimization problems; using neural networks to process complex nonlinear relationships and high-dimensional data; Data visualization and interaction module: used to display data analysis results in a graphical manner and use an interactive analysis platform to provide real-time data analysis and visualization environment; Data security and privacy protection module: Encryption algorithms are used to encrypt data and protect data confidentiality; hash algorithms are used to verify data integrity; Diffie-Hellman key exchange algorithm is used for secure communication; data governance and management are used to manage data quality, metadata, and data lifecycle; System security and operation and maintenance module: security audit and monitoring mechanisms ensure system security; fault recovery and backup strategies ensure data and service continuity; performance optimization and capacity expansion mechanisms dynamically adjust resources according to system load.

[0023] The big data information management system provided by the present invention realizes efficient storage and management of massive data through distributed storage and computing, efficient data processing, advanced visualization, comprehensive data security, perfect system operation and maintenance, and flexible data collection, improves data processing and analysis capabilities, enhances data readability and security, ensures system stability and reliability, and at the same time improves the flexibility and efficiency of data collection and integration.

[0024] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0025] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments can be combined with each other.

[0026] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0027] Example 1, please refer to Figure 1 , a big data information management system, the distributed computing framework in its data processing module specifically adopts the following algorithm: MapReduce algorithm: f(inputdata)={(k 1 ,v 1 ),(k 2 ,v 2 ),…,(k n ,v n )}; Where (k 1 ,v 1 ),(k 2 ,v 2 ),…,(k n ,v n） It is a key-value pair generated by function f according to the input data, and each key k i are all associated with a corresponding value v i associated; Map phase: Divide the input data into multiple subtasks for processing and output key-value pairs; its function expression is: (key,value)pairs=f(inputdata); Where (key, value) pairs represents a set, list, or dictionary structure consisting of key-value pairs, f is a function that accepts some input and outputs a key-value pair, and inputdata is the input data of function f; Reduce phase: Group and aggregate the key-value pairs output in the Map phase to generate the final result; its mathematical model is expressed as: textReduce(Map(D))={g(k 1 {v 11 ,v 12 ,…}),g(k 2 ,{v 21 ,v 22 ,…})…}; Where {g(k 1 {v 11 ,v 12 ,…}),g(k 2 ,{v 21 ,v 22 ,…})…} is the key grouping; Combined operation: Combine the Map and Reduce operations to get the final result R: R = Reduce (Map (D)); In summary, this process can be concisely expressed in mathematical symbols as follows: R = {g(k,{v1,v2,…})|(k,{v1,v2,…})∈Group(Map(D))} Spark algorithm: used to represent distributed data sets; RDD transformations and operations are represented as; RDD=(T,P,F)(D); Where T is a function that defines the type or mode of data in RDD, P is a function or mapping that describes the partitioning scheme of RDD, that is, how to distribute data to different nodes, F is a function that represents the dependency of RDD, that is, how RDD is calculated from other data sets, and D is the initial data set or data source; The graph algorithm includes the following algorithm formulas: Dijkstra algorithm: d[i] represents the shortest path length from the starting point to node i, V is the node set, w(x,i) is the edge weight from node x to node i; iteratively updates the shortest path length of each node until the shortest path from the starting point to all nodes is found; d[i]=(min / (x∈V)){d[x]+w(x,i)}; Where d[i] represents the shortest distance or minimum cost from a source node to node i, w(x,i) represents the edge weight from node x to node i, and V is the set of all nodes in the graph; A* search algorithm: The A* algorithm expands by continuously selecting nodes with the smallest f(n) value to find the optimal path from the starting point to the target; Valuation function: f(n)=g(n)+h(n); Where f(n) represents the estimated total cost from the starting point to node n, g(n) represents the actual distance cost from the starting point to node n, and h(n) represents the estimated distance cost from node n to the target node; The data preprocessing uses the following algorithm: x i ′={ xi xˉ ; Where xˉ is the mean of the data set {x1,x2,…,xn}, calculated by the following formula: xˉ=(1 / n non-missing)∑ j∈非缺失索引 x j ; Where nnonmissing is the number of nonmissing values ​​in the dataset; Data sorting algorithms: Data sorting algorithms include quick sort and heap sort; Quick Sort: Recursive formula: Quick sort divides the array into two parts through the partition function and recursively sorts the two parts; T(n)=O(n)+T(q−p)+T(r−q); O(n) is the time complexity of the partition operation, q−p is the size of the left subarray, T(q−p) is its sorting time complexity, r−q is the size of the right subarray, T(r−q) is its sorting time complexity; Max-heap properties: For any node i, A[i]>=A[2i+1]&A[i]>=A[2i+2]; Mini-heap property: For any node i, A[i]<=A[2i+1]&A[i]<=A[2i+2].

[0028] By utilizing a distributed computing framework and algorithms such as sorting and graph algorithms, the system can efficiently process and analyze big data and improve processing efficiency through parallel computing; it facilitates solving the problem of low efficiency of traditional systems when processing big data.

[0029] Please refer to Figure 1 , a big data information management system, whose data analysis and mining specifically include the following algorithms: Decision Tree: Information entropy formula: H(D)=−∑ i=1 n p i log 2 (pi); Where H(D) represents the information entropy of data set D, p i Represents the proportion of samples of the i-th category in the data set D, log 2 represents the logarithm with base 2, and the sum symbol ∑ traverses all possible categories in the data set D; Information gain: used to measure the amount of information reduction obtained by partitioning on a given attribute A; Gain(A)=H(D)−∑ v∈Values(A) ∣D v ∣ / ∣D∣H(D v ); Where Gain(A) represents the information gain obtained by partitioning on attribute A, H(D) is the information entropy of data set D, Values(A) represents the set of all possible values ​​of attribute A, and D v represents the subset of data set D whose attribute A has value v, |D v | and |D| represent the number of samples in the subset Dv and the entire dataset D, respectively. v | / |D| represents subset D v The proportion of H(D v） It is a subset D v The information entropy is calculated according to the information entropy formula; Mean formula: used to calculate the average of all values ​​in a data set; xˉ=(1 / n)∑ i=1 n x i ; Where xˉ represents the average value of this group of numbers, n is the number of numbers in this group, and x i represents the i-th number in this set of numbers, ∑ i= 1 n x i It represents the sum of n numbers from x1 to xn; Variance formula: used to measure the degree of dispersion of data; the larger the variance, the more dispersed the data distribution; the smaller the variance, the more concentrated the data distribution; σ 2 =(1 / n)∑ i=1 n (x i −xˉ) 2 ; where σ 2represents the variance of this set of numbers, which measures the degree of dispersion between the data and its mean, n is the number of this set of numbers, x i represents the i-th data point in this group of numbers, xˉ represents the average value of this group of numbers, and x i −xˉ) 2 represents the square of the difference between each data point and the mean value ∑ i=1 n (x i −xˉ) 2 represents the sum of the squares of the differences between all data points and the mean, It should be noted that sometimes when calculating the variance, the denominator is n−1 instead of n. In particular, in the calculation of sample variance, the variance formula of n−1 is used to obtain an unbiased estimate of the sample variance. In this case, the variance formula is: s 2 =(1 / n-1)∑ i=1 n (x i −xˉ) 2 ; Where s2 represents the sample variance; Linear regression formula: used to study the relationship between variables; y=β 0 +β 1 x+e; Among them, y is the dependent variable, which represents the variable we want to predict and explain, β 0 is the regression intercept, which is a constant term that represents the expected value of the dependent variable y when the independent variable x is 0, β 1 is the regression slope, which indicates the strength and direction of the linear relationship between the independent variable x and the dependent variable y. Specifically, β1 tells us how many units y increases (and decreases) on average when x increases by one unit. x is the independent variable (also called the explanatory variable and input), which indicates the variable we use to predict and explain the dependent variable y. e is the error term (also called the residual), which indicates the difference between the actual observed value and the model predicted value. Ideally, the error term is random and its mean is 0, which means that the model can accurately predict the value of the dependent variable. However, in practical applications, due to various factors (such as measurement error and omitted variables), the error term is usually 0. K-means clustering algorithm formula: used to divide the data set into multiple similar subsets J=∑ i=1 k ∑ j=1 ni ∣∣x ij –μ i ∣∣ 2 ; Where J is the value of the objective function, which represents the sum of the squared distances between the data points and their corresponding cluster centers in all clusters, k is the number of clusters, that is, the number of categories we want to divide the data into, and n i is the number of data points in the ith cluster, x ij is the jth data point in the ith cluster, μ i is the center of the ith cluster, which is usually the average of all data points in the cluster, ||x ij –μ i ∣∣ represents the data point x ij and cluster center μ i The Euclidean distance between ij –μ i ∣∣ 2 is the square of the distance; H(D)=–∑ i=1 k p i log(p i ); Where: H(D) represents the information entropy of the data set D, which measures the uncertainty or randomness of the data set, k is the number of possible values ​​in the data set D, p i is the probability of the i-th value appearing in the data set D, log(p i ) is p i The logarithm of the data set D is usually based on base 2, but other bases can also be used. The lower the value of information entropy H(D), the smaller the uncertainty of the data set D, that is, the more concentrated the values ​​(or categories) in the data set; conversely, the higher the value of H(D), the greater the uncertainty of the data set D, that is, the more dispersed the values ​​(or categories) in the data set.

[0030] The data analysis and mining module provides a wealth of data analysis and mining tools, such as supervised learning, unsupervised learning, etc., to help users discover potential patterns and rules in the data and provide support for decision-making.

[0031] Example 2 further optimizes the big data information management system provided in Example 1. Specifically, Figure 1 As shown in the figure, the data visualization and interaction module specifically adopts the following algorithm: Support Vector Machines: Decision function: f(x)=w·x+b Constraints: yi(w·xi+b)≥1,∀i Where f(x) is the decision function of SVM, which is used to classify new data; yi is the category label of sample xi, w is the weight vector, and b is the bias term; the goal of SVM is to find the optimal decision boundary by maximizing the interval so that samples of different categories can be correctly classified; Expectation-maximization algorithm: The EM algorithm is an iterative algorithm used to estimate parameters in the presence of hidden variables; the E step is the expectation calculation step, which is used to calculate the posterior distribution of the hidden variable based on the current parameter estimate; the M step is the maximization step, which is used to update the parameter estimate based on the expected value obtained in the E step; E-step formula: Q(θ|θ(t))=∑ z p(z∣X,θ(t))logp(X,z∣θ); M-step formula: θ(t+1)=argmax θ Q(θ|θ(t)); where p(z|X,θ(t)) is the posterior probability distribution of the latent variable z given the observed data X and the current parameter estimate θ(t), logp(X,z|θ) is the log-likelihood function of the complete data, and argmax θ It means finding the value of θ that maximizes Q(θ|θ(t)).

[0032] The data visualization and interaction module displays the data analysis results in a graphical way and provides an interactive analysis platform, allowing users to understand the data and analysis results more intuitively, improving the readability and ease of use of the data.

[0033] Further, such as Figure 1 As shown, the data security and privacy protection module specifically adopts the following algorithm: The Huffman coding formula constructs a prefix code table based on the frequency of character occurrence, so that characters with high frequency use shorter codes, thereby achieving data compression; Diffie-Hellman key exchange algorithm: s=g a mod p; s=g b mod p; Where a and b are the private keys chosen by both parties, g is a public generator, and p is a large prime number; Since s is g a and g b The remainder under modulo p, so we have: g a ≡ b (mod p); This means that ga and gb have the same remainder after division by p; g a −g b =kp; Where k is some integer; However, we are usually more concerned with the relationship between a and b, especially in discrete logarithm problems; if g is a primitive root modulo p (that is, the power set generated by g modulo p covers all integers from 1 to p-1), then a and b must be congruent modulo ϕ(p) (the value of the Euler function of p, which for prime p is p-1), that is: a≡b(modϕ(p)); If p is a large prime number and g is one of its primitive roots; AES encryption algorithm: AES is a symmetric encryption algorithm that uses the same key for encryption and decryption. The core of the AES algorithm is the round function, and each round includes byte substitution (SubBytes), row shift (ShiftRows), column confusion (MixColumns) and round key addition (AddRoundKey) operations. Encryption process: C=EK(P,K); Where EK is a function that takes plaintext P and key K as input, and outputs ciphertext C; Decryption process: P=DK(C,K); Where DK is a function that takes ciphertext C and key K as input, and outputs plaintext P; It should be noted that the encryption and decryption functions must be inverse to each other, that is, for any plaintext P and key K, we have: textDK(EK(P,K),K)=P; Moreover, for any ciphertext C and key K (assuming that the ciphertext is valid and encrypted by the same key), it can be expressed as: textEK(DK(C,K),K)=C.

[0034] The data security and privacy protection module uses advanced encryption algorithms, hash algorithms and key exchange algorithms to ensure the confidentiality, integrity and secure communication of data; it is used to solve the problem of difficult to ensure data security in a big data environment; through data governance and management, the system can effectively manage data quality, metadata and data life cycle, further improving data security and management efficiency.

[0035] Example 3 further optimizes the big data information management system provided in Example 1 or 2. Specifically, Figure 1 and Figure 2 As shown, the data capture technology is implemented in the following ways: Identify the type of data source that needs to be captured; determine the data fields and formats that need to be captured, and select appropriate tools and technologies based on the type of data source and access restrictions; for web page data, use web browsers, Python programming language, and Selenium data capture tools; for database data, use database connection tools and write SQL query statements; for API interface data, use Python's requests library HTTP request library to call the API interface; Write crawling scripts: Use Python programming and tools to write crawling scripts; the scripts need to implement the functions of sending requests, parsing responses, extracting data, and storing data; Verify and clean data: Verify the captured data to ensure its integrity and accuracy; use data cleaning to remove duplicate data, process missing values, and format data.

[0036] Further, such as Figure 1 and Figure 3 As shown, the implementation methods of web crawler technology include: Use Python programming language and requests library to write crawler code; use multi-threading, multi-process and asynchronous request technology to improve crawler efficiency, control crawler crawling frequency and sequence, analyze the target website's anti-crawler strategy, and take corresponding countermeasures; Storing and exporting data: storing captured data in local files, databases, and other storage systems; using the Pandas library to perform data cleaning, deduplication, and formatting preprocessing operations, while also providing data export functions.

[0037] Further, such as Figure 1 and Figure 4 As shown, the implementation methods of API interface technology include: Write code to call the API interface: Use Python's requests to write code to call the API interface; pass parameters and API keys in the request to obtain the required data; Processing API responses: Parsing the responses returned by the API interface, using the json library to parse the JSON data, and converting it into a data structure that is easy to process; Error handling and data validation: Implement error handling mechanisms and validate returned data to ensure data integrity and accuracy; Caching and current limiting: Store frequently accessed data locally and in memory to increase access speed and reduce pressure on the API interface; implement a current limiting mechanism to control the frequency of access to the API interface to avoid exceeding the usage restrictions of the API interface.

[0038] Further, such as Figure 1 and Figure 5 , the specific implementation of the data storage module includes: Determine business requirements: Identify the business scenarios and data types that the data storage module needs to support, evaluate the growth rate and expected scale of data to determine the scalability requirements of the storage solution, and select storage technology based on business requirements and data characteristics; Database installation and configuration: Select the MongoDB database management system, install the database software, and perform basic configuration; create the database and table structure according to the data model.

[0039] Embodiment 4: The big data information management system provided in the above embodiments is further optimized, such as Figure 1 As shown in the figure, the specific implementation of the system security and operation and maintenance module is as follows: use the HikariCP connection pool management tool to manage database connections; use the ACID properties provided by the database to manage transactions to ensure data consistency and integrity; use Caffeine local cache and distributed cache Memcached to reduce the access pressure on the database and improve query performance; encrypt and store sensitive data to ensure data security. The encryption technology is the same as the encryption technology in the data security and privacy protection module, and regularly back up database data to prevent data loss; use the Prometheus database monitoring tool to monitor database performance in real time; and use the Alertmanager alarm notification tool to send alarm notifications.

[0040] The system security and operation and maintenance module has established a complete security audit and monitoring mechanism to ensure the security of the system; at the same time, it has formulated fault recovery and backup strategies, as well as performance optimization and capacity expansion mechanisms to cope with the ever-changing system load and ensure the stability and performance of the system; this improves the reliability and maintainability of the system.

[0041] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection, an electrical connection, or communication with each other; it can be a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0042] Obviously, the embodiments described above are only some embodiments of the present invention, rather than all embodiments. The preferred embodiments of the present invention are given in the accompanying drawings, but they do not limit the patent scope of the present invention. The present invention can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive. Although the present invention has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific embodiments, or to perform equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of the present invention, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present invention.

Claims

1. A big data information management system, characterized in that: Includes the following modules: Data collection module: using data capture, web crawlers and API interface technology to collect data from multiple sources in real time and batch; Data storage module: uses a distributed storage system to store massive data to ensure high availability and scalability of data; Data processing module: Uses a distributed computing framework to process and analyze big data, and improves processing efficiency through parallel computing; Use sorting algorithms to sort data; use graph algorithms to process graph structured data; Data analysis and mining module: using supervised learning algorithms for classification and prediction; using unsupervised learning algorithms to discover potential patterns in data; Use association rule mining to discover associations between data items; Optimization algorithms are used to solve complex optimization problems; neural networks are used to process complex nonlinear relationships and high-dimensional data; Data visualization and interaction module: used to display data analysis results in a graphical manner and use an interactive analysis platform to provide real-time data analysis and visualization environment; Data security and privacy protection module: Encryption algorithms are used to encrypt data and protect data confidentiality; hash algorithms are used to verify data integrity; Diffie-Hellman key exchange algorithm is used for secure communication; data governance and management are used to manage data quality, metadata, and data lifecycle; System security and operation and maintenance module: security audit and monitoring mechanisms ensure system security; fault recovery and backup strategies ensure data and service continuity; performance optimization and capacity expansion mechanisms dynamically adjust resources according to system load.

2. A big data information management system according to claim 1, characterized in that: The distributed computing framework in the data processing module specifically adopts the following algorithm: MapReduce algorithm: f(inputdata)={(k1,v1),(k2,v2),…,(k n ,v n )}; where (k1,v1),(k2,v2),…,(k n ,v n） It is a key-value pair generated by function f according to the input data, and each key k i are all associated with a corresponding value v i associated; Map phase: divide the input data into multiple subtasks for processing and output key-value pairs; Its function expression is: (key,value)pairs=f(inputdata); Where (key, value) pairs represents a set, list, or dictionary structure consisting of key-value pairs, f is a function that accepts some input and outputs a key-value pair, and inputdata is the input data of function f; Reduce phase: Group and aggregate the key-value pairs output in the Map phase to generate the final result; its mathematical model is expressed as: textReduce(Map(D))={g(k1{v 11 ,v 12 ,…}),g(k2,{v 21 ,v 22 ,…})…}; where {g(k1{v 11 ,v 12 ,…}),g(k2,{v 21 ,v 22 ,…})…} is the key grouping; Combined operation: Combine the Map and Reduce operations to get the final result R: R = Reduce (Map (D)); Spark algorithm: used to represent distributed data sets; RDD transformations and operations are represented as; RDD=(T,P,F)(D); Where T is a function, P is a function or mapping, F is a function, and D is the initial dataset or data source.

3. A big data information management system according to claim 2, characterized in that: The data analysis and mining specifically include the following algorithms: Decision Tree: Information entropy formula: H(D)=−∑ i=1 n p i log2(pi); Where H(D) represents the information entropy of data set D, p i represents the proportion of samples of the i-th category in the data set D, and log2 represents the logarithm with base 2; Information Gain: Gain(A)=H(D)−∑ v∈Values(A) ∣D v ∣ / ∣D∣H(D v ); Where Gain(A) represents the information gain obtained by partitioning on attribute A, H(D) is the information entropy of data set D, Values(A) represents the set of all possible values ​​of attribute A, and D v represents the subset of data set D whose attribute A has value v, |D v | and |D| represent the number of samples in the subset Dv and the entire dataset D, respectively. v | / |D| represents subset D v The proportion of H(D v） It is a subset D v The information entropy is calculated according to the information entropy formula; Variance formula: used to measure the degree of dispersion of data; the larger the variance, the more dispersed the data distribution; the smaller the variance, the more concentrated the data distribution; s 2 = (1 / n)∑ i=1 n (x) i −xˉ) 2 ; where σ 2 represents the variance of this set of numbers, n is the number of numbers in this set, x i represents the i-th data point in this group of numbers, xˉ represents the average value of this group of numbers, and x i −xˉ) 2 represents the square of the difference between each data point and the mean value ∑ i=1 n (x i −xˉ) 2 Represents the sum of the squares of the differences between all data points and the mean.

4. A big data information management system according to claim 3, characterized in that: The data visualization and interaction module specifically adopts the following algorithm: Support Vector Machines: Decision function: f(x)=w·x+b; Constraints: yi(w·xi+b)≥1,∀i; Where f(x) is the decision function of SVM, which is used to classify new data; yi is the category label of sample xi, w is the weight vector, and b is the bias term; Expectation-Maximization Algorithm: E-step formula: Q(θ|θ(t))=∑ z p(z∣X,θ(t))logp(X,z∣θ); M-step formula: θ(t+1)=argmax θ Q(θ|θ(t)); where p(z|X,θ(t)) is the posterior probability distribution of the latent variable z given the observed data X and the current parameter estimate θ(t), logp(X,z|θ) is the log-likelihood function of the complete data, and argmax θ It means finding the value of θ that maximizes Q(θ|θ(t)).

5. A big data information management system according to claim 4, characterized in that: The data security and privacy protection module specifically adopts the following algorithm: The Huffman coding formula constructs a prefix code table based on the frequency of character occurrence, so that characters with high frequency use shorter codes, thereby achieving data compression; Diffie-Hellman key exchange algorithm: s=g a mod p; s=g b mod p; Where a and b are the private keys chosen by both parties, g is a public generator, and p is a large prime number; AES encryption algorithm: AES is a symmetric encryption algorithm that uses the same key for encryption and decryption. The core of the AES algorithm is the round function, and each round includes byte replacement, row shift, column confusion, and round key addition operations. Encryption process: C=EK(P,K); Where EK is a function that takes plaintext P and key K as input, and outputs ciphertext C; Decryption process: P=DK(C,K); Where DK is a function that takes ciphertext C and key K as input, and outputs plaintext P.

6. A big data information management system according to claim 5, characterized in that: The data capture technology is implemented in the following ways: Identify the type of data source that needs to be captured; determine the data fields and formats that need to be captured, and select appropriate tools and technologies based on the type of data source and access restrictions; for web page data, use web browsers, Python programming language, and Selenium data capture tools; for database data, use database connection tools and write SQL query statements; for API interface data, use Python's requests library HTTP request library to call the API interface; Write crawling scripts: Use Python programming and tools to write crawling scripts; the scripts need to implement the functions of sending requests, parsing responses, extracting data, and storing data; Verify and clean data: Verify the captured data to ensure its integrity and accuracy; use data cleaning to remove duplicate data, process missing values, and format data.

7. A big data information management system according to claim 6, characterized in that: The implementation of the web crawler technology includes: Use Python programming language and requests library to write crawler code; use multi-threading, multi-process and asynchronous request technology to improve crawler efficiency, control crawler crawling frequency and sequence, analyze the target website's anti-crawler strategy, and take corresponding countermeasures; Storing and exporting data: storing captured data in local files, databases, and other storage systems; using the Pandas library to perform data cleaning, deduplication, and formatting preprocessing operations, while also providing data export functions.

8. A big data information management system according to claim 7, characterized in that: The implementation of the API interface technology includes: Write code to call the API interface: Use Python's requests to write code to call the API interface; pass parameters and API keys in the request to obtain the required data; Processing API responses: Parsing the responses returned by the API interface, using the json library to parse JSON data, and converting it into a data structure that is easy to process; Error handling and data validation: Implement error handling mechanisms and validate returned data to ensure data integrity and accuracy; Caching and current limiting: Store frequently accessed data locally and in memory to increase access speed and reduce pressure on the API interface; implement a current limiting mechanism to control the frequency of access to the API interface to avoid exceeding the usage restrictions of the API interface.

9. A big data information management system according to claim 8, characterized in that: The specific implementation of the data storage module includes: Determine business requirements: Identify the business scenarios and data types that the data storage module needs to support, evaluate the growth rate and expected scale of data to determine the scalability requirements of the storage solution, and select storage technology based on business requirements and data characteristics; Database installation and configuration: Select the MongoDB database management system, install the database software, and perform basic configuration; create the database and table structure according to the data model.

10. A big data information management system according to claim 9, characterized in that: The system security and operation and maintenance module is specifically implemented as follows: using the HikariCP connection pool management tool to manage database connections; using the ACID properties provided by the database to manage transactions to ensure data consistency and integrity; using Caffeine local cache and distributed cache Memcached to reduce database access pressure and improve query performance; encrypting and storing sensitive data to ensure data security, and regularly backing up database data to prevent data loss; Use the Prometheus database monitoring tool to monitor database performance in real time; use the Alertmanager alarm notification tool to send alarm notifications.

Images