Big data analysis method based on adaptive data management and privacy protection

Abstract

According to the big data analysis method based on adaptive data management and privacy protection, an efficient, safe, visual and automatic data analysis solution can be provided, and the requirements of the big data era for data processing and analysis are met. The core innovation point of the invention lies in a self-adaptive data storage architecture and a dynamic data compression and decompression mechanism. The adaptive data storage architecture intelligently selects an optimal storage format and an optimal index strategy by analyzing data characteristics and an access mode in real time, so that the data storage efficiency and the query performance are remarkably improved. Meanwhile, a dynamic data compression and decompression mechanism can automatically adjust a compression algorithm and parameters according to a real-time access mode and content characteristics of the data, so that optimization of a data storage space and balance of an access speed are realized. The two innovations not only solve the efficiency problem in big data storage and access, but also provide a solid foundation for efficient processing and analysis of data.

Description

Technical Field

[0001] The present invention relates to the field of intelligent analysis technology, and in particular to a big data analysis method based on adaptive data management and privacy protection. Background Art

[0002] With the rapid development of information technology, big data has become a key area, playing an important role in business intelligence, scientific research and decision-making. However, existing technologies have shown obvious shortcomings in dealing with the challenges brought by big data. The problem of data storage efficiency is particularly prominent. Traditional storage solutions cannot effectively cope with the diversity of data and the dynamic changes in access patterns, resulting in waste of storage resources and inefficient access speed. In addition, data transmission bottlenecks are becoming increasingly apparent. With the surge in data volume, the efficiency of existing networks and storage systems in the data transmission process has become particularly serious, affecting the real-time nature and overall performance of data processing.

[0003] The issue of privacy protection cannot be ignored either. The risks of personal privacy leakage and data abuse are increasing during data collection and analysis, and existing technologies are often unable to protect personal privacy. The limitation of real-time data processing capabilities is also a significant problem. When processing high-speed real-time data, existing systems often lack the necessary processing power and flexibility to meet the market's demand for instant analysis and response. The singleness of data visualization methods limits users' understanding and insight into complex data sets. Existing tools need to be improved in terms of interactivity, intuitiveness, and expressiveness.

[0004] In high-risk areas such as financial markets, the formulation of risk management strategies usually relies on the experience and intuition of experts, lacking automated and systematic methods, which may lead to slow response and wrong decisions in a rapidly changing market environment. In view of these challenges, there is an urgent need to develop a new type of intelligent big data analysis method that can overcome the limitations of existing technologies through innovative technologies such as adaptive data management, advanced privacy protection technology, efficient real-time data processing framework, innovative data visualization tools and automated risk management strategy generators, and provide an efficient, secure, intuitive and automated big data solution to meet the growing needs of data processing and analysis.

[0005] At present, big data processing and analysis technologies include distributed computing frameworks such as Hadoop and Spark, flexible NoSQL databases, powerful data warehouse solutions, real-time stream processing systems, advanced data mining and machine learning algorithms, intuitive data visualization tools, and scalable services provided by cloud computing platforms. Although these technologies provide strong support for processing and analyzing large-scale data, there are still some limitations, including challenges in data quality control, difficulties in data security and privacy protection, complexity in heterogeneous data integration, latency issues in real-time processing, high complexity thresholds of technology, cost-effectiveness considerations, regulatory compliance requirements, model interpretability, and update pressures brought by technology iterations. These limitations point out the direction in which further innovation and improvement are needed in the field of big data processing and analysis.

[0006] 1. Data storage efficiency issues: Traditional storage solutions cannot effectively cope with the dynamic changes in data diversity and access patterns, resulting in waste of storage resources and reduced access speed.

[0007] 2. Data transmission bottleneck: With the surge in data volume, the efficiency of existing networks and storage systems in the data transmission process has become particularly serious, affecting the real-time and overall performance of data processing.

[0008] 3. Insufficient privacy protection: The risks of personal privacy leakage and data abuse continue to increase during the process of data collection and analysis, and existing technologies have defects in protecting personal privacy.

[0009] 4. Limitations in real-time data processing capabilities: Existing systems often lack the necessary processing capabilities and flexibility to process high-speed inflows of real-time data and are unable to meet the needs of instant analysis and response.

[0010] 5. Single means of data visualization: Existing tools are limited in interactivity, intuitiveness, and expressiveness, which restricts users’ understanding and insight into complex data sets.

[0011] 6. Risk management strategies rely on manual labor: In high-risk areas such as financial markets, the formulation of risk management strategies usually relies on expert experience and intuition, lacking automated and systematic methods. Summary of the invention

[0012] The purpose of the present invention is to provide a big data analysis method based on adaptive data management and privacy protection, which can provide an efficient, secure, intuitive and automated data analysis solution to meet the needs of data processing and analysis in the big data era.

[0013] In order to achieve the above object, the technical solution adopted by the present invention is: a big data analysis method based on adaptive data management and privacy protection, comprising the following steps:

[0014] S1, the system starts and configures the data acquisition module, preparing to receive data streams from multiple specified data sources in real time;

[0015] S2, data quality assessor automatically detects the completeness, consistency and accuracy of the data set, and identifies missing values ​​and outliers;

[0016] S3, data flows into the data preprocessing and cleaning module for preliminary data cleaning, formatting and standardization;

[0017] S4, the cleaned data is stored in the optimal storage structure according to the decision of the adaptive data storage architecture;

[0018] S5, the data stream is compressed through dynamic data compression and decompression mechanism to optimize storage space and access speed;

[0019] S6. The privacy protection framework automatically deploys differential privacy and data desensitization technologies to ensure that personal privacy in the data is protected;

[0020] S7, real-time data processing engine receives and processes high-speed data streams and uses complex event processing technology for real-time analysis;

[0021] S8, Historical Data Analyzer performs in-depth analysis of accumulated data to identify long-term trends and patterns;

[0022] S9. Risk assessment models quantitatively assess the risk level of assets and investment portfolios based on historical and market information;

[0023] S10, the strategy recommendation engine integrates risk assessment and user preferences to generate personalized risk management strategies;

[0024] S11. The analysis results are presented in the form of graphs or charts through an advanced data visualization platform to enhance the user’s understanding of the data;

[0025] S12, large-scale data parallel processing framework efficiently schedules and executes parallel tasks to process large-scale data sets;

[0026] S13. Multi-source heterogeneous data fusion technology integrates data from different sources and formats to form a comprehensive and consistent data view;

[0027] S14. Users obtain analysis reports through a customized report generator, and the system is continuously optimized based on user feedback.

[0028] Preferably, the automated data preprocessing and cleaning module in S3 performs preliminary data cleaning, formatting and standardization processing including the following steps:

[0029] S301, the automated data cleaner intervenes according to the evaluation results and applies intelligent algorithms to deal with each problem in the data set in a targeted manner;

[0030] S302, the data preprocessor performs further data conversion and feature engineering, including data encoding, feature construction, and dimensionality reduction operations.

[0031] Preferably, the storage decision of the adaptive data storage architecture in S4 comprises the following steps:

[0032] S401, the system starts a data characteristic analyzer to analyze the detailed attributes of the incoming data. The analyzer uses statistical methods and machine learning techniques to deeply mine the data patterns and identify the changing trends of the data over time;

[0033] S402, the storage format decision engine receives the report from the data characteristic analyzer, and dynamically determines the storage format of the data according to the analysis result and the preset storage optimization strategy;

[0034] S403, start the dynamic index builder, and update the index structure in real time according to the access mode provided by the data characteristic analyzer and the decision of the storage format decision engine;

[0035] S404, the continuous performance monitoring component of the adaptive data storage architecture continuously evaluates storage and query performance, and based on these evaluation results, the system automatically triggers adjustments to cope with the continuous evolution of data access patterns;

[0036] S405: The system feeds back the user access behavior and system performance to the data characteristic analyzer, forming a closed-loop optimization process.

[0037] Preferably, the compression process of the dynamic data compression and decompression mechanism in S5 comprises the following steps:

[0038] S501, the system starts a real-time data monitor to track and record the access mode and usage of the data stream, and monitor the access frequency, access time, and usage mode of the data;

[0039] S502, as the real-time data monitor collects data, the compression strategy evaluator analyzes the monitored data and evaluates the characteristics and access patterns of the data to determine the most appropriate compression algorithm and parameters;

[0040] S503, after evaluating the optimal compression strategy, the compression strategy evaluator selects a specific compression algorithm based on data characteristics and access patterns, as well as expected compression effects and computing resource consumption;

[0041] S504, the dynamic compression executor automatically performs compression strategy adjustment according to the selection of the evaluator;

[0042] S505, after the data is compressed, the system again uses the real-time data monitor to monitor the compression effect;

[0043] S506. The compression effect obtained through monitoring is fed back to the compression strategy evaluator, forming a closed-loop optimization process. The evaluator adjusts the compression strategy based on the feedback, and the executor adjusts the compression operation again to ensure that the entire compression and decompression mechanism always adapts to the current data characteristics and access requirements.

[0044] Preferably, the process of automatically deploying differential privacy and data desensitization technology by the privacy protection framework in S6 includes the following steps:

[0045] S601, Privacy Policy Definer dynamically updates privacy rules and policies based on global data protection regulations, providing compliance guidance for the system;

[0046] S602. The data desensitization processor processes sensitive information using data masking, generalization, and perturbation techniques to convert personal information into a format that cannot be traced while maintaining the statistical properties of the data;

[0047] S603, the differential privacy executor introduces an appropriate amount of randomness in the data analysis phase, and protects individual data from being accurately traced by adding carefully calculated noise, thus balancing privacy protection and data analysis accuracy;

[0048] S604. The system evaluates and adjusts privacy protection measures through continuous monitoring and feedback mechanisms to respond to regulatory changes and technological evolution, ensuring the continued effectiveness and adaptability of the privacy protection framework.

[0049] Preferably, the real-time data processing engine in S7 processes and analyzes the high-speed data stream including the following steps:

[0050] S701, data stream receiver efficiently receives real-time data streams from multiple sources. The data stream receiver is designed with high concurrent processing capabilities and can process thousands or even tens of thousands of data input channels at the same time, ensuring no data loss under high throughput. It uses advanced buffering technology and load balancing algorithms to optimize the data receiving process and reduce delays.

[0051] S702, the received data stream is then sent to the stream processing analyzer, which uses complex event processing technology CEP and pattern matching algorithm to perform real-time screening, correlation and aggregation operations on the data stream;

[0052] S703. After the stream processing analyzer completes data processing, the real-time result outputter immediately outputs the analysis results in a low-latency manner, using fast data push technology to ensure that the analysis results can be quickly delivered to the front-end monitoring system or decision support tool;

[0053] S704, the performance monitoring module of the real-time data processing engine continuously tracks the running status of the processing engine. Based on these monitoring data, the engine automatically adjusts resource allocation and processing strategies to cope with different data flows and complexities;

[0054] S705. In order to cope with the ever-changing data traffic and processing requirements, the real-time data processing engine adopts microservice architecture and containerized deployment to ensure the scalability and fault tolerance of the system;

[0055] S706. In the process of processing data streams, the real-time data processing engine also integrates security and privacy protection measures to ensure the security of data during transmission and processing; this includes data encryption transmission, access control and data desensitization to prevent the leakage of sensitive information.

[0056] Preferably, the risk assessment model in S9 assesses the risk level of assets and investment portfolios including the following steps:

[0057] S901, the system first collects and analyzes market data through a market trend analyzer, using machine learning algorithms to identify market trends and patterns;

[0058] S902. Risk assessment models use historical data to assess the risk level of assets;

[0059] S903, in the user preference analysis step, the system will consider the user's risk preference and investment objectives, and collect the preference information input by the user through the user interface;

[0060] S904, the strategy recommendation engine generates personalized risk management strategies based on market trends, risk assessments, and user preferences;

[0061] S905. The generated strategy will be backtested using historical data to evaluate its performance under different market conditions;

[0062] S906. The real-time monitoring module continuously tracks market dynamics and portfolio performance. When market conditions change, the system adjusts strategies according to preset rules.

[0063] S907. The system collects user feedback on the strategy and performs iterative optimization accordingly.

[0064] Preferably, the large-scale data parallel processing framework in S12 processes a large-scale data set, including the following steps:

[0065] S1201, the task splitter decomposes the large-scale data processing task into multiple subtasks that can be executed in parallel;

[0066] S1202, after segmentation, the subtasks are assigned to available processing units;

[0067] S1203, the parallel computing scheduler efficiently schedules these subtasks to each node on the multi-core processor or distributed computing cluster;

[0068] S1204, the subtasks are executed in parallel on the allocated computing resources;

[0069] S1205, the subtask is completed, and the results are collected from each computing node;

[0070] S1206, the result merger receives the intermediate results from each node and integrates them into the final output;

[0071] S1207. After the results are merged, the system performs a consistency check to ensure that the results of all subtasks are correctly integrated without data loss or errors;

[0072] S1208. If any inconsistency or error is found, the result merger will perform optimization steps and re-execute the problematic subtasks if necessary to ensure the quality of the final result;

[0073] S1209. The verified and optimized final results are output for further analysis or application.

[0074] Preferably, the multi-source heterogeneous data fusion technology in S13 integrates data from different sources and formats to form a comprehensive and consistent data view, including the following steps:

[0075] S1301, the data source identifier automatically executes to identify different data source types accessing the system;

[0076] S1302: After identifying the data source, the system classifies and labels the data;

[0077] S1303, the data format converter converts the data in the heterogeneous data sources into a unified internal format;

[0078] S1304. While converting data, the system performs data cleaning and standardization operations, removes invalid or erroneous data records, corrects inconsistent data items, and unifies measurements and data formats;

[0079] S1305, the data fusion engine integrates the converted and cleaned data;

[0080] S1306. During the data fusion process, the system performs a consistency check to identify conflicts and inconsistencies in the data;

[0081] S1307. The fused data is integrated into an optimized storage solution;

[0082] S1308. The system provides data services and application interfaces, allowing different applications and services to access and utilize the integrated data.

[0083] Preferably, the intelligent data transmission optimization for the big data analysis method based on adaptive data management and privacy protection comprises the following steps:

[0084] S1501, network shrinkage status monitoring real-time measurement of network shrinkage bandwidth B, monitoring network delay L, where D is the amount of data transferred, T is the transmission time, and RTT is the round-trip time;

[0085] S1502, perform data transmission scheduling, algorithm R = f (B, L, P), where function f considers high bandwidth and low latency Increase the transmission rate and provide more bandwidth resources for high-priority data;

[0086] S1503, the transport protocol optimizer selects TCP or UDP according to the network environment; for TCP, the congestion window W is adjusted based on the network congestion situation, using the formula W=min(W max , W ssthresh +α·(W max -W ssthresh )) to gradually increase the window size, where α is a factor adjusted according to network feedback;

[0087] S1504, perform data transmission and calculate the amount of data successfully transmitted per unit time, where V is the amount of data transmitted, T elapsed It is the time that has passed;

[0088] S1505: After the transmission is completed, the system evaluates the transmission efficiency E and reliability Rel; the efficiency E is Calculate and measure the ratio of actual throughput to network bandwidth; Rel reliability is Rel = 1-P loss The calculation reflects the loss of data packets during transmission;

[0089] S1506. The system feeds back the evaluation result of the transmission effect to the scheduler and optimizer, and uses the self-learning algorithm to update the transmission strategy.

[0090] The beneficial effects of the present invention are:

[0091] This solution can provide an efficient, secure, intuitive and automated data analysis solution to meet the needs of data processing and analysis in the big data era. The technical problems solved mainly include:

[0092] 1. Improve data storage efficiency: Solve the efficiency problems of traditional storage solutions under the dynamic changes of data diversity and access patterns, reduce storage resource waste, and improve access speed.

[0093] 2. Improve data transmission efficiency: Overcome the bandwidth limitations and delays of existing networks and storage systems during data transmission, and ensure the real-time and overall performance of data processing.

[0094] 3. Strengthen data privacy protection: respond to the risks of personal privacy leakage and data abuse, adopt effective technologies to protect personal privacy, and meet the privacy protection needs during data collection and analysis.

[0095] 4. Enhance real-time data processing capabilities: Improve the system's ability to handle high-speed inflow of real-time data to ensure that it can meet the market's demand for instant analysis and response.

[0096] 5. Improve data visualization technology: Provide richer and more interactive data visualization tools to help users understand complex data sets more intuitively and enhance data insights

[0097] 6. Automate risk management strategies: Reduce reliance on expert experience and intuition in high-risk areas such as financial markets, respond quickly to market changes through automated methods, and improve decision-making efficiency.

[0098] 7. Optimize parallel processing of large-scale data: Improve the efficiency of resource allocation and task scheduling, give full play to the performance of multi-core processors and distributed computing clusters, and process large-scale data sets.

[0099] 8. Automated data preprocessing process: Solve data quality issues such as missing values, outliers and data standardization, and improve the speed and quality of data preprocessing.

[0100] 9. Fusion of multi-source heterogeneous data: Integrate data from different sources and formats, resolve data redundancy and conflict issues, and form a unified, accurate and reliable data view.

[0101] The core innovation of the present invention lies in its adaptive data storage architecture and dynamic data compression and decompression mechanism. The adaptive data storage architecture analyzes data characteristics and access patterns in real time, intelligently selects the optimal storage format and indexing strategy, and significantly improves data storage efficiency and query performance. At the same time, the dynamic data compression and decompression mechanism can automatically adjust the compression algorithm and parameters according to the real-time access pattern and content characteristics of the data, so as to achieve the optimization of data storage space and the balance of access speed. These two innovations not only solve the efficiency problems in big data storage and access, but also provide a solid foundation for efficient data processing and analysis.

[0102] The present invention achieves a significant improvement in data storage and access efficiency through its innovative adaptive data storage architecture and dynamic data compression and decompression mechanism, while ensuring the flexibility and real-time performance of data compression. The beneficial effects brought about by this technical solution include reducing storage costs, increasing data retrieval speed, optimizing network bandwidth usage, enhancing data privacy protection, improving real-time data processing capabilities, improving user experience through advanced data visualization, and automated risk management and strategy generation, thereby providing users with an efficient, secure, intuitive and powerful big data analysis tool. In addition, through automated data preprocessing and multi-source heterogeneous data fusion technology, the system further improves the quality of data processing and the accuracy of analysis, meeting the complex needs of data processing and analysis in the big data era. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0104] Figure 1 It is a process step diagram of the present invention;

[0105] Figure 2 It is a simplified system block diagram of the present invention. DETAILED DESCRIPTION

[0106] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0107] The present invention discloses a big data analysis method based on adaptive data management and privacy protection. Figure 1 , Figure 2 As shown, the embodiment includes the following steps:

[0108] S1, the system starts and configures the data acquisition module, preparing to receive data streams from multiple specified data sources in real time;

[0109] S2, data quality assessor automatically detects the completeness, consistency and accuracy of the data set, and identifies missing values ​​and outliers;

[0110] S3, data flows into the data preprocessing and cleaning module for preliminary data cleaning, formatting and standardization;

[0111] S4, the cleaned data is stored in the optimal storage structure according to the decision of the adaptive data storage architecture;

[0112] S5, the data stream is compressed through dynamic data compression and decompression mechanism to optimize storage space and access speed;

[0113] S6. The privacy protection framework automatically deploys differential privacy and data desensitization technologies to ensure that personal privacy in the data is protected;

[0114] S7, real-time data processing engine receives and processes high-speed data streams, and uses complex event processing technology for real-time analysis;

[0115] S8, Historical Data Analyzer performs in-depth analysis of accumulated data to identify long-term trends and patterns;

[0116] S9. Risk assessment models quantitatively assess the risk level of assets and investment portfolios based on historical and market information;

[0117] S10, the strategy recommendation engine integrates risk assessment and user preferences to generate personalized risk management strategies;

[0118] S11. The analysis results are presented in the form of graphs or charts through an advanced data visualization platform to enhance the user's understanding of the data;

[0119] S12, large-scale data parallel processing framework efficiently schedules and executes parallel tasks to process large-scale data sets;

[0120] S13. Multi-source heterogeneous data fusion technology integrates data from different sources and formats to form a comprehensive and consistent data view;

[0121] S14. Users obtain analysis reports through a customized report generator, and the system is continuously optimized based on user feedback.

[0122] Preferably, the automated data preprocessing and cleaning module in S3 is used to improve data quality and analysis efficiency: to ensure high data quality, thereby improving the accuracy and efficiency of analysis. The automated data preprocessing and cleaning module performs preliminary data cleaning, formatting and standardization processing, including the following steps:

[0123] S301. The automated data cleaner intervenes based on the evaluation results and applies intelligent algorithms to handle each problem in the data set in a targeted manner; for example, interpolation technology is used to fill missing values, clustering algorithms are used to identify and process outliers, and data scales are adjusted through standardization methods to ensure data consistency and comparability.

[0124] S302, the data preprocessor performs further data conversion and feature engineering, which may include data encoding, feature construction, dimensionality reduction and other operations, so as to prepare a clean, standardized and informative high-quality data set for data analysis. The entire process is automated, greatly reducing the need for manual intervention and improving the speed and quality of data processing.

[0125] The storage decision of the adaptive data storage architecture in S4 includes the following steps:

[0126] S401, Data Characteristics Analysis: In the initial stage, the system starts the data characteristics analyzer. This advanced algorithm component is responsible for analyzing the detailed attributes of the incoming data, including data dimension, type, scale, and access frequency. The analyzer uses statistical methods and machine learning techniques to deeply mine data patterns, identify data trends over time, and provide detailed data characteristics reports for subsequent storage decisions.

[0127] S402, storage format decision: Next, the storage format decision engine receives the report from the data feature analyzer. The engine uses intelligent algorithms to dynamically determine the storage format of the data based on the analysis results and the preset storage optimization strategy. For example, for data analysis workloads that frequently perform aggregation operations, the column storage format is selected to improve query efficiency; for workloads that require fast transaction processing, the row storage format may be selected to optimize read and write speeds.

[0128] S403, Index structure update: After the data storage format is determined, the dynamic index builder is started immediately. As an automated tool, it updates the index structure in real time based on the access pattern provided by the data feature analyzer and the decision of the storage format decision engine. This process ensures that the physical layout of the data storage is synchronized with the index logic to adapt to changes in data access patterns, optimize query paths, and reduce query latency.

[0129] S404, Continuous Performance Monitoring and Optimization: The adaptive data storage architecture also includes a continuous performance monitoring component that continuously evaluates storage and query performance. Based on these evaluation results, the system automatically triggers adjustments, such as rebalancing data distribution, optimizing index configuration, or adjusting storage policies to cope with the continuous evolution of data access patterns.

[0130] S405, Feedback loop integration: Finally, the system feeds back user access behavior and system performance to the data feature analyzer to form a closed-loop optimization process. In this way, the adaptive data storage architecture can continuously learn and adapt, predict future trends, and automatically adjust to keep the storage solution optimized.

[0131] The compression process of the dynamic data compression and decompression mechanism in S5 includes the following steps:

[0132] S501, Real-time data monitoring: First, the system starts the real-time data monitor, which is a continuously running component responsible for tracking and recording the access pattern and usage of data streams. It monitors the access frequency, access time, and usage pattern of data, collects this key information, and provides the necessary real-time data for subsequent compression strategies.

[0133] S502, Compression Strategy Evaluation: As the real-time data monitor collects data, the compression strategy evaluator starts working. This component analyzes the monitored data, evaluates the characteristics and access patterns of the data, and determines the most appropriate compression algorithm and parameters. It uses advanced algorithms to evaluate the effects of different compression techniques, such as dictionary-based compression techniques or transform-based compression techniques, and selects the optimal compression strategy to balance compression rate and access performance.

[0134] S503, compression algorithm selection: After evaluating the optimal compression strategy, the compression strategy evaluator selects a specific compression algorithm based on data characteristics and access patterns, as well as expected compression effects and computing resource consumption.

[0135] S504, Dynamic Compression Execution: Next, the dynamic compression executor automatically performs compression strategy adjustments based on the selection of the evaluator. This includes setting the compression level, adjusting algorithm parameters, and switching compression algorithms when necessary. The executor is responsible for applying the selected compression algorithm to the data, performing the actual compression operation, and ensuring that the compression process does not affect the availability and access speed of the data.

[0136] S505, compression effect monitoring: After data compression, the system again uses the real-time data monitor to monitor the compression effect, including evaluating the access delay of compressed data, storage space saving and decompression performance.

[0137] S506, Feedback loop and optimization: Finally, the compression effect obtained by monitoring is fed back to the compression strategy evaluator, forming a closed-loop optimization process. The evaluator adjusts the compression strategy based on the feedback, and the executor adjusts the compression operation again to ensure that the entire compression and decompression mechanism always adapts to the current data characteristics and access requirements.

[0138] The privacy protection framework in S6 automatically deploys differential privacy and data desensitization technologies to ensure data security and user privacy: Data security and user privacy are ensured through a series of carefully designed steps, including the following steps:

[0139] S601, Privacy Policy Definer dynamically updates privacy rules and policies based on global data protection regulations, providing compliance guidance for the system;

[0140] S602. The data desensitization processor processes sensitive information using data masking, generalization, and perturbation techniques to convert personal information into a format that cannot be traced while maintaining the statistical properties of the data;

[0141] S603, the differential privacy executor introduces an appropriate amount of randomness in the data analysis phase, and protects individual data from being accurately traced by adding carefully calculated noise, thus balancing privacy protection and data analysis accuracy;

[0142] S604. The system evaluates and adjusts privacy protection measures through continuous monitoring and feedback mechanisms to respond to regulatory changes and technological evolution, ensuring the continued effectiveness and adaptability of the privacy protection framework. The entire process forms a closed loop, which not only ensures the security of data processing, but also protects the privacy rights of users.

[0143] The real-time data processing engine in S7 processes and analyzes high-speed data streams including the following steps:

[0144] S701. First, as the first link of the system, the data stream receiver is responsible for efficiently receiving real-time data streams from multiple sources. The receiver is designed with high concurrent processing capabilities and can process thousands or even tens of thousands of data input channels at the same time, ensuring no data loss under high throughput. It uses advanced buffering technology and load balancing algorithms to optimize the data receiving process and reduce delays.

[0145] S702, Stream Processing Analysis: The received data stream is then sent to the stream processing analyzer. The analyzer uses complex event processing technology CEP and pattern matching algorithms to perform real-time screening, association and aggregation operations on the data stream. By setting up a rule engine, the stream processing analyzer can identify key events and abnormal patterns, perform real-time statistical analysis and trend prediction, and provide a basis for decision-making.

[0146] S703, Real-time analysis result generation: After the stream processing analyzer completes data processing, the real-time result outputter immediately outputs the analysis results in a low-latency manner. This component uses fast data push technology, such as WebSocket or MQTT protocol, to ensure that the analysis results can be quickly delivered to the front-end monitoring system or decision support tool. The output results include but are not limited to real-time statistics, abnormal alarms, predicted trends, etc., supporting users to monitor in real time and respond quickly.

[0147] S704, Performance Optimization and Adjustment: The real-time data processing engine also includes a performance monitoring module, which continuously tracks the operating status of the processing engine, including indicators such as processing latency, system load, and event processing rate. Based on these monitoring data, the engine automatically adjusts resource allocation and processing strategies, such as adding processing threads, optimizing query algorithms, or adjusting buffer sizes to cope with different data traffic and complexity.

[0148] S705, Scalability and fault tolerance design: In order to cope with the ever-changing data traffic and processing requirements, the real-time data processing engine adopts microservice architecture and containerized deployment to ensure the scalability and fault tolerance of the system. When a component fails or data traffic increases suddenly, the engine can automatically expand processing capacity or reallocate tasks to ensure the continuity and stability of data processing.

[0149] S706, Security and Privacy Protection: In the process of processing data streams, the real-time data processing engine also integrates security and privacy protection measures to ensure the security of data during transmission and processing. This includes data encryption transmission, access control and data desensitization to prevent sensitive information leakage.

[0150] The risk assessment model in S9 assesses the risk level of assets and investment portfolios by including the following steps:

[0151] S901, Market Trend Analysis: The system first collects and analyzes market data through the market trend analyzer, and uses machine learning algorithms to identify market trends and patterns. For example, a classification model is trained using historical price data to identify market up or down trends.

[0152] S902, Risk Assessment: Next, the risk assessment model uses historical data to assess the risk level of the asset. This process involves calculating the standard deviation of the asset's return as a measure of risk. We analyze past return series to determine their volatility and then assess risk.

[0153] S903, User Preference Analysis: In the user preference analysis step, the system will consider the user's risk preference and investment goals. This does not require complex formulas, but collects user input preference information through the user interface, such as risk tolerance level or expected return.

[0154] S904, Strategy Generation: The strategy recommendation engine integrates market trends, risk assessment and user preferences to generate personalized risk management strategies. For example, if the user prefers low risk, the system may recommend an asset allocation strategy that focuses on stable assets such as bonds.

[0155] S905, Strategy backtesting and optimization: The generated strategy will be backtested through historical data to evaluate its performance under different market conditions. This process does not require a specific formula, but is carried out by simulating the performance of the strategy on historical data.

[0156] S906. Real-time monitoring and adjustment: The real-time monitoring module continuously tracks market dynamics and portfolio performance. When market conditions change, the system adjusts strategies according to preset rules. For example, if the volatility of an asset class exceeds the user's risk tolerance threshold, the system will automatically reduce the position of that asset.

[0157] S907, User Feedback and Iteration: Finally, the system will collect user feedback on the strategy and perform iterative optimization based on it. This process involves fine-tuning the strategy generation algorithm to better meet user needs, but does not require complex mathematical formulas.

[0158] Through these steps, the automated risk management strategy generator is able to provide each user with a tailored risk management strategy while ensuring that the strategy can adapt to the ever-changing market. The implementation of these steps ensures the transparency and explainability of the strategy generation process, allowing users to fully understand the logic behind the strategy.

[0159] The advanced data visualization platform in the S11 enables users to intuitively understand and interact with data: It is designed to enable users to understand and operate data in an intuitive and interactive way. The platform first uses a multidimensional data displayer, which uses advanced graphics rendering technology to display the time series changes, geographical distribution characteristics and hierarchical relationships of data, allowing users to observe and analyze data from different angles. Then, the interactive chart tool provides users with a wealth of chart options, such as line charts, bar charts, pie charts, scatter charts and heat maps, as well as a variety of interactive functions, including dragging and dropping to adjust chart elements, zooming to view data details and filtering to select specific data sets. These functions work together to enhance the flexibility and depth of user exploration of data. The personalized view generator is one of the core components of the platform. It automatically or manually generates customized data views and dashboards by intelligently identifying the user's role, preferences and needs. These views and dashboards not only display key performance indicators KPIs and data analysis results, but also allow users to customize the display and layout of information according to their own focus. For example, a financial analyst may get a dashboard that highlights financial ratios and trends, while a marketing expert may get a view that focuses on customer behavior and market segmentation. Through the combination of these features, the advanced data visualization platform not only improves the intuitiveness of data presentation, but also greatly enhances the user experience of interacting with data, enabling users to gain insights and make data-based decisions more quickly.

[0160] The large-scale data parallel processing framework in S12 is a high-performance system designed to fully utilize modern computing resources to process large-scale data sets. The large-scale data parallel processing framework processes large-scale data sets including the following steps:

[0161] S1201, Task Splitting: In the initial stage, the task splitter intervenes, and the intelligent algorithm decomposes the large-scale data processing task into multiple subtasks that can be executed in parallel. The purpose of this step is to split a large task into multiple small tasks so that they can be processed simultaneously in a parallel computing environment to improve processing efficiency.

[0162] S1202, subtask allocation: After segmentation, the subtasks are allocated to available processing units. This step does not require complex formulas, but relies on the algorithm designed by the task segmenter to ensure that each subtask can be executed independently on different processors or computing nodes.

[0163] S1203, Parallel computing scheduling: The parallel computing scheduler is responsible for efficiently scheduling these subtasks to various nodes on the multi-core processor or distributed computing cluster. The scheduler uses a resource allocation algorithm to consider the current system load, task priority and processing power to intelligently arrange task execution.

[0164] S1204, executing parallel tasks: The subtasks are executed in parallel on the allocated computing resources. Each computing node independently processes the data subset allocated to it, which may involve steps such as data loading, computing, and storing intermediate results.

[0165] S1205, result collection and preliminary integration: Once the subtasks are completed, the results need to be collected from each computing node. This step involves network communication and data transmission, and the integrity and reliability of the data during transmission need to be ensured.

[0166] S1206, result merging and optimization: The result merger receives the intermediate results from each node and integrates them into the final output. This process may involve operations such as sorting, merging and deduplication of data to ensure the consistency and accuracy of the final result.

[0167] S1207, data consistency check: After the results are merged, the system performs a consistency check to ensure that the results of all subtasks are correctly integrated without data loss or errors.

[0168] S1208, Optimization and Error Handling: If any inconsistency or error is found, the result merger will perform optimization steps and re-execute the problematic subtasks if necessary to ensure the quality of the final result.

[0169] S1209, output final results: Finally, the verified and optimized final results are output for further analysis or application.

[0170] Through this coherent process, the massively data-parallel processing framework can efficiently process large-scale data sets while ensuring the accuracy of the processing process and the full utilization of computing resources.

[0171] Intelligent data transmission optimization based on big data analysis methods with adaptive data management and privacy protection to improve data transmission efficiency in complex network environments: including the following steps:

[0172] S1501, network status detection: network shrinkage status monitoring real-time measurement of network shrinkage bandwidth B, using the formula Where D is the amount of data transferred and T is the transfer time. This provides an indication of the network's current transfer capacity. To monitor network latency L, calculate the round-trip time RTT using It reports the round trip delay of a packet from when it is sent to when it is received.

[0173] S1502, perform data transmission scheduling, algorithm R = f (B, L, P), where function f considers high bandwidth and low latency Increase the transmission rate and provide more bandwidth resources for high-priority data;

[0174] S1503, the transport protocol optimizer selects TCP or UDP according to the network environment; for TCP, the congestion window W is adjusted based on the network congestion situation, using the formula W=min(W max , W ssthresh +α·(W max -W ssthresh )) to gradually increase the window size, where α is a factor adjusted according to network feedback;

[0175] S1504, perform data transmission and calculate the amount of data successfully transmitted per unit time, where V is the amount of data transmitted, T elapsed It is the time that has passed;

[0176] S1505: After the transmission is completed, the system evaluates the transmission efficiency E and reliability Rel; the efficiency E is Calculate and measure the ratio of actual throughput to network bandwidth; Rel reliability is Rel = 1-P loss The calculation reflects the loss of data packets during transmission;

[0177] S1506. The system feeds back the evaluation results of the transmission effect to the scheduler and optimizer, and uses the self-learning algorithm to update the transmission strategy, such as strategy = strategy + β (evaluation result - expected target). Here β is the learning rate, which adjusts the strategy according to the gap between the evaluation result and the expected target to gradually optimize the transmission performance.

[0178] Preferably, the multi-source heterogeneous data fusion technology in S13 integrates data from different sources and formats to form a comprehensive and consistent data view, including the following steps:

[0179] S1301. Data source identification: The data source identifier automatically identifies different types of data sources connected to the system. It distinguishes between structured data, unstructured data, or semi-structured data by analyzing the structure, format, and metadata of the data, ensuring that each type of data can be processed and converted correctly.

[0180] S1302, data classification and labeling: After identifying the data source, the system classifies and labels the data. This step helps the system understand the intrinsic characteristics of the data and the applicable processing flow by assigning labels and attributes to the data, thereby preparing for further processing and fusion of the data.

[0181] S1303, data format conversion: The data format converter converts the data in heterogeneous data sources into a unified internal format. For example, time series data is standardized into a unified timestamp format, and text data from different sources is converted into a unified encoding and separation format to facilitate subsequent integration and analysis.

[0182] S1304, Data cleaning and standardization: While converting data, the system performs data cleaning and standardization operations to remove invalid or erroneous data records, correct inconsistent data items, and unify measurements and data formats to improve data quality and ensure data accuracy and reliability.

[0183] S1305, Data Fusion: The data fusion engine uses advanced algorithms to integrate the converted and cleaned data. This step builds a comprehensive and consistent data view by identifying the same entities in different data sets, merging duplicate information, and solving data redundancy problems.

[0184] S1306, consistency check and conflict resolution: During the data integration process, the system performs consistency checks to identify conflicts and inconsistencies in the data. By applying preset rules or intelligent algorithms, the system selects the most reliable data source or performs data reconciliation to ensure the consistency and accuracy of the final data set.

[0185] S1307, Data Integration and Optimized Storage: The fused data is integrated into an optimized storage solution, which may be a relational database, NoSQL database or data warehouse, selected based on the characteristics of the data and query requirements to support efficient data access and analysis.

[0186] S1308, Data Services and Application Interfaces: Finally, the system provides data services and application interfaces, allowing different applications and services to access and utilize the fused data. Through APIs or other integrated tools, users and systems can easily access data to support various data analysis and business applications.

[0187] Through these steps, multi-source heterogeneous data fusion technology effectively integrates data from different sources and formats, providing a unified, accurate and reliable data view, laying a solid foundation for data analysis, decision support and business insights.

[0188] Through these technical solutions, the intelligent big data analysis method of the present invention can provide an efficient, safe, intuitive and automated data analysis solution to meet the needs of data processing and analysis in the big data era.

[0189] In the specific implementation:

[0190] The system in S1 realizes real-time data reception by configuring a specific data source and collection frequency, such as setting the stock market API to collect data every 5 seconds. This step ensures that the data collection module can obtain the latest data stream in a timely manner, laying the foundation for subsequent processing.

[0191] The system in S2 automatically evaluates the quality of the data set by setting integrity check standards, such as marking any field with more than 10% missing values ​​as low-quality data, to ensure the accuracy and reliability of the data.

[0192] The data preprocessing module in S3 uses predefined cleaning rules, such as using regular expressions to remove HTML tags in text data and using column means to fill missing values ​​in numerical data, to ensure the cleanliness and consistency of the data set.

[0193] The data in S4 is adaptively allocated according to its type and access mode, for example, transactional data is stored in rows to optimize read and write efficiency, and analytical data is stored in columns to improve query performance.

[0194] The system in S5 applies the Deflate compression algorithm to historical data that is not frequently accessed, and sets the compression level to 9 to achieve the most efficient storage savings while ensuring the fast decompression requirements of the data.

[0195] The differential privacy technology in S6 is applied to the data analysis process, and the privacy budget ε is set to 0.1. The user data is protected by adding Laplace noise to ensure the privacy security of the analysis results.

[0196] The real-time data processing engine in the S7 adopts an event-driven model to monitor stock price changes. Once the price change exceeds the preset 1% threshold, the system triggers an alarm to achieve a rapid response to market dynamics.

[0197] The system in S8 uses the ARIMA model to fit the time series data, and by setting the parameters of p=1, d=1, q=1, it analyzes the long-term trend and periodicity of the historical data to provide a basis for investment decisions.

[0198] The VaR model is used in S9 to calculate the risk value of the investment portfolio, setting a 99% confidence level to assess the maximum potential loss of the investment portfolio under extreme market conditions.

[0199] In S10, according to the conservative risk preference selected by the user, the system recommends an asset allocation strategy in which bond assets account for 60% and stock assets account for 40% to reduce the overall volatility of the investment portfolio.

[0200] The data visualization platform in S11 displays a line chart of stock prices and trading volumes, with the time range set to the past year. Different data series are clearly distinguished by colors and labels, allowing users to intuitively understand market trends.

[0201] The parallel processing framework in S12 divides the data set into 1024 data blocks and distributes them to available computing nodes. It efficiently processes large-scale data sets through the MapReduce model, significantly improving the data processing speed.

[0202] In S13, a graph-based data fusion algorithm is used to integrate data from the MySQL database and the Twitter API, and predefined entity correspondences and data conversion rules are used to solve data redundancy and conflict problems and form a unified data view.

[0203] The system in S14 generates a customized analysis report including a summary of key indicators and charts according to the report template selected by the user, collects user feedback through online surveys, adjusts the report content and system functions according to the feedback, and improves user satisfaction.

[0204] Through these specific step descriptions, the entire process of the intelligent big data analysis method from data collection to report generation is explained in detail. Each step has clear parameter settings and expected results, ensuring the efficient operation and continuous optimization of the entire system.

[0205] It should be noted that the parts not described in detail in the above embodiments are all prior art.

[0206] The above description is only a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention should be covered within the protection scope of the present invention.

Claims

1. A big data analysis method based on adaptive data management and privacy protection, characterized by: The following steps are involved: S1, the system starts and configures the data acquisition module, preparing to receive data streams from multiple specified data sources in real time; S2, data quality assessor automatically detects the completeness, consistency and accuracy of the data set, and identifies missing values ​​and outliers; S3, data flows into the data preprocessing and cleaning module for preliminary data cleaning, formatting and standardization; S4, the cleaned data is stored in the optimal storage structure according to the decision of the adaptive data storage architecture; S5, the data stream is compressed through dynamic data compression and decompression mechanism to optimize storage space and access speed; S6. The privacy protection framework automatically deploys differential privacy and data desensitization technologies to ensure that personal privacy in the data is protected; S7, real-time data processing engine receives and processes high-speed data streams and uses complex event processing technology for real-time analysis; S8, Historical Data Analyzer performs in-depth analysis of accumulated data to identify long-term trends and patterns; S9. Risk assessment models quantitatively assess the risk level of assets and investment portfolios based on historical and market information; S10, the strategy recommendation engine integrates risk assessment and user preferences to generate personalized risk management strategies; S11. The analysis results are presented in the form of graphs or charts through an advanced data visualization platform to enhance the user’s understanding of the data; S12, large-scale data parallel processing framework efficiently schedules and executes parallel tasks to process large-scale data sets; S13. Multi-source heterogeneous data fusion technology integrates data from different sources and formats to form a comprehensive and consistent data view; S14. Users obtain analysis reports through a customized report generator, and the system is continuously optimized based on user feedback.

2. The big data analysis method based on adaptive data management and privacy protection according to claim 1, characterized in that: The automated data preprocessing and cleaning module in S3 performs preliminary data cleaning, formatting and standardization processing, including the following steps: S301, the automated data cleaner intervenes according to the evaluation results and applies intelligent algorithms to deal with each problem in the data set in a targeted manner; S302, the data preprocessor performs further data conversion and feature engineering, including data encoding, feature construction, and dimensionality reduction operations.

3. The big data analysis method based on adaptive data management and privacy protection according to claim 1, characterized in that: The storage decision of the adaptive data storage architecture in S4 includes the following steps: S401, the system starts a data characteristic analyzer to analyze the detailed attributes of the incoming data. The analyzer uses statistical methods and machine learning techniques to deeply mine the data patterns and identify the changing trends of the data over time; S402, the storage format decision engine receives the report from the data characteristic analyzer, and dynamically determines the storage format of the data according to the analysis result and the preset storage optimization strategy; S403, start the dynamic index builder, and update the index structure in real time according to the access mode provided by the data characteristic analyzer and the decision of the storage format decision engine; S404, the continuous performance monitoring component of the adaptive data storage architecture continuously evaluates storage and query performance, and based on these evaluation results, the system automatically triggers adjustments to cope with the continuous evolution of data access patterns; S405: The system feeds back the user access behavior and system performance to the data characteristic analyzer, forming a closed-loop optimization process.

4. The big data analysis method based on adaptive data management and privacy protection according to claim 1, characterized in that: The compression process of the dynamic data compression and decompression mechanism in S5 includes the following steps: S501, the system starts a real-time data monitor to track and record the access mode and usage of the data stream, and monitor the access frequency, access time, and usage mode of the data; S502, as the real-time data monitor collects data, the compression strategy evaluator analyzes the monitored data and evaluates the characteristics and access patterns of the data to determine the most appropriate compression algorithm and parameters; S503, after evaluating the optimal compression strategy, the compression strategy evaluator selects a specific compression algorithm based on data characteristics and access patterns, as well as expected compression effects and computing resource consumption; S504, the dynamic compression executor automatically performs compression strategy adjustment according to the selection of the evaluator; S505, after the data is compressed, the system again uses the real-time data monitor to monitor the compression effect; S506. The compression effect obtained through monitoring is fed back to the compression strategy evaluator, forming a closed-loop optimization process. The evaluator adjusts the compression strategy based on the feedback, and the executor adjusts the compression operation again to ensure that the entire compression and decompression mechanism always adapts to the current data characteristics and access requirements.

5. The big data analysis method based on adaptive data management and privacy protection according to claim 1, characterized in that: The process of automatically deploying differential privacy and data desensitization technology in the privacy protection framework in S6 includes the following steps: S601, Privacy Policy Definer dynamically updates privacy rules and policies based on global data protection regulations, providing compliance guidance for the system; S602. The data desensitization processor processes sensitive information using data masking, generalization, and perturbation techniques to convert personal information into a format that cannot be traced while maintaining the statistical properties of the data; S603, the differential privacy executor introduces an appropriate amount of randomness in the data analysis phase, and protects individual data from being accurately traced by adding carefully calculated noise, thus balancing privacy protection and data analysis accuracy; S604. The system evaluates and adjusts privacy protection measures through continuous monitoring and feedback mechanisms to respond to regulatory changes and technological evolution, ensuring the continued effectiveness and adaptability of the privacy protection framework.

6. The big data analysis method based on adaptive data management and privacy protection according to claim 1, characterized in that: The real-time data processing engine in S7 processes and analyzes high-speed data streams including the following steps: S701, data stream receiver efficiently receives real-time data streams from multiple sources. The data stream receiver is designed with high concurrent processing capabilities and can process thousands or even tens of thousands of data input channels at the same time, ensuring no data loss under high throughput. It uses advanced buffering technology and load balancing algorithms to optimize the data receiving process and reduce delays. S702, the received data stream is then sent to the stream processing analyzer, which uses complex event processing technology CEP and pattern matching algorithm to perform real-time screening, correlation and aggregation operations on the data stream; S703. After the stream processing analyzer completes data processing, the real-time result outputter immediately outputs the analysis results in a low-latency manner, using fast data push technology to ensure that the analysis results can be quickly delivered to the front-end monitoring system or decision support tool; S704, the performance monitoring module of the real-time data processing engine continuously tracks the running status of the processing engine. Based on these monitoring data, the engine automatically adjusts resource allocation and processing strategies to cope with different data flows and complexities; S705. In order to cope with the ever-changing data traffic and processing requirements, the real-time data processing engine adopts microservice architecture and containerized deployment to ensure the scalability and fault tolerance of the system; S706. In the process of processing data streams, the real-time data processing engine also integrates security and privacy protection measures to ensure the security of data during transmission and processing; this includes data encryption transmission, access control and data desensitization to prevent the leakage of sensitive information.

7. The big data analysis method based on adaptive data management and privacy protection according to claim 1, characterized in that: The risk assessment model in S9 assesses the risk level of assets and investment portfolios by including the following steps: S901, the system first collects and analyzes market data through a market trend analyzer, using machine learning algorithms to identify market trends and patterns; S902. Risk assessment models use historical data to assess the risk level of assets; S903, in the user preference analysis step, the system will consider the user's risk preference and investment objectives, and collect the preference information input by the user through the user interface; S904, the strategy recommendation engine integrates market trends, risk assessments and user preferences to generate personalized risk management strategies; S905. The generated strategy will be backtested using historical data to evaluate its performance under different market conditions; S906. The real-time monitoring module continuously tracks market dynamics and portfolio performance. When market conditions change, the system adjusts strategies according to preset rules. S907. The system collects user feedback on the strategy and performs iterative optimization accordingly.

8. The big data analysis method based on adaptive data management and privacy protection according to claim 1, characterized in that: The large-scale data parallel processing framework in S12 processes large-scale data sets including the following steps: S1201, the task splitter decomposes the large-scale data processing task into multiple subtasks that can be executed in parallel; S1202, after segmentation, the subtasks are assigned to available processing units; S1203, the parallel computing scheduler efficiently schedules these subtasks to each node on the multi-core processor or distributed computing cluster; S1204, the subtasks are executed in parallel on the allocated computing resources; S1205, the subtask is completed, and the results are collected from each computing node; S1206, the result merger receives the intermediate results from each node and integrates them into the final output; S1207. After the results are merged, the system performs a consistency check to ensure that the results of all subtasks are correctly integrated without data loss or errors; S1208. If any inconsistency or error is found, the result merger will perform optimization steps and re-execute the problematic subtasks if necessary to ensure the quality of the final result; S1209. The verified and optimized final results are output for further analysis or application.

9. The big data analysis method based on adaptive data management and privacy protection according to claim 1, characterized in that: The multi-source heterogeneous data fusion technology in S13 integrates data from different sources and formats to form a comprehensive and consistent data view, including the following steps: S1301, the data source identifier automatically executes to identify different data source types accessing the system; S1302: After identifying the data source, the system classifies and labels the data; S1303, the data format converter converts the data in the heterogeneous data sources into a unified internal format; S1304. While converting data, the system performs data cleaning and standardization operations, removes invalid or erroneous data records, corrects inconsistent data items, and unifies measurements and data formats; S1305, the data fusion engine integrates the converted and cleaned data; S1306. During the data fusion process, the system performs a consistency check to identify conflicts and inconsistencies in the data; S1307. The fused data is integrated into an optimized storage solution; S1308. The system provides data services and application interfaces, allowing different applications and services to access and utilize the integrated data.

10. The big data analysis method based on adaptive data management and privacy protection according to claim 1, characterized in that: Intelligent data transmission optimization for big data analysis methods based on adaptive data management and privacy protection includes the following steps: S1501, network shrinkage status monitoring real-time measurement of network shrinkage bandwidth B, monitoring network delay L, where D is the amount of data transferred, T is the transmission time, and RTT is the round-trip time; S1502, perform data transmission scheduling, algorithm R=f(B, L, P), where function f considers increasing the transmission rate when the bandwidth is high and the delay is low, and provides more bandwidth resources for high priority data; S1503, the transport protocol optimizer selects TCP or UDP according to the network environment; for TCP, the congestion window W is adjusted based on the network congestion situation, using the formula W=min(W max , W ssthresh +α·(W max -W ssthresh )) to gradually increase the window size, where α is a factor adjusted according to network feedback; S1504, perform data transmission and calculate the amount of data successfully transmitted per unit time, where V is the amount of data transmitted, T elapsed It is the time that has passed; S1505: After the transmission is completed, the system evaluates the transmission efficiency E and reliability Rel; the efficiency E is Calculate and measure the ratio of actual throughput to network bandwidth; Rel reliability is Rel = 1-P loss The calculation reflects the loss of data packets during transmission; S1506. The evaluation result of the system transmission effect is fed back to the scheduler and optimizer, and the transmission strategy is updated using a self-learning algorithm.

Images