Big data analysis platform system based on artificial intelligence

Abstract

The invention relates to the technical field of artificial intelligence and big data processing, and discloses an artificial intelligence-based big data analysis platform system, which comprises a data access layer, a data preprocessing layer, an AI analysis engine layer, a data storage layer, an application service layer and a monitoring operation and maintenance layer which are in communication connection in sequence. The data access layer is used for realizing multi-source heterogeneous data access and high-frequency data caching; the preprocessing layer completes data cleaning conversion and quality detection; the AI analysis engine layer realizes full-life-cycle management, intelligent analysis and result optimization and explanation of the model; the data storage layer adopts a mixed framework for classified storage and remote backup; the application service layer provides a visual display and open interface; the monitoring operation and maintenance layer guarantees stable operation of the system. According to the invention, data processing efficiency and analysis precision are improved, and data security and user experience are enhanced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and big data processing technology, and more specifically, to a big data analysis platform system based on artificial intelligence. Background Technology

[0002] With the rapid development of information technology, especially the widespread application of the Internet of Things, mobile internet, and cloud computing, data is experiencing explosive growth, and the types of data are becoming increasingly diverse, encompassing structured, semi-structured, and unstructured data. This massive, multi-source, and heterogeneous data contains rich business value. How to efficiently collect, store, process, and analyze this data has become a crucial problem that enterprises and research institutions urgently need to solve. Traditional big data processing platforms often suffer from limitations such as limited data access capabilities, inadequate preprocessing, simplistic analytical models lacking dynamic optimization mechanisms, low storage efficiency, and complex system maintenance, making it difficult to meet the demands of modern intelligent data analysis.

[0003] Meanwhile, the rapid development of artificial intelligence (AI) technology has provided new impetus for big data analytics. Through technologies such as machine learning and deep learning, deeper patterns and trends can be extracted from complex data, improving the accuracy and intelligence of the analysis. However, existing AI-powered big data analytics platforms still face many challenges in model management, result interpretation, and overall system collaboration. For example, models are difficult to update and customize flexibly, analysis results lack interpretability, and the system struggles to achieve efficient data flow and resource monitoring, limiting their widespread adoption and effectiveness in practical applications.

[0004] Therefore, there is an urgent need for an intelligent big data analytics platform system that can support efficient access and processing of multi-source heterogeneous data, possess a robust data quality assurance mechanism, integrate advanced artificial intelligence analysis models, and support dynamic optimization and interpretability. Simultaneously, this system should have an efficient data storage solution and sound monitoring and maintenance capabilities to ensure stable and reliable platform operation, meeting the diverse needs of enterprises in data-driven decision-making, business optimization, and intelligent services. Based on this, developing a well-structured, fully functional, and high-performance artificial intelligence-based big data analytics platform system has become an important direction for current technological development and an urgent requirement for practical applications. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, the present invention aims to overcome the deficiencies of existing big data analysis systems and provide an artificial intelligence-based big data analysis platform system that enables efficient access and caching of multi-source heterogeneous data, accurate data preprocessing, intelligent analysis and result optimization, classified and secure storage, and convenient visualization application services, while ensuring stable system operation through comprehensive monitoring and maintenance.

[0006] To achieve the aforementioned objective, the technical solution of this invention is implemented as follows: A big data analysis platform system based on artificial intelligence includes a data access layer, a data preprocessing layer, an AI analysis engine layer, a data storage layer, and an application service layer, which are sequentially connected in communication; it also includes a monitoring and maintenance layer that is connected in communication with the above layers. The specific structure and function of each layer are as follows: The data access layer is used to access multi-source heterogeneous data, including but not limited to structured data (such as relational database data), semi-structured data (such as XML and JSON data), and unstructured data (such as text, images, and audio data). To improve the processing efficiency of real-time high-frequency data, the data access layer is also equipped with a data caching module, which is used to temporarily cache the real-time accessed high-frequency data, reduce data transmission latency, and ensure the efficient operation of subsequent processing stages.

[0007] The data preprocessing layer cleanses, transforms, integrates, and reduces the raw data received from the data access layer to obtain standardized data for analysis. Specifically, the data preprocessing layer includes a data transformation module and a data quality detection module. The data transformation module standardizes data from different formats, converting heterogeneous data from multiple sources into a format that facilitates subsequent analysis and processing. The data quality detection module detects and corrects missing, outlier, and duplicate values ​​in the raw data. For example, it corrects missing values ​​using mean imputation and median imputation, identifies and removes or corrects outliers using deviation analysis, and removes duplicate values ​​using hash comparison, thereby ensuring the accuracy and reliability of the preprocessed data.

[0008] The AI ​​analytics engine layer is used to invoke analytical models to perform intelligent analysis on standardized data and optimize the results. This layer specifically includes a model management submodule, an intelligent analysis submodule, and a result optimization submodule. The model management submodule is used to register, deploy, update, and deregister analysis models. It also includes a model training submodule, which can train or fine-tune basic models (such as neural network models, decision tree models, regression analysis models, etc.) based on user-provided labeled datasets to generate customized analysis models that meet specific user needs. The intelligent analysis submodule is used to call the target analysis model in the model management submodule to perform feature extraction, trend prediction and correlation analysis on the standardized data to be analyzed, and to explore the potential value behind the data. The results optimization submodule is used to verify the accuracy of the analysis results from the intelligent analysis submodule. It presets an accuracy threshold, and when the accuracy of the analysis results does not reach the threshold, it iteratively optimizes the parameters of the target analysis model until the accuracy reaches the target. In addition, this submodule also includes an analysis results interpretation submodule, which uses interpretable AI algorithms (such as LIME algorithm and SHAP algorithm) to provide a visual explanation of the generation process and key influencing factors of the analysis results, thereby improving the credibility and understandability of the analysis results.

[0009] The data storage layer is used to classify and store raw data, preprocessed data, and analysis results. It employs a hybrid storage architecture, including relational databases, distributed file systems, and time-series databases. The relational database stores structured data, the distributed file system stores unstructured data, and the time-series database stores time-series analysis data (such as sensor data and log data collected in time series). This collaborative approach of different storage media improves the efficiency and relevance of data storage. Simultaneously, the data storage layer includes a data backup submodule, which uses a combination of incremental and full backups to perform regular backups of the stored data, such as daily incremental backups and weekly full backups. The backup data is stored on off-site backup nodes to further ensure data security and recoverability.

[0010] The application service layer provides users with interfaces for visual data display and analysis services. The visual data display interface supports the generation and style customization of various data visualization charts, including bar charts, line charts, pie charts, heatmaps, and radar charts. Users can select chart types and adjust chart styles (such as colors, fonts, and axis settings) according to their needs. The analysis service interface allows users to call the platform's analysis functions via API, enabling secondary development or integration with other business systems.

[0011] The monitoring and maintenance layer establishes communication connections with the data access layer, data preprocessing layer, AI analysis engine layer, data storage layer, and application service layer. This is used to monitor the real-time operating status of each module (such as whether it is running normally or whether there are any faults), resource utilization (such as CPU utilization, memory utilization, and disk utilization), and task execution progress (such as data access progress, preprocessing progress, and analysis task progress). Thresholds for each monitoring indicator are preset. When an indicator is detected to exceed the threshold or a module is found to be operating abnormally, an alarm mechanism is triggered (such as SMS alarms, email alarms, and platform pop-up alarms) to facilitate timely handling by maintenance personnel and ensure the stable and reliable operation of the system.

[0012] The beneficial effects of this invention are reflected in: The artificial intelligence-based big data analytics platform system proposed in this invention, through a scientifically designed architecture and modular division, achieves efficient access and processing of multi-source heterogeneous data, significantly improving the flexibility of data processing and the system's scalability. The data access layer supports multiple data formats and access methods, and is equipped with a data caching module, effectively alleviating the transmission pressure of real-time, high-frequency data and ensuring data continuity and real-time performance. The data preprocessing layer employs standardized transformation and a rigorous data quality detection mechanism to ensure the accuracy and consistency of the data required for subsequent analysis, greatly enhancing the credibility of the data and the reliability of the analysis results.

[0013] The AI ​​analytics engine layer integrates model management, intelligent analysis, and result optimization functions. It not only supports model management and training for various mainstream deep learning and machine learning frameworks, but also improves the predictive accuracy and transparency of the analysis models through iterative optimization mechanisms and interpretable AI algorithms, enhancing user understanding and trust in the results. The data storage layer adopts a hybrid storage architecture, employing the most suitable storage solution for different types of data, effectively improving storage efficiency and data access speed. Simultaneously, combined with incremental and full backup strategies and off-site backup measures, it greatly ensures data security and recoverability.

[0014] The application service layer provides rich visualizations and flexible service interfaces to meet diverse data analysis and secondary development needs, improving the system's usability and applicability. The monitoring and maintenance layer enables real-time monitoring and alerting of the entire system's operational status, resource usage, and task execution progress, ensuring stable platform operation and timely response to potential faults, thus reducing maintenance difficulty and system risks.

[0015] In summary, this invention not only achieves multi-level and multi-functional collaborative work in terms of technology, giving full play to the advantages of artificial intelligence and big data technologies, but also shows significant improvements in system stability, data security, analysis accuracy, and user experience. It has good practical value and broad application prospects, providing enterprises and organizations with an efficient, reliable, and intelligent big data analysis solution. Attached Figure Description

[0016] In the attached diagram: Figure 1 This is a diagram of the overall system architecture (top-level structure) of the present invention. Figure 2 This is a detailed diagram of the internal structure of the data access layer of the present invention; Figure 3 This is a detailed diagram of the internal structure of the data preprocessing layer of the present invention; Figure 4 This is a detailed diagram of the internal structure of the AI ​​analysis engine layer of the present invention; Figure 5 This is a detailed diagram of the internal structure of the data storage layer of the present invention; Figure 6 This is a detailed diagram of the functional structure of the application service layer of the present invention; Figure 7 This is a detailed diagram of the monitoring and maintenance layer monitoring scope of the present invention; Explanation of reference numerals in the attached figures: Data access layer 1; data preprocessing layer 2; AI analysis engine layer 3; data storage layer 4; application service layer 5; monitoring and maintenance layer 6. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of the invention, and not all of them. Unless otherwise specified, the embodiments and features described in this application can be combined with each other. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0018] Furthermore, "multiple" refers to two or more. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of a person skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the invention.

[0019] Example 1 like Figure 1-4 As shown, this embodiment provides a big data analysis platform system based on artificial intelligence, including a data access layer 1, a data preprocessing layer 2, an AI analysis engine layer 3, a data storage layer 4, and an application service layer 5 that are connected in sequence, and a monitoring and maintenance layer 6 that is connected in communication with the above layers.

[0020] Specific implementation of data access layer 1 Data access layer 1 adopts a distributed data access framework, supporting multiple access methods such as FTP, HTTP, direct database connection, and message queues (e.g., Kafka). It can access structured data from the enterprise's internal ERP system, semi-structured data from user behavior logs, and unstructured data from product images. Data caching module 11 uses a Redis cache database to temporarily cache high-frequency data such as real-time user clicks and transaction records. The cache time is set to 1 hour. When the cached data reaches a preset threshold or the cache time expires, it is transferred in batches to data preprocessing layer 2, reducing the system resource consumption caused by frequent data transfers.

[0021] Specific implementation of data preprocessing layer 2 The data conversion module 21 uses an ETL tool to convert the incoming JSON format log data and XML format configuration data into standardized CSV format data. The data quality inspection module 22 uses a Python-written data quality inspection script to inspect the converted standardized data: for missing user age data, the median imputation method is used to repair it; for transaction amount data that exceeds the normal range (such as retail transaction amounts greater than 1 million yuan), it is judged as an outlier and manually reviewed and corrected; duplicate user registration data is removed by calculating the data hash value and comparing it. After preprocessing, the standardized data is transmitted to the AI ​​analysis engine layer 3.

[0022] Specific implementation of AI analysis engine layer 3 The model management submodule 31 manages analysis models using a model repository, supporting the registration, deployment, updating, and deregistration of models from various frameworks such as TensorFlow and PyTorch. The model training submodule 311 fine-tunes the basic neural network model based on the labeled user purchase behavior dataset provided by the user, generating a user purchase trend prediction model. The intelligent analysis submodule 32 calls this user purchase trend prediction model to perform feature extraction (such as extracting user consumption frequency and amount), trend prediction (predicting the probability of user purchases in the next 3 months), and correlation analysis (analyzing the related purchase relationships of different products) on the preprocessed user data. The result optimization submodule 33 presets an accuracy threshold of 90% and verifies the accuracy of the prediction results. When the accuracy does not reach the threshold, it iterative optimization is performed by adjusting parameters such as the learning rate and number of iterations of the neural network model. The analysis result interpretation submodule 331 uses the SHAP algorithm to generate a visual feature importance chart, showing the degree of influence of each feature on the prediction results.

[0023] Specific implementation of data storage layer 4 Data storage layer 4 adopts a hybrid storage architecture: MySQL is used as the relational database to store structured data such as basic user information and transaction orders; HDFS is used as the distributed file system to store unstructured data such as product images and videos; and InfluxDB is used as the time-series database to store real-time user behavior data. The data backup submodule 44 uses shell scripts to implement the backup strategy, performing incremental backups daily at 2 AM and full backups every Sunday at 2 AM. Backup data is transmitted encrypted to a remote backup server, and the backup retention period is 30 days.

[0024] Specific implementation of application service layer 5 Application service layer 5 uses Vue + ECharts to build a visualization interface, allowing users to generate bar charts, line charts and other visual charts by dragging and dropping, and customize the chart colors and font sizes; it provides a RESTful API-based analysis service interface, allowing users to call the platform's user purchase trend prediction function through the API to achieve integration with the enterprise CRM system.

[0025] Specific implementation of monitoring and maintenance layer 6 The monitoring and maintenance layer 6 uses Prometheus + Grafana to implement monitoring functions, collecting real-time operational status data of each module (such as the access success rate of the data access layer and the processing time of the data preprocessing layer), resource utilization data (such as the CPU utilization threshold set to 80% and the memory utilization threshold set to 85%), and task execution progress data. When the CPU utilization exceeds 80% or the data access failure rate exceeds 5%, an alarm email is sent to the maintenance personnel via the email server, and the alarm information is displayed on the platform's maintenance interface.

[0026] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0027] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0028] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A big data analysis platform system based on artificial intelligence, characterized in that, It includes a data access layer, a data preprocessing layer, an AI analysis engine layer, a data storage layer, and an application service layer that are connected in sequence. The data access layer is used to access multi-source heterogeneous data; The data preprocessing layer is used to clean, transform, integrate, and reduce the raw data received by the data access layer to obtain standardized data to be analyzed. The AI ​​analysis engine layer is used to call the analysis model to perform intelligent analysis on the standardized data to be analyzed and to optimize the analysis results; The data storage layer is used to classify and store raw data, preprocessed data, and analysis results data. The application service layer is used to provide users with a visual data display and analysis service interface.

2. The big data analysis platform system based on artificial intelligence according to claim 1, characterized in that, The data access layer is also equipped with a data caching module for temporarily caching high-frequency data that is accessed in real time.

3. The big data analysis platform system based on artificial intelligence according to claim 1, characterized in that, The data preprocessing layer includes a data conversion module and a data quality detection module; the data conversion module is used to standardize the conversion of data in different formats; the data quality detection module is used to detect and repair missing values, outliers, and duplicate values ​​in the original data.

4. The big data analysis platform system based on artificial intelligence according to claim 1, characterized in that, The AI ​​analysis engine layer includes a model management submodule, an intelligent analysis submodule, and a result optimization submodule; The model management submodule is used to register, deploy, update, and deregister analysis models; The intelligent analysis submodule is used to call the target analysis model in the model management submodule to perform feature extraction, trend prediction and correlation analysis on the standardized data to be analyzed. The result optimization submodule is used to verify the accuracy of the analysis results of the intelligent analysis submodule, and to iteratively optimize the parameters of the target analysis model when the accuracy does not reach a preset threshold.

5. The big data analysis platform system based on artificial intelligence according to claim 4, characterized in that, The model management submodule also includes a model training submodule, which is used to train or fine-tune the basic model based on the labeled dataset provided by the user to generate a customized analysis model.

6. The big data analysis platform system based on artificial intelligence according to claim 4, characterized in that, The result optimization submodule also includes an analysis result interpretation submodule, which uses an interpretable AI algorithm to provide a visual explanation of the generation process and key influencing factors of the analysis results.

7. The big data analysis platform system based on artificial intelligence according to claim 1, characterized in that, The data storage layer adopts a hybrid storage architecture, including a relational database, a distributed file system, and a time-series database, which are used to store structured data, unstructured data, and time-series analysis data, respectively.

8. A big data analysis platform system based on artificial intelligence according to claim 1 or 7, characterized in that, The data storage layer also includes a data backup submodule, which is used to perform regular backups of the stored data using a combination of incremental backups and full backups, and to store the backup data to an off-site backup node.

9. The big data analysis platform system based on artificial intelligence according to claim 1, characterized in that, The application service layer provides a visualization data display interface that supports the generation of various data visualization charts and style customization.

10. The big data analysis platform system based on artificial intelligence according to claim 1, characterized in that, It also includes a monitoring and maintenance layer, which is connected to the data access layer, data preprocessing layer, AI analysis engine layer, data storage layer and application service layer. It is used to monitor the operating status, resource utilization and task execution progress of each module in real time, and trigger alarms when an anomaly is detected.

Images