An offline data analysis system based on big data offline computing framework

Abstract

The present invention provides an offline data analysis system based on a big data offline computing framework, which belongs to the field of data processing technology and includes: a data configuration module: determining a number of data sources and configuring all data sources; a component determination module: determining a number of data analysis components; a process construction module: encapsulating all data analysis components, and integrating the data analysis components based on a preset analysis process template, thereby constructing a number of initial data analysis processes; a process optimization module: generating business processing logic based on the initial data analysis process, and arranging the initial data analysis process based on the business processing logic, thereby generating a final data analysis process; a code execution module: analyzing the preset tasks, and generating data analysis code in combination with the final data analysis process, thereby executing the data analysis code for data analysis. The automation level and execution efficiency of data analysis are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an offline data analysis system based on a big data offline computing framework. Background Art

[0002] With the rapid development of big data technology, the storage and processing capabilities of massive data have become an important support for the information society. In order to meet business needs, enterprises often use offline data analysis methods to conduct in-depth mining and analysis of historical data.

[0003] Existing offline data analysis systems are usually based on a distributed computing framework, and complete offline data processing by manually configuring data sources, designing analysis processes, and writing analysis codes. However, these systems have significant limitations: the data source configuration process is complicated and lacks unified standardized management, which increases the difficulty of operation; the design of data analysis processes mostly relies on manual labor, making it difficult to reuse existing analysis logic, resulting in low efficiency; the lack of automated process optimization and code generation mechanisms results in low efficiency of offline data analysis and difficulty in meeting business needs. Based on the shortcomings of the above-mentioned existing technical solutions, there is an urgent need for an offline data analysis system that can simplify data configuration, automate analysis process design, and optimize analysis efficiency to address the shortcomings of the existing technology.

[0004] Therefore, the present invention provides an offline data analysis system based on a big data offline computing framework. Summary of the invention

[0005] The present invention provides an offline data analysis system based on a big data offline computing framework, which is used to achieve efficient configuration of data sources, automatic determination of data analysis components, and rapid construction and optimization of analysis processes through modular design, and generates efficient data analysis code in combination with business logic, thereby greatly improving the automation level and execution efficiency of data analysis, reducing manual intervention and error rate, and enhancing the flexibility and reusability of the process, and is suitable for large-scale offline data analysis scenarios.

[0006] The present invention provides an offline data analysis system based on a big data offline computing framework, comprising:

[0007] Data configuration module: obtains and analyzes business requirements, determines several data sources based on the analysis results, and configures all data sources;

[0008] Component determination module: performs initial analysis on the configured data source and determines several data analysis components based on the initial analysis results;

[0009] Process building module: encapsulates all data analysis components and integrates them based on the preset analysis process template to build several initial data analysis processes;

[0010] Process optimization module: Generates business processing logic based on the initial data analysis process, and arranges the initial data analysis process based on the business processing logic to generate the final data analysis process;

[0011] Code execution module: Analyzes the preset tasks, generates data analysis code based on the final data analysis process, and then executes the data analysis code to perform data analysis.

[0012] The present invention provides an offline data analysis system based on a big data offline computing framework, a data configuration module, including:

[0013] Target determination unit: guides users through the preset low-code platform interface to determine the user's analysis goals;

[0014] Demand determination unit: obtains the initial business requirements input by the user on the preset low-code platform, and then determines the user's business requirements in combination with the user's analysis goals;

[0015] Demand analysis unit: Analyzes the user's business needs and determines several data sources to be accessed;

[0016] Data configuration unit: determines the type of each data source, and then determines the configuration mode of each data source based on a preset type-configuration mode data table, and then configures the corresponding data source based on the configuration mode of each data source.

[0017] The present invention provides an offline data analysis system based on a big data offline computing framework, a component determination module, including:

[0018] Coefficient determination unit: performs initial analysis on the configured data sources to determine the component matching coefficient of each data source;

[0019] Component determination unit: determining a number of initial data analysis components corresponding to each data source based on the component matching coefficient of each data source and a preset coefficient-component database;

[0020] A coefficient analysis unit: when the number of data sources with the same component matching coefficient is greater than a preset number, each data source with the same component matching coefficient is determined as a first data source;

[0021] Complex analysis unit: performing complex analysis on all first data sources, thereby determining a complex component matching coefficient of each first data source;

[0022] Coefficient matching unit: obtaining a number of data stream collision components that match the complex component matching coefficient from the complex coefficient-collision component database;

[0023] Component combination unit: based on a number of data stream collision components and initial data analysis components corresponding to each first data source, determines the data analysis component corresponding to each first data source.

[0024] The present invention provides an offline data analysis system based on a big data offline computing framework, a coefficient determination unit, comprising:

[0025] Perform an initial analysis of the configured data sources to determine the component matching coefficients for each data source:

[0026] ;in, is the component matching coefficient of the i-th data source, is the average usage frequency coefficient of the historical components of the i-th data source, is the missing value of the data of the ith data source, is the total value of data from the ith data source, is the data outlier of the ith data source, is the scale factor of the ith data source, is the preset maximum scale factor, is the weight corresponding to the usage frequency of the historical component of the i-th data source, is the weight corresponding to the quality of the i-th data source, is the weight corresponding to the scale of the i-th data source, The preset adjustment coefficient for missing values ​​in the data source. The preset adjustment factor for data outliers in the data source.

[0027] The present invention provides an offline data analysis system based on a big data offline computing framework, an initial data analysis component, including: a data preprocessing component, a feature engineering component, a model selection component, a data visualization component and a model evaluation component.

[0028] The present invention provides an offline data analysis system based on a big data offline computing framework, a process building module, including:

[0029] Interface configuration unit: configures a unified input and output interface for each data analysis component based on a preset configuration device, thereby determining the input and output requirements of each data analysis component;

[0030] Parameter determination unit: obtains the characteristics of each data analysis component and determines the initial parameters of each data analysis component based on a preset characteristic-parameter data table;

[0031] Template retrieval unit: retrieves several analysis process templates that meet business requirements from a preset analysis process template library;

[0032] Template acquisition unit: based on the characteristics, input and output requirements and business needs of each data analysis component, retrieve from the preset analysis process template library to obtain several analysis process templates;

[0033] Process building unit: Arrange all data analysis components in the order of steps defined in each analysis process template, and then preliminarily build several analysis processes;

[0034] Result determination unit: performs dependency analysis on the data analysis components with connection relationships, and then determines the matching results between all the data analysis components with connection relationships;

[0035] Component matching unit: if the data analysis components with connection relationship do not match, the unmatched data analysis components are adjusted based on a preset adjustment method until all the data analysis components with connection relationship match;

[0036] Process generation unit: Integrate all data analysis components according to the preset analysis process template to generate several initial data analysis processes.

[0037] The present invention provides an offline data analysis system based on a big data offline computing framework, and an analysis process template, including: a cleaning process template, a feature engineering process template, and a model training process template;

[0038] The initial data analysis process includes: component chain process, data flow diagram process and detailed configuration information process between components.

[0039] The present invention provides an offline data analysis system based on a big data offline computing framework, a process optimization module, including:

[0040] Relationship acquisition unit: Analyze business requirements and extract key process mapping relationships;

[0041] Process analysis unit: Analyze the initial data analysis process to obtain the input and output relationship of the initial data analysis process;

[0042] Logical construction unit: constructs business processing logic based on key process mapping relationships and input and output relationships of the initial data analysis process;

[0043] Process orchestration unit: orchestrates the initial data analysis process based on business processing logic to generate the final data analysis process.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] Through modular design, efficient configuration of data sources, automatic determination of data analysis components, and rapid construction and optimization of analysis processes are achieved. In addition, efficient data analysis code is generated in combination with business logic, which greatly improves the automation level and execution efficiency of data analysis, reduces manual intervention and error rates, and enhances the flexibility and reusability of the process, making it suitable for large-scale offline data analysis scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 It is a structural diagram of an offline data analysis system based on a big data offline computing framework provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Embodiment 1:

[0050] The embodiment of the present invention provides an offline data analysis system based on a big data offline computing framework, such as Figure 1 As shown, including:

[0051] Data configuration module: obtains and analyzes business requirements, determines several data sources based on the analysis results, and configures all data sources;

[0052] Component determination module: performs initial analysis on the configured data source and determines several data analysis components based on the initial analysis results;

[0053] Process building module: encapsulates all data analysis components and integrates them based on the preset analysis process template to build several initial data analysis processes;

[0054] Process optimization module: Generates business processing logic based on the initial data analysis process, and arranges the initial data analysis process based on the business processing logic to generate the final data analysis process;

[0055] Code execution module: Analyzes the preset tasks, generates data analysis code based on the final data analysis process, and then executes the data analysis code to perform data analysis.

[0056] In this embodiment, initial analysis refers to basic data processing and analysis operations on the configured data source, the purpose of which is to provide basic information and reference for subsequent data analysis processes. This analysis usually includes the following steps: data integrity check (such as missing values, duplicate values, etc.); preliminary statistics of data (such as basic statistical indicators such as maximum value, minimum value, mean, variance, etc.); basic feature extraction of data distribution (such as normal distribution, skewed distribution, etc.). Initial analysis provides the necessary input for determining appropriate data analysis components and analysis methods, ensuring that the subsequent analysis process is more accurate and efficient;

[0057] In this embodiment, the data analysis component is a modular functional unit designed for specific data processing and analysis tasks. Each component can independently complete specific functions and can be used in combination in the analysis process. For example, in user behavior analysis, the "cleaning component" can be used to remove noise data, and then the "aggregation component" can be used to count the frequency of various behaviors, and finally the "modeling component" can be called to build a user behavior prediction model.

[0058] In this embodiment, the preset analysis process template is a structured analysis process design framework, which includes several standardized steps to guide and constrain the construction of data analysis processes. Templates can be predefined according to different business needs and scenarios, and have a certain degree of flexibility. Function: Provide a general process framework to reduce the complexity of analysis design; quickly reuse existing processes to improve development efficiency; ensure the standardization and consistency of analysis processes. For example: A preset process template for sales data analysis may include the following steps: data loading and cleaning; grouping and statistics of sales data; user characteristics and sales trend analysis; data modeling and forecasting. Developers can directly replace specific data analysis components based on the template without having to design the entire process from scratch.

[0059] In this embodiment, the business processing logic is the core operation rules extracted and executed from the analysis process based on specific business needs. These rules are used to achieve business goals and bridge the data analysis process with actual business needs. For example, in marketing data analysis, the business processing logic may include the following rules: calculate the user's purchase frequency and group users according to frequency; push new products to high-frequency users and provide discounts and promotions to low-frequency users. These logics will be adjusted dynamically and rely on data analysis results to optimize decisions.

[0060] In this embodiment, the data analysis code refers to the executable code generated based on the final data analysis process, which can be directly run on the big data computing framework to complete specific data analysis tasks. It is the core implementation form of converting the analysis process into a computing operation. Assuming that the final data analysis process needs to clean data, count user behavior frequency and generate reports, the data analysis code may be composed of the following: code snippets for data cleaning (such as Spark SQL statements to filter missing values); user behavior statistics code (such as Python code based on MapReduce calculations); report generation code (such as Python code that calls visualization libraries). The beneficial effects of the above technical solution are: through modular design, efficient configuration of data sources, automatic determination of data analysis components, rapid construction and optimization of analysis processes are achieved, and efficient data analysis code is generated in combination with business logic, which greatly improves the automation level and execution efficiency of data analysis, reduces manual intervention and error rates, and enhances the flexibility and reusability of the process, which is suitable for large-scale offline data analysis scenarios.

[0061] Embodiment 2:

[0062] The embodiment of the present invention provides an offline data analysis system based on a big data offline computing framework, a data configuration module, including:

[0063] Target determination unit: guides users through the preset low-code platform interface to determine the user's analysis goals;

[0064] Demand determination unit: obtains the initial business requirements input by the user on the preset low-code platform, and then determines the user's business requirements in combination with the user's analysis goals;

[0065] Demand analysis unit: Analyzes the user's business needs and determines several data sources to be accessed;

[0066] Data configuration unit: determines the type of each data source, and then determines the configuration mode of each data source based on a preset type-configuration mode data table, and then configures the corresponding data source based on the configuration mode of each data source.

[0067] In this embodiment, the preset low-code platform refers to a development platform that provides users with a simple graphical interface and interactive functions. Users can achieve complex data analysis needs by dragging components, filling in parameters or simple configuration operations without having to master programming technology in depth.

[0068] For example, if a user wants to analyze e-commerce sales data, he can select the "Sales Trend Analysis" module through the low-code platform interface, enter the relevant time period and product information, and quickly complete the configuration and implementation of the analysis goal without writing code;

[0069] In this embodiment, the user's analysis goal refers to the specific business goal or expected result that the user hopes to achieve through data analysis, which is the starting point and key guide for the entire analysis process. The analysis goal can be selected or customized by the user through the low-code platform interface. For example, the user's analysis goals may include: identifying high-value customers on the e-commerce platform; analyzing sales trends in a specific region; and performing fault detection on industrial sensor data.

[0070] In this embodiment, the initial business requirement is the demand information that the user first enters or describes on the low-code platform, which is used to explain the specific requirements of its business background and data analysis. It is a refinement and supplement of the analysis goal, and is also the core input of the data analysis process. For example:

[0071] The user wants to analyze "sales data for the past three months", which is the data range described in the initial business requirements; the user wants to find "the product with the fastest sales growth", which is a specific analysis task requirement;

[0072] In this embodiment, the user's business needs are specific data analysis needs determined by analyzing and refining the initial business needs. It is a more clear and structured form of needs after processing, and is an important basis for guiding the subsequent analysis process. For example:

[0073] The initial business requirement is "analyze the sales data of the past three months", and the refined business requirements may include: data source: e-commerce platform sales data; analysis dimension: group by time (day / week); output: product sales growth trend chart. The initial requirement is "discover the user's purchasing habits", and the refined business requirements may include: data source: user behavior log; task: identify the user's purchase frequency, preferred product type, etc.

[0074] In this embodiment, the configuration method of the data source refers to the predefined standardized configuration method used for the specific attributes of each data source (such as data type, source method, access rights, etc.) to complete the access and management of the data source. Features: Standardization: Based on the preset "type-configuration method" data table, quickly match the best configuration scheme for the data source. The main contents of the configuration method: data source type: such as MySQL database, CSV file, Kafka stream data; access parameters: including database connection URL, username / password, file path, data stream access port, etc.; data structure information: such as field type, data format, etc. For example: If the data source is a MySQL database, the configuration method may include: type: relational database; configuration parameters: database URL, table name, user credentials; access method: access through JDBC driver. If the data source is a CSV file, the configuration method may include: type: file type data; configuration parameters: file path, delimiter, encoding format; access method: local file reading or distributed storage system (such as HDFS) loading.

[0075] The beneficial effects of the above technical solution are: guiding users to determine analysis goals and business needs through the low-code platform, and analyzing the needs to access multiple data sources. After determining the data source type and configuration method, the data source is automatically configured, effectively lowering the user's technical threshold, improving the efficiency of the data analysis process and the accuracy of the configuration, meeting complex business needs while significantly improving the user experience.

[0076] Embodiment 3:

[0077] The embodiment of the present invention provides an offline data analysis system based on a big data offline computing framework, and a component determination module, including:

[0078] Coefficient determination unit: performs initial analysis on the configured data sources to determine the component matching coefficient of each data source;

[0079] Component determination unit: determining a number of initial data analysis components corresponding to each data source based on the component matching coefficient of each data source and a preset coefficient-component database;

[0080] A coefficient analysis unit: when the number of data sources with the same component matching coefficient is greater than a preset number, each data source with the same component matching coefficient is determined as a first data source;

[0081] Complex analysis unit: performing complex analysis on all first data sources, thereby determining a complex component matching coefficient of each first data source;

[0082] Coefficient matching unit: obtaining a number of data stream collision components that match the complex component matching coefficient from the complex coefficient-collision component database;

[0083] Component combination unit: based on a number of data stream collision components and initial data analysis components corresponding to each first data source, determines the data analysis component corresponding to each first data source.

[0084] In this embodiment, the component matching coefficient of each data source is a numerical indicator used to measure the degree of adaptation between a data source and a specific data analysis component. It reflects the possibility and priority of a data source being processed or analyzed by a component, and is derived based on the characteristics of the data source (such as data format, data structure, size, source, etc.). Matching rules: The matching coefficient is obtained by comparing the attributes of the data source (such as type, field, quality) with the requirements of the analysis component. For example: a data source is a CSV file (structured data), and the fields include timestamps and transaction amounts. The goal is to perform time series prediction: Analysis component A (suitable for processing CSV data for time series analysis): matching coefficient = 0.9; Analysis component B (suitable for processing graph structure data): matching coefficient = 0.2; The system will give priority to component A with a higher matching coefficient for initial analysis. If the data source is real-time streaming data (such as Kafka stream), its component matching coefficient may be biased towards the stream processing component, and the matching coefficient will reflect this tendency.

[0085] In this embodiment, the complex component matching coefficient is an advanced matching index further calculated for the "first data source" (data sources with the same component matching coefficient and exceeding the preset number), which is used to screen analysis components that are more suitable for complex analysis tasks. This coefficient is usually calculated based on factors such as the specific business requirements of the data source, the data level, and the processing complexity.

[0086] Assume that the system needs to analyze multiple sales data sources (A, B, C), and their initial component matching coefficients are all 0.85: the attributes of data source A indicate that its data volume is large and contains complex time series relationships, and the complex matching coefficient = 0.92; the attributes of data source B indicate that its data structure is simple, and the complex matching coefficient = 0.83; the attributes of data source C show that its data involves multi-table associations, and the complex matching coefficient = 0.88. The system preferentially selects components suitable for high-complexity analysis for data source A. Data source A may eventually match an advanced component that supports complex time series analysis, while data source B may match a basic component that supports simple data processing.

[0087] In this embodiment, the data stream collision component is a special analysis component for processing conflicts or fusions between multiple data streams, and can analyze the interaction relationship, conflict points or merging logic between data streams. It is usually used in real-time or batch data analysis scenarios to help the system coordinate the interaction of multiple data sources and optimize data processing efficiency or accuracy. For example: real-time e-commerce platform case: data stream 1: user click log (real-time stream); data stream 2: order transaction data (real-time stream), the task of the data stream collision component: match the time window of the two data streams to ensure that the user click log and order data are aligned; handle conflicting fields, such as repeated user IDs; merge the two streams to generate a unified analysis stream for calculating user conversion rate, smart manufacturing case: data stream 1: temperature data of sensor A; data stream 2: pressure data of sensor B. The task of the data stream collision component: merge the temperature and pressure data streams to generate a comprehensive indicator stream of "equipment health"; trigger the alarm logic when temperature anomalies and pressure anomalies occur at the same time.

[0088] The beneficial effects of the above technical solution are: by analyzing the component matching coefficients of the data source in stages and gradually optimizing the component selection process, multi-level component matching from initial data analysis to complex analysis is achieved. Through the combination of complex analysis units and collision component databases, it is ensured that suitable analysis components are matched for the data source efficiently and accurately, thereby improving the accuracy and adaptability of data analysis, effectively responding to the challenges brought by the complexity of the data source, and improving the intelligence of the system.

[0089] Embodiment 4:

[0090] The embodiment of the present invention provides an offline data analysis system based on a big data offline computing framework, a coefficient determination unit, including:

[0091] Perform an initial analysis of the configured data sources to determine the component matching coefficients for each data source:

[0092] ;in, is the component matching coefficient of the i-th data source, is the average usage frequency coefficient of the historical components of the i-th data source, is the missing value of the data of the ith data source, is the total value of data from the ith data source, is the data outlier of the ith data source, is the scale factor of the ith data source, is the preset maximum scale factor, is the weight corresponding to the usage frequency of the historical component of the i-th data source, is the weight corresponding to the quality of the i-th data source, is the weight corresponding to the scale of the i-th data source, The preset adjustment coefficient for missing values ​​in the data source. The preset adjustment factor for data outliers in the data source.

[0093] In this embodiment, the historical component average usage frequency coefficient refers to the average value of the frequency at which a component (such as a system module, algorithm model, etc.) of a data source is actually used in the past period of time. This coefficient can be obtained through statistical analysis of historical usage records, and reflects the usage tendency or importance of certain components of the data source in historical scenarios;

[0094] In this embodiment, the maximum scale factor is a preset fixed value used to represent the weight or standard value when the scale of the system or data source is the largest. It can be used as a comparison benchmark for scale factors of other data sources. Usually, the coefficient is designed as an absolute reference value to measure and adjust the differences between the scales of different data sources.

[0095] The beneficial effect of the above technical solution is: by comprehensively considering the historical component usage frequency of the data source, data quality (missing values ​​and outliers) and scale, etc., the component matching coefficient is determined, and the accuracy and robustness of the matching process are ensured by combining the preset weights and adjustment coefficients. This solution optimizes the adaptation effect of data sources and analysis components, and improves the system's processing capabilities and analysis efficiency for diversified data sources.

[0096] Embodiment 5:

[0097] An embodiment of the present invention provides an offline data analysis system based on a big data offline computing framework, and an initial data analysis component, including: a data preprocessing component, a feature engineering component, a model selection component, a data visualization component, and a model evaluation component.

[0098] In this embodiment, the data preprocessing component is used to clean the original data, convert the format, fill missing values, remove duplicates, etc., to ensure the integrity, consistency and availability of the data. For example, suppose there is a data set of user behavior logs, including timestamps, user IDs and behavior types, but there are duplicate records and missing values ​​in the data. Duplicate record removal: remove duplicate user behavior records, missing value filling: fill the missing timestamp field with the average value or default value, format conversion: convert the timestamp from a string format to a standard time format (such as ISO 8601);

[0099] In this embodiment, the feature engineering component is responsible for extracting and converting features from the data for subsequent modeling to improve the performance of the model. It includes operations such as feature selection, feature construction, and feature normalization. Assuming that the goal is to predict whether a user will buy a certain product, the existing raw data includes user browsing time, number of clicks, etc. Feature construction: create new features, such as user average click time = browsing time / number of clicks, feature normalization: scale browsing time and number of clicks to the range of 0 to 1 to avoid large differences in magnitude between features;

[0100] In this embodiment, the model selection component is responsible for selecting a suitable data analysis or machine learning model according to the analysis target, such as a regression, classification or clustering model. If the task is to predict the user's future purchase amount, a linear regression or decision tree regression model can be selected; if the task is to predict whether the user will purchase, a logistic regression or random forest classification model can be selected;

[0101] In this embodiment, the data visualization component is used to present information such as data distribution and analysis results in the form of charts and graphs to facilitate understanding and decision-making. For example, a bar chart is used to show the sales volume distribution of different commodities, a line chart is used to show the daily average activity trend of a user group, and a scatter plot is used to show the correlation between the number of user clicks and the purchase amount;

[0102] In this embodiment, the model evaluation component is responsible for evaluating the trained model to measure whether its performance meets the requirements. Indicators such as accuracy, recall, mean square error, etc. are usually used. For example, the accuracy and F1 score of the classification model are used to evaluate the classification effect of the model, the mean square error (MSE) of the regression model is used to evaluate the gap between the predicted value and the true value, and the recall rate of the recommendation system is calculated to measure whether the recommended products meet the user's interests.

[0103] The beneficial effects of the above technical solution are: through the division of labor and cooperation of the initial data analysis components, the whole process from data cleaning, feature extraction to model selection, evaluation and visualization is automated, which improves the efficiency and accuracy of data analysis, reduces manual intervention, improves the ability to handle complex data, and provides high-quality support for subsequent analysis.

[0104] Embodiment 6:

[0105] The embodiment of the present invention provides an offline data analysis system based on a big data offline computing framework, and a process building module, including:

[0106] Interface configuration unit: configures a unified input and output interface for each data analysis component based on a preset configuration device, thereby determining the input and output requirements of each data analysis component;

[0107] Parameter determination unit: obtains the characteristics of each data analysis component and determines the initial parameters of each data analysis component based on a preset characteristic-parameter data table;

[0108] Template retrieval unit: retrieves several analysis process templates that meet business requirements from a preset analysis process template library;

[0109] Template acquisition unit: based on the characteristics, input and output requirements and business needs of each data analysis component, retrieve from the preset analysis process template library to obtain several analysis process templates;

[0110] Process building unit: Arrange all data analysis components in the order of steps defined in each analysis process template, and then preliminarily build several analysis processes;

[0111] Result determination unit: performs dependency analysis on the data analysis components with connection relationships, and then determines the matching results between all the data analysis components with connection relationships;

[0112] Component matching unit: if the data analysis components with connection relationship do not match, the unmatched data analysis components are adjusted based on a preset adjustment method until all the data analysis components with connection relationship match;

[0113] Process generation unit: Integrate all data analysis components according to the preset analysis process template to generate several initial data analysis processes.

[0114] In this embodiment, the preset configuration device is used to configure a unified input and output interface for each data analysis component, so as to achieve seamless connection between components. The input and output requirements of each component are defined based on its functional characteristics to ensure that data can be shared and transferred between components. These interfaces include requirements such as data format, field definition, and data stream type. Through unified interface configuration, process interruptions caused by data incompatibility between components can be avoided. Assume that there are two components in the system: a data cleaning component (input: original log data, output: cleaned structured data), and a feature extraction component (input: structured data, output: extracted feature vector). The function of the preset configuration device is to configure the output interface of the data cleaning component to ensure that its output format (such as CSV file, JSON format, etc.) meets the input requirements of the feature extraction component. If the cleaning component outputs in JSON format and the feature extraction component requires CSV format, automatic format conversion is achieved through interface configuration.

[0115] In this embodiment, the initial parameters of each data analysis component are determined by the characteristics of the component and business requirements, and are pre-defined through the characteristic-parameter data table. For example, the characteristics include the algorithm type, input data scale, running time limit, etc. of the component. The initial parameters provide default values ​​for the operation of the component, but can be adjusted according to the actual business scenario. For example: Assume that there is a model training component in the system, and its characteristic is "supervised learning". The initial parameters may include: learning rate: 0.01, maximum number of iterations: 100, loss function: cross entropy. If a specific task requires a faster convergence speed, the learning rate can be adjusted to 0.05; if the amount of data is large, the maximum number of iterations can be increased to 200.

[0116] In this embodiment, the connection relationship refers to the upstream and downstream dependency relationship between data analysis components. The matching result is used to measure whether the data format, interface protocol, etc. between components can meet the docking requirements. If they do not match, the input and output interfaces or operating parameters of the components need to be adjusted to make them compatible. Assume that there are the following components: data cleaning component (output: structured CSV file), feature selection component (input: standardized feature matrix). If the output of the data cleaning component is directly passed to the feature selection component, mismatch may occur (for example, the feature selection component requires standardized data, while the cleaning component only outputs the original structured data). Solution: Adjust the connection relationship: Add a standardization component responsible for converting the cleaned CSV file into a standardized feature matrix, and adjust the parameters: If the standardization component has multiple standardization methods (such as Z-Score, Min-Max), select the appropriate standardization method according to the requirements of the feature selection component. Finally, the adjusted connection relationship is as follows: data cleaning component → data standardization component → feature selection component, matching result analysis: After standardization, the data format is consistent with the input requirements of the feature selection component, and the match is successful.

[0117] The beneficial effects of the above technical solution are: unified interface configuration, parameter initialization, templated retrieval and dependency matching of components are realized through the process construction module, efficient docking and automated adjustment between components are ensured, a stable initial analysis process is generated, the process construction efficiency and component adaptability are improved, complex business needs are met, and the flexibility and reliability of the system are enhanced.

[0118] Embodiment 7:

[0119] The embodiment of the present invention provides an offline data analysis system based on a big data offline computing framework.

[0120] Analysis process templates, including: cleaning process template, feature engineering process template, and model training process template;

[0121] The initial data analysis process includes: component chain process, data flow diagram process and detailed configuration information process between components.

[0122] In this embodiment, the component chain process is an important part of the initial data analysis process, which represents the sequential relationship and dependency structure between the data analysis components. By defining the connection order of the components, a complete analysis task chain is formed, from data input to result output, to ensure the logic and coherence of the process. Assuming that the task is user behavior analysis, the component chain process is as follows: data cleaning component → feature engineering component → model training component → model evaluation component. In this chain: the data cleaning component cleans the original log data, removes duplicates and fills missing values, the feature engineering component extracts user behavior features (such as click frequency, dwell time, etc.), the model training component trains the prediction model based on the extracted features (such as whether the user will buy), and the model evaluation component evaluates the accuracy and robustness of the model. Through this chain process, the data flows smoothly and the components work together to complete the task.

[0123] In this embodiment, the data flow diagram process is a visual representation of the initial data analysis process, describing the data transmission path between components and the change process of data form, helping analysts to intuitively understand the data processing process. The data flow diagram not only reflects the order of data flow, but also contains detailed information such as the format of data input and output, interface requirements, etc. Taking the user behavior analysis task as an example, the data flow diagram process may be as follows: data input node: original user behavior log data (format: JSON), data cleaning node: output is a structured data table (format: CSV, fields include user ID, behavior time, number of clicks, etc.), feature engineering node: output is a standardized feature matrix (format: numpy array, fields include residence time, click frequency and other features), model training node: input feature matrix, output prediction model (format: PMML or binary file), model evaluation node: output model evaluation results (format: chart or CSV).

[0124] The beneficial effects of the above technical solution are: by pre-setting analysis process templates such as cleaning, feature engineering and model training, as well as component chains, data flow diagrams and detailed configuration information in the initial data analysis process, the design and execution of the analysis process are standardized, which improves the process construction efficiency, component collaboration and data processing accuracy, and meets the needs of multiple scenarios.

[0125] Embodiment 8:

[0126] The embodiment of the present invention provides an offline data analysis system based on a big data offline computing framework, a process optimization module, including:

[0127] Relationship acquisition unit: Analyze business requirements and extract key process mapping relationships;

[0128] Process analysis unit: Analyze the initial data analysis process to obtain the input and output relationship of the initial data analysis process;

[0129] Logical construction unit: constructs business processing logic based on key process mapping relationships and input and output relationships of the initial data analysis process;

[0130] Process orchestration unit: orchestrates the initial data analysis process based on business processing logic to generate the final data analysis process.

[0131] In this embodiment, the process mapping relationship refers to the correspondence between business requirements and various links of the data analysis process, which is used to convert high-level business goals into specific analysis process components and their connection logic. Through the process mapping relationship, it is possible to clarify which components and steps correspond to specific business requirements and achieve accurate customization of the process.

[0132] The beneficial effects of the above technical solution are: extracting key process mapping relationships through the process optimization module, analyzing the input-output relationship of the initial process, building business processing logic, and orchestrating and optimizing the process, thereby improving the rationality and execution efficiency of the process, ensuring that the data analysis process is more in line with business needs and has greater flexibility and accuracy.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An offline data analysis system based on a big data offline computing framework, characterized in that: include: Data configuration module: obtains and analyzes business requirements, determines several data sources based on the analysis results, and configures all data sources; Component determination module: performs initial analysis on the configured data source and determines several data analysis components based on the initial analysis results; Process building module: encapsulates all data analysis components and integrates them based on the preset analysis process template to build several initial data analysis processes; Process optimization module: Generates business processing logic based on the initial data analysis process, and arranges the initial data analysis process based on the business processing logic to generate the final data analysis process; Code execution module: analyzes the preset tasks, generates data analysis code based on the final data analysis process, and then executes the data analysis code to perform data analysis; Among them, the component determination module includes: Coefficient determination unit: performs initial analysis on the configured data sources to determine the component matching coefficient of each data source; Component determination unit: determining a number of initial data analysis components corresponding to each data source based on the component matching coefficient of each data source and a preset coefficient-component database; A coefficient analysis unit: when the number of data sources with the same component matching coefficient is greater than a preset number, each data source with the same component matching coefficient is determined as a first data source; Complex analysis unit: performing complex analysis on all first data sources, thereby determining a complex component matching coefficient of each first data source; Coefficient matching unit: obtaining a number of data stream collision components that match the complex component matching coefficient from the complex coefficient-collision component database; Component combination unit: based on a number of data stream collision components and initial data analysis components corresponding to each first data source, determines the data analysis component corresponding to each first data source.

2. According to claim 1, an offline data analysis system based on a big data offline computing framework is characterized in that: Data configuration module, including: Target determination unit: guides users through the preset low-code platform interface to determine the user's analysis goals; Demand determination unit: obtains the initial business requirements input by the user on the preset low-code platform, and then determines the user's business requirements in combination with the user's analysis goals; Demand analysis unit: Analyzes the user's business needs and determines several data sources to be accessed; Data configuration unit: determines the type of each data source, and then determines the configuration mode of each data source based on a preset type-configuration mode data table, and then configures the corresponding data source based on the configuration mode of each data source.

3. The offline data analysis system based on the big data offline computing framework according to claim 1 is characterized in that: Coefficient determination unit, including: Perform an initial analysis of the configured data sources to determine the component matching coefficients for each data source: ;in, is the component matching coefficient of the i-th data source, is the average usage frequency coefficient of the historical components of the i-th data source, is the missing value of the data of the ith data source, is the total value of data from the ith data source, is the data outlier of the ith data source, is the scale factor of the ith data source, is the preset maximum scale factor, is the weight corresponding to the usage frequency of the historical component of the i-th data source, is the weight corresponding to the quality of the i-th data source, is the weight corresponding to the scale of the i-th data source, The preset adjustment coefficient for missing values ​​in the data source. The preset adjustment factor for data outliers in the data source.

4. The offline data analysis system based on the big data offline computing framework according to claim 1 is characterized in that: Initial data analysis components, including: data preprocessing component, feature engineering component, model selection component, data visualization component, and model evaluation component.

5. The offline data analysis system based on the big data offline computing framework according to claim 1 is characterized in that: Process building blocks, including: Interface configuration unit: configures a unified input and output interface for each data analysis component based on a preset configuration device, thereby determining the input and output requirements of each data analysis component; Parameter determination unit: obtains the characteristics of each data analysis component and determines the initial parameters of each data analysis component based on a preset characteristic-parameter data table; Template retrieval unit: retrieves several analysis process templates that meet business requirements from a preset analysis process template library; Template acquisition unit: based on the characteristics, input and output requirements and business needs of each data analysis component, retrieve from the preset analysis process template library to obtain several analysis process templates; Process building unit: Arrange all data analysis components in the order of steps defined in each analysis process template, and then preliminarily build several analysis processes; Result determination unit: performs dependency analysis on the data analysis components with connection relationships, and then determines the matching results between all the data analysis components with connection relationships; Component matching unit: if the data analysis components with connection relationship do not match, the unmatched data analysis components are adjusted based on a preset adjustment method until all the data analysis components with connection relationship match; Process generation unit: Integrate all data analysis components according to the preset analysis process template to generate several initial data analysis processes.

6. The offline data analysis system based on the big data offline computing framework according to claim 5 is characterized in that: Analysis process templates, including: cleaning process template, feature engineering process template, and model training process template; The initial data analysis process includes: component chain process, data flow diagram process and detailed configuration information process between components.

7. The offline data analysis system based on the big data offline computing framework according to claim 1 is characterized in that: Process optimization module, including: Relationship acquisition unit: Analyze business requirements and extract key process mapping relationships; Process analysis unit: Analyze the initial data analysis process to obtain the input and output relationship of the initial data analysis process; Logical construction unit: constructs business processing logic based on key process mapping relationships and input and output relationships of the initial data analysis process; Process orchestration unit: orchestrates the initial data analysis process based on business processing logic to generate the final data analysis process.

Images