Data prediction analysis method and system based on big data

Abstract

The invention relates to the technical field of data analysis, and discloses a data prediction analysis method and system based on big data, and the analysis method comprises the steps: S1, data base construction, S2, on-demand modeling and preparation, S3, batch flow fusion and service packaging, and S4, intelligent embedding and continuous learning. According to the method, the original data directly enters the lake, and the structure is redefined during analysis, so that the processing efficiency of semi-structured and non-structured data is greatly improved, the data access and analysis path is shortened, and the trial and error cost is reduced; through the batch flow integrated architecture, the real-time and offline data processing logic unification is realized, the data aperture difference is fundamentally eliminated, the uniqueness of the decision basis is ensured, and meanwhile, the real-time decision and response in the event are supported; through automatic monitoring and feedback closed loop, the model can continuously sense data changes and actively complete self-iteration, it is ensured that the prediction accuracy is maintained at a high level for a long time, and large-scale intelligent operation is achieved with lower cost.

Description

Technical Field

[0001] This invention belongs to the field of data analysis technology, specifically relating to a data prediction and analysis method and system based on big data. Background Technology

[0002] In today's data-driven business environment, enterprises rely on data predictive analytics systems for accurate decision-making, operational optimization, and market insights. These systems, through the mining and analysis of historical and real-time data, predict future trends and user behavior, and have become a key component of core competitiveness. Existing data predictive analytics systems are typically built on classic data warehouse architectures. While these systems can meet current market demands, a rigorous data schema must be defined before data is written to the data warehouse. This process is slow, inflexible, and unable to adapt to changing data formats. Moreover, if the initial schema design is flawed or business logic changes, the entire ETL process and data warehouse tables may need to be rebuilt, resulting in significant costs. Furthermore, traditional data processing architectures typically employ a batch processing-centric paradigm supplemented by stream processing. This approach leads to independent real-time and offline data processing links, separating the technology stack from business logic. Consequently, the data analytics application lacks a single, reliable source of authoritative data, increasing the complexity of decision-making and the risk of misjudgment. Therefore, designing a data predictive analytics method and system based on big data is essential. Summary of the Invention

[0003] The purpose of this invention is to provide a data prediction and analysis method and system based on big data that is simple in structure and reasonably designed in order to solve the above problems.

[0004] The present invention achieves the above objectives through the following technical solutions:

[0005] A big data-based data prediction and analysis method, specifically including the following steps:

[0006] S1. Data Infrastructure Construction: Data of various formats is fed into the data lake through multimodal access, then streaming access is used as the backbone to ensure that data is not lost abnormally, and finally a data table layer for integrated lake warehouse management is built on top of the data lake.

[0007] S2. On-demand modeling and preparation: Dynamically prepare and transform data according to specific analytical needs;

[0008] S3, Batch and Stream Fusion and Service-Oriented Encapsulation: Develop unified data processing logic, select execution mode and deployment tasks, then synchronize and export the result data as the first stage output, and encapsulate and expose the service, and then consume and monitor the service in downstream business.

[0009] S4, Intelligent Embedding and Continuous Learning: Deeply embed machine learning into the analysis process so that it can automatically adapt to changes in data.

[0010] As a further optimization of the present invention, in step S1, during the process of flowing data of various formats into the data lake in a multimodal access manner, tools such as Apache NiFi or Kafka Connect are used to configure the connector. Then, data from multiple sources such as relational databases, log files, IoT sensors, API interfaces, and social media are flowed into the data lake, and the format of the flowing data is consistent with the source. When streaming access is used as the backbone, Apache Kafka is used as the data hub. All real-time data flowing into the data lake is first distributed and buffered through Kafka topics to ensure that data is not lost and to decouple production and consumption. The data table layer for integrated lake warehouse management is built by Apache Iceberg, Delta Lake, or Apache Hudi. The established table layer can provide files in the data lake with database-like table structure, ACID transactions, version management, and efficient metadata management capabilities, realizing the combination of the flexibility of the data lake and the governance and management capabilities of the data warehouse.

[0011] As a further optimization of the present invention, in step S2, during dynamic preparation, an ad-hoc query engine such as Trino or Spark SQL is used to directly perform exploratory queries on the data in the data lake. During data query, a schema is defined for the data. Then, based on the exploration results, an SQL or Spark job is written to extract relevant fields from the original data and perform cleaning, transformation, association, and aggregation. Transforming the data means writing the processed data results back to the data lake to form a new, topic-oriented aggregation table for subsequent analysis.

[0012] As a further optimization of the present invention, in step S3, developing unified data processing logic requires providing specific business requirements. Then, developers use Flink SQL or Flink DataStream API to write core business code, defining the data source, transformation process, and output structure through the code. The execution mode uses stream processing as path A and batch processing as path B. When immediate results are needed, path A is activated, submitting the core business logic program as a resident stream task to the Flink cluster. The data source for this task is configured as the corresponding Topic in Kafka, continuously consuming new order messages from Kafka, calculating and incrementally updating results in real time, and continuously outputting incremental result streams. When full data, complex calculations, or corrections to real-time data are needed, path B is activated, submitting the same logic program as a one-time batch task to the Flink cluster. The data source for this task is configured as historical data in the data lake. The table finally processes all historical data at the set time node, automatically ending after generating full results and outputting a complete snapshot result covering the entire history. During service encapsulation and exposure, the result data stream or result file produced in the first stage is used to configure the task's sink, writing the result data to the appropriate service layer storage. For real-time incremental results of path A, they are directly written to Redis; for full results of path B, they are written to cloud storage, and then imported into analytical databases such as ClickHouse via scheduled tasks. When exposing the data, different service access methods are selected according to the storage type. For analytical databases, BI tools are directly used to configure the data source connection, and real-time queries can be performed via SQL. For high-speed cache, Redis key construction rules need to be provided, and queries can be performed directly in the code via the Redis client. For complex logic, microservices need to be developed. These services encapsulate the query and business logic for multiple data sources. The microservices are registered on the API gateway and exposed to the front-end application in the form of RESTful APIs.

[0013] As a further optimization of the present invention, in step S4, feature engineering is unified and platformized, then the model is trained and managed, then the model is put into service, and the model put into service is monitored and closed-loop feedback is provided, and finally the model is allowed to automatically and continuously iterate and learn.

[0014] As a further optimization of the present invention, in step S4, the feature engineering is unified and platformized by defining the required features according to the business logic and declaring the features in the feature platform using SQL or Python SDK. The feature channel is processed periodically using batch processing. Spark or Flink batch mode reads the full historical data from the data lake to calculate the features. The results are written to the offline storage of the feature platform for model training. The feature channel is processed by real-time event-driven stream processing. During the process, Flink or Spark Streaming consumes the Kafka data stream to calculate the features in real time. The results are written to the online storage of the feature platform for online models to access in milliseconds.

[0015] As a further optimization of the present invention, in step S4, during the process of training the model and managing the training, the required features are extracted from the offline storage of the feature platform using the Python SDK and associated with the label data to generate a dataset for model training. In Jupyter Notebook, the Scikit-learn framework is used for model training and parameter tuning. The Tracking API of MLflow is used to automatically record the parameters, evaluation metrics, model files, code versions, and dataset versions of each experiment. When a model performs well in the experiment, it is registered and put into the production environment.

[0016] As a further optimization of the present invention, in step S4, during the process of deploying the model to the service and monitoring it, a specified version is first pulled from the MLflow Registry, the model is packaged into a RESTful API and deployed to a Kubernetes cluster, or the model file is pushed to the online service of the feature platform, so that feature acquisition and model inference are completed in the same service; during the model inference process, the model service receives the user ID, and the service then initiates a query to the online storage of the feature platform to obtain the real-time feature vector of the user. Subsequently, the model service inputs the feature vector into the loaded model, obtains the prediction result, and returns the prediction result to the business application; during the closed-loop feedback process, the model performance is monitored, data drift is monitored, and service health is monitored, and a feedback data pipeline is established to send the GroundTruth generated by the business application back to the data lake or Kafka.

[0017] As a further optimization of the present invention, in step S4, the automatic continuous iterative learning is triggered at a fixed time according to a preset time, or when the model performance drops below a threshold, or when severe data drift is detected. After triggering the iterative learning task, the latest features and feedback data are obtained from the feature library, a new training set is generated, and the training code is automatically executed on the cluster to generate a new model. The new model is evaluated on the reserved test set or validation set. After the model performance is detected to be improved, it is automatically registered and deployed to the online environment. At the same time, the old model is archived and all metadata of this update is recorded.

[0018] A big data-based data prediction and analysis system includes a preprocessing module, which is connected to a modeling and calculation module and an external data source. The modeling and calculation module is connected to a service output module and a learning feedback module, and the learning feedback module is connected to the modeling and calculation module and the service output module.

[0019] The preprocessing module is used to perform step S1 to form a unified and manageable data base, and subsequent steps provide data services.

[0020] The modeling and calculation module is used to execute step S2, write unified data processing logic, and send the processing results to the service output module and the processed feature data to the learning feedback module.

[0021] The service output module is used to execute step S3, providing a real-time data interface for business applications, and also providing model prediction services generated by the learning feedback module to the outside world.

[0022] The learning feedback module obtains raw data from the preprocessing module and feature data from the modeling and calculation module. The feature platform uniformly manages the definition and calculation of features.

[0023] The beneficial effects of this invention are as follows:

[0024] 1. The data of this invention is directly injected into the data lake in its original format for storage. The schema is only applied when the data is read, as required by a specific analysis task. This allows for rapid adaptation to semi-structured and unstructured data, significantly reducing the cycle of new data access. It eliminates the need to wait for lengthy data modeling and ETL development processes, allowing direct exploration of the raw data in the data lake. This greatly shortens the path from data to analysis, breaks down data silos, unleashes the data creativity of frontline business personnel, and provides great analytical flexibility. Furthermore, when data logic anomalies occur during analysis, it is not necessary to overturn the entire ETL process and data warehouse tables; only the logic needs to be modified. The raw data serves as a basis for easy reprocessing, greatly reducing the trial-and-error costs of data analysis.

[0025] 2. This invention employs a batch-stream fusion processing framework, transforming the data processing chain from traditional post-event batch computation to continuous event stream processing. By maintaining computational state, processing unbounded data streams, and tightly integrating with low-latency storage and service layers, the system can complete detection, analysis, and execution the instant a business event occurs. Simultaneously, real-time data analysis and offline data processing share the same set of business logic code, fundamentally eliminating the data caliber discrepancies caused by two independent systems. This ensures the inherent consistency of data analysis results at different time granularities, providing a unique and reliable data benchmark for data analysis applications and reducing the complexity of decision-making and the risk of misjudgment.

[0026] 3. Through continuous monitoring and automated feedback loops, this invention enables the model to perceive changes in application trends and shifts in user behavior, and to proactively iterate itself, ensuring that the prediction accuracy remains at a high level. This directly translates into more precise inventory management, marketing effectiveness, and revenue growth, avoiding redundant development and enabling large-scale operation with lower costs and manpower. Attached Figure Description

[0027] Figure 1 This is a flowchart of the method of the present invention;

[0028] Figure 2 This is a system flowchart of the present invention.

[0029] In the diagram: 1. Preprocessing module; 2. Modeling and calculation module; 3. Service output module; 4. Learning feedback module. Detailed Implementation

[0030] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0031] Example: Please refer to Figure 1 A data prediction and analysis method based on big data, specifically including the following steps:

[0032] S1. Data Infrastructure Construction: Connectors are configured using tools such as Apache NiFi or Kafka Connect. Data from various sources, including relational databases, log files, IoT sensors, API interfaces, and social media, flows into the data lake, ensuring that the incoming data format matches the source. When streaming access is the backbone, Apache Kafka serves as the data hub. All real-time data flowing into the data lake is first distributed and buffered through Kafka topics. Finally, a lake-warehouse integrated management data table layer is built on top of the data lake, constructed using Apache Iceberg, Delta Lake, or Apache Hudi.

[0033] S2. On-demand Modeling and Preparation: Based on specific analytical needs, exploratory queries are performed directly on the data in the data lake using ad-hoc query engines such as Trino or Spark SQL. A schema is defined for the data during the query process. Then, based on the exploration results, SQL or Spark jobs are written to extract relevant fields from the raw data, performing cleaning, transformation, association, and aggregation. Transforming the data involves writing the processed results back to the data lake to form new, topic-oriented aggregation tables for subsequent analysis.

[0034] S3. Batch / Stream Fusion and Service-Oriented Encapsulation: Developing unified data processing logic requires providing specific business requirements. Developers then use Flink SQL or the Flink DataStream API to write core business code, defining the data source, transformation process, and output structure. Execution modes use stream processing as path A and batch processing as path B. When immediate results are needed, path A is used, submitting the core business logic as a persistent stream task to the Flink cluster. The data source for this task is configured as the corresponding topic in Kafka, continuously consuming new order messages from Kafka, calculating and incrementally updating results in real time, and continuously outputting incremental result streams. When full data, complex calculations, or real-time data correction are required, path B is used, submitting the same logic as a one-time batch task to the Flink cluster. The data source for this task is configured as historical data from the data lake. The table finally processes all historical data at the set time node, automatically ending after generating full results and outputting a complete snapshot result covering the entire history. During service encapsulation and exposure, the result data stream or result file produced in the first stage is used to configure the task sink, and the result data is written to the appropriate service layer storage. For real-time incremental results of path A, they are directly written to Redis, and for full results of path B, they are written to cloud storage. Then, they are imported into analytical databases such as ClickHouse through scheduled tasks. When exposing data, different service access methods are selected according to the storage type. For analytical databases, BI tools are directly used to configure the data source connection, and real-time queries can be performed through SQL. For high-speed cache, Redis key construction rules need to be provided, and queries can be performed directly through the Redis client in the code. For complex logic, microservices need to be developed. These services encapsulate the query and business logic of multiple data sources. The microservices are registered on the API gateway and exposed to the front-end application in the form of RESTful API.

[0035] S4. Intelligent Embedding and Continuous Learning: Deeply embedding machine learning into the analysis process enables it to automatically adapt to data changes. This involves unifying and platformizing feature engineering, training the model and managing its training, then deploying the model to the service and monitoring and providing closed-loop feedback. Finally, the model undergoes automatic and continuous iterative learning. Unifying and platformizing feature engineering requires defining the required features based on business logic and declaring them in the feature platform using SQL or Python SDK. Batch processing features are processed periodically using Spark or Flink batch mode to read all historical data from the data lake to calculate features. The results are written to the feature platform's offline storage for model training. Real-time event-driven streaming of the feature channel is used, with Flink or Spark Streaming consuming Kafka data streams to calculate features in real time. The results are written to the feature platform's online storage for millisecond-level access by online models. During model training and management, the required features are extracted from the feature platform's offline storage using Python SDK and associated with label data to generate a dataset for model training. This dataset is then processed in Jupyter. In the notebook, the Scikit-learn framework is used for model training and hyperparameter tuning. MLflow's Tracking API automatically records the parameters, evaluation metrics, model files, code versions, and dataset versions for each experiment. When a model performs well in an experiment, it is registered and deployed to the production environment. During model deployment and monitoring, the specified version is first pulled from the MLflow Registry, the model is packaged into a RESTful API and deployed to a Kubernetes cluster, or the model file is pushed to the feature platform's online service, enabling feature acquisition and model inference to be completed within the same service. During model inference, the model service receives the user ID and immediately queries the feature platform's online storage to obtain the user's real-time feature vector. The model service then inputs the feature vector into the loaded model to obtain the prediction result and returns it to the business application. In the closed-loop feedback process, model performance, data drift, and service health are monitored, and a feedback data pipeline is established to process grounded data generated by the business application. Truth data is fed back to the data lake or Kafka; automatic continuous iterative learning is triggered at preset times, or automatically when model performance drops below a threshold, or when severe data drift is detected. After triggering the iterative learning task, the latest features and feedback data are obtained from the feature library, a new training set is generated, and the training code is automatically executed on the cluster to generate a new model. The new model is evaluated on the reserved test set or validation set. After the model performance is improved, it is automatically registered and deployed to the online environment, while the old model is archived and all metadata of this update is recorded.

[0036] Please see Figure 2 It includes a preprocessing module 1, which is connected to the modeling and calculation module 2 and the external data source; the modeling and calculation module 2 is connected to the service output module 3 and the learning feedback module 4; and the learning feedback module 4 is connected to the modeling and calculation module 2 and the service output module 3.

[0037] Preprocessing module 1 is used to execute step S1. Through tools such as Apache NiFi, heterogeneous data from multiple sources such as relational databases, logs, and APIs are connected to the system. All real-time data first enters Apache Kafka for buffering and distribution to ensure that no data is lost. The data is finally stored in a lake warehouse integrated table built by Apache Iceberg, forming a unified and manageable data base, which provides data services in subsequent steps.

[0038] Modeling and calculation module 2 is used to execute step S2. It uses Trino to query the data in module 1 in real time and writes unified data processing logic using Apache Flink based on the requirements. The processing results are sent to service output module 3 and the processed feature data is sent to learning feedback module 4.

[0039] Service output module 3 is used to execute step S3, write real-time incremental results to Redis, write batch full results to ClickHouse, connect to ClickHouse through BI tools to provide reports for analysts, query Redis or ClickHouse through microservice API to provide real-time data interfaces for business applications, and also provide model prediction services generated by learning feedback module 4 to the outside world.

[0040] The learning feedback module 4 obtains raw data from the preprocessing module 1 and feature data from the modeling and calculation module 2. The feature platform uniformly manages the definition and calculation of features, uses tools such as MLflow for model training, management and deployment, deploys the trained model as an online service, and provides it to the outside world through the service output module 3. At the same time, the learning feedback module 4 continuously monitors the prediction performance and data distribution of the online model, and sends the real data generated in the business application back to the preprocessing module 1 through Kafka. When the model performance degrades or the data changes, the model is automatically retrained and deployed, starting a new round of optimization cycle.

[0041] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A data prediction and analysis method based on big data, characterized in that: This big data-based predictive analysis method specifically includes the following steps: S1. Data Infrastructure Construction: Data of various formats is fed into the data lake through multimodal access, then streaming access is used as the backbone to ensure that data is not lost abnormally, and finally a data table layer for integrated lake warehouse management is built on top of the data lake. S2. On-demand modeling and preparation: Dynamically prepare and transform data according to specific analytical needs; S3, Batch and Stream Fusion and Service-Oriented Encapsulation: Develop unified data processing logic, select execution mode and deployment tasks, then synchronize and export the result data as the first stage output, and encapsulate and expose the service, and then consume and monitor the service in downstream business. S4, Intelligent Embedding and Continuous Learning: Deeply embed machine learning into the analysis process so that it can automatically adapt to changes in data.

2. The data prediction and analysis method based on big data according to claim 1, characterized in that: In step S1, during the process of inflowing data of various formats into the data lake via multimodal access, tools such as Apache NiFi or Kafka Connect are used to configure connectors. Data from various sources, including relational databases, log files, IoT sensors, API interfaces, and social media, is then inflowed into the data lake, ensuring that the inflowing data format matches the source. When streaming access is used as the backbone, Apache Kafka serves as the data hub. All real-time data flowing into the data lake is first distributed and buffered through Kafka topics to ensure no data loss and decouple production from consumption. The integrated lakehouse management data table layer is built using Apache Iceberg, Delta Lake, or Apache Hudi. This layer provides database-like table structures, ACID transactions, version management, and efficient metadata management capabilities for files in the data lake, combining the flexibility of the data lake with the governance and management capabilities of a data warehouse.

3. The data prediction and analysis method based on big data according to claim 1, characterized in that: In step S2, during dynamic preparation, ad-hoc query engines such as Trino or Spark SQL are used to directly perform exploratory queries on the data in the data lake. During data query, a schema is defined for the data. Then, based on the exploration results, SQL or Spark jobs are written to extract relevant fields from the raw data and perform cleaning, transformation, association, and aggregation. Transforming the data means writing the processed data results back to the data lake to form new, topic-oriented aggregation tables for subsequent analysis.

4. The data prediction and analysis method based on big data according to claim 1, characterized in that: In step S3, developing unified data processing logic requires providing specific business requirements. Developers then use Flink SQL or the FlinkDataStream API to write core business code, defining the data source, transformation process, and output structure. The execution mode uses stream processing as path A and batch processing as path B. When immediate results are needed, path A is used, submitting the core business logic as a persistent stream task to the Flink cluster. The data source for this task is configured as the corresponding Topic in Kafka, continuously consuming new order messages from Kafka, calculating and incrementally updating results in real time, and continuously outputting incremental result streams. When full data, complex calculations, or real-time data correction are needed, path B is used, submitting the same logic as a one-time batch task to the Flink cluster. The data source for this task is configured as a historical data table in the data lake. Finally, at the set time points, all historical data is processed, and after generating full results, the process automatically terminates, outputting a complete snapshot result covering all historical data. During service encapsulation and exposure, the result data stream or result file produced in the first phase is used to configure the task sink. The result data is written to the appropriate service layer storage. Real-time incremental results for path A are written directly to Redis, while full results for path B are written to cloud storage. Then, they are imported into analytical databases such as ClickHouse via scheduled tasks. When exposing the data, different service access methods are selected according to the storage type. For analytical databases, BI tools are used to configure the data source connection, and real-time queries can be performed via SQL. For high-speed cache, Redis key construction rules need to be provided, and queries can be performed directly in the code via the Redis client. For complex logic, microservices need to be developed. These services encapsulate the query and business logic for multiple data sources. The microservices are registered on the API gateway and exposed to the front-end application in the form of RESTful APIs.

5. The data prediction and analysis method based on big data according to claim 1, characterized in that: In step S4, feature engineering is unified and platformized, then the model is trained and managed, then the model is put into service, and the model put into service is monitored and closed-loop feedback is provided. Finally, the model is allowed to automatically and continuously iterate and learn.

6. The data prediction and analysis method based on big data according to claim 5, characterized in that: In step S4, unifying and platformizing feature engineering requires defining the required features based on business logic and declaring them in the feature platform using SQL or Python SDK. Batch feature channels are used periodically for processing, with Spark or Flink batch mode reading all historical data from the data lake to calculate features. The results are written to the offline storage of the feature platform for model training. Real-time event-driven stream processing of the feature channels is used, with Flink or Spark Streaming consuming Kafka data streams to calculate features in real time. The results are written to the online storage of the feature platform for millisecond-level access by online models.

7. The data prediction and analysis method based on big data according to claim 5, characterized in that: In step S4, during the training and management of the model, the required features are extracted from the offline storage of the feature platform using the Python SDK and associated with the label data to generate a dataset for model training. In Jupyter Notebook, the Scikit-learn framework is used for model training and parameter tuning. MLflow's Tracking API is used to automatically record the parameters, evaluation metrics, model files, code versions, and dataset versions for each experiment. When a model performs well in the experiment, it is registered and deployed to the production environment.

8. The data prediction and analysis method based on big data according to claim 5, characterized in that: In step S4, during the process of deploying and monitoring the model to the service, the specified version is first pulled from the MLflow Registry, the model is packaged into a RESTful API and deployed to the Kubernetes cluster, or the model file is pushed to the online service of the feature platform, so that feature acquisition and model inference are completed within the same service. During the model inference process, the model service receives the user ID, and the service immediately queries the online storage of the feature platform to obtain the real-time feature vector of the user. Then, the model service inputs the feature vector into the loaded model, obtains the prediction result, and returns the prediction result to the business application. In the closed-loop feedback process, the model's performance, data drift, and service health are monitored, and a feedback data pipeline is established to send the Ground Truth generated by the business application back to the data lake or Kafka.

9. The data prediction and analysis method based on big data according to claim 5, characterized in that: In step S4, the automatic continuous iterative learning is triggered at a fixed time according to a preset time, or when the model performance drops below a threshold, or when severe data drift is detected. After triggering the iterative learning task, the latest features and feedback data are obtained from the feature library, a new training set is generated, and the training code is automatically executed on the cluster to generate a new model. The new model is evaluated on the reserved test set or validation set. After the model performance is detected to be improved, it is automatically registered and deployed to the online environment. At the same time, the old model is archived and all metadata of this update is recorded.

10. A data prediction and analysis system based on big data, based on the data prediction and analysis method based on big data according to any one of claims 1-9, comprising a preprocessing module (1), characterized in that: The preprocessing module (1) connects the modeling calculation module (2) and the external data source. The modeling calculation module (2) connects the service output module (3) and the learning feedback module (4). The learning feedback module (4) connects the modeling calculation module (2) and the service output module (3). The preprocessing module (1) is used to perform step S1 to form a unified and manageable data base, and subsequent steps provide data services; The modeling and calculation module (2) is used to execute step S2, write unified data processing logic, send the processing results to the service output module (3), and send the processed feature data to the learning feedback module (4). The service output module (3) is used to execute step S3, providing a real-time data interface for business applications, and also providing model prediction services generated by the learning feedback module (4) to the outside world. The learning feedback module (4) obtains raw data from the preprocessing module (1) and feature data from the modeling and calculation module (2). The feature platform uniformly manages the definition and calculation of features.

Images