Credit investigation intelligent evaluation method and system based on batch-flow integrated computing engine

Abstract

The application provides a credit intelligent evaluation method based on a batch-flow integrated computing engine, and relates to the technical field of credit intelligent evaluation, wherein the method comprises the following steps: acquiring user credit data of multiple dimensions through Kafka, and dividing the user credit data into to-be-trained credit data and to-be-predicted credit data according to topics; acquiring a pre-trained credit evaluation model, inputting the to-be-predicted credit data into the credit evaluation model for real-time evaluation, and obtaining an evaluation result, wherein the credit evaluation model is dynamically updated by using the training data after the original training data is incrementally processed by the to-be-trained credit data. The application dynamically predicts the credit intelligent score by using a classical neural network model algorithm, can realize online learning of the evaluation model according to real-time streaming credit data, can update the model evaluation effect by combining historical rules and real-time changes, can eliminate the instability of the model, and thus the accuracy of the credit intelligent evaluation is improved.

Description

Technical Field

[0001] This application relates to the field of credit intelligent assessment technology, and in particular to a credit intelligent assessment method, apparatus and system based on a batch-stream integrated computing engine. Background Technology

[0002] Credit reporting is an activity in which professional, independent third-party institutions establish credit files for individuals or businesses, legally collect and objectively record their credit information, and legally provide credit information services to the public. Credit reporting plays a crucial role in facilitating personal credit, assisting financial lending decisions, and preventing credit risks, and it plays a fundamental role in promoting the development of the credit economy and the construction of a social credit system.

[0003] Traditional credit reporting is a professional credit management service where specialized institutions collect, process, and report financial and transaction information using fixed models. It is the most prevalent credit reporting model globally. However, traditional credit reporting faces challenges such as incomplete data, untimely updates, and high barriers to entry. In recent years, the rapid development of the internet and digital economy, along with big data technology, has injected new vitality into traditional credit reporting, broadening the sources of credit information and expanding the scope of traditional lending information sharing, ushering in the era of big data credit reporting. Big data credit reporting utilizes big data technology to redesign credit evaluation models and algorithms. Through multi-dimensional credit information examination, it expands the available dimensions of credit information, more comprehensively assesses the creditworthiness of individuals, and achieves more accurate credit assessments.

[0004] Currently, the main research methods for big data credit assessment are: (1) Statistical analysis method, which selects variables of a certain dimension from several indicators that can evaluate the credit characteristics of the object, and conducts credit assessment based on algorithms such as multivariate discriminant analysis and logistic regression; (2) Traditional machine learning method, which uses traditional machine learning algorithms such as support vector machine, decision tree, Bayesian network, and ensemble learning to construct a credit assessment model and conduct assessment through the collected credit data; (3) Neural network method based on deep learning, which inputs massive credit data as a training set into a deep neural network to train the model, and uses the trained model to conduct credit assessment.

[0005] However, existing big data credit assessment methods have many problems in practical applications, mainly including:

[0006] Method (1) uses statistical analysis to process and analyze credit data, which has strong interpretability and robustness. However, it is only suitable for non-high-dimensional data that is small in volume, simple in structure, fixed in source, and manually recorded. In the case of massive high-dimensional data, due to the limited indicator features, it is difficult to complete accurate modeling, and the evaluation results often suffer from overfitting.

[0007] Both methods (2) and (3) essentially extract features from existing credit data. By utilizing massive, multi-dimensional credit information, more accurate credit assessment models can be constructed. In the context of big data credit reporting, credit assessment data is characterized by its massive volume, real-time generation, and streaming nature. However, once a credit assessment model is built, the evaluation results rely solely on the historical credit data used for training and cannot be updated based on real-time generated credit data. The static characteristics of the model cannot adapt to the dynamic changes in credit data, making the model evaluation results unstable.

[0008] In summary, in the current research on intelligent credit assessment using big data, statistical analysis-based methods offer strong interpretability but struggle to maintain accuracy under high-dimensional credit data. Data-driven machine learning and deep learning methods can effectively mine the digital features of massive amounts of credit data and construct credit assessment models, but they face significant challenges in addressing the patterns of historical data accumulation, the potential risks of real-time changes, ensuring timely updates to assessment models, and maintaining the stability of assessment results. Summary of the Invention

[0009] This application aims to at least partially address one of the technical problems in the related art.

[0010] Therefore, the first objective of this application is to propose a credit intelligent assessment method based on a batch-stream integrated computing engine, which solves the technical problems of inaccurate and unstable credit scoring in existing credit intelligent assessment methods. It uses classic neural network model algorithms to dynamically predict credit intelligent scores and can realize online learning of the assessment model based on real-time streaming credit data. At the same time, it can update the model assessment effect by combining historical patterns and real-time changes, eliminating model instability and thus improving the accuracy of credit intelligent assessment.

[0011] The second objective of this application is to propose a credit assessment device based on a batch-stream integrated computing engine.

[0012] The third objective of this application is to propose a credit assessment system based on a batch-stream integrated computing engine.

[0013] The fourth objective of this application is to provide a non-transitory computer-readable storage medium.

[0014] To achieve the above objectives, the first aspect of this application proposes a credit assessment method based on a batch-stream integrated computing engine, comprising: acquiring user credit data from multiple dimensions through Kafka, and dividing the user credit data into credit data to be trained and credit data to be predicted according to the topic; acquiring a pre-trained credit assessment model, inputting the credit data to be predicted into the credit assessment model for real-time assessment, and obtaining the assessment result, wherein the credit assessment model is dynamically updated by incrementally adding data to the original training data using the credit data to be trained, and then using the incrementalized training data.

[0015] Optionally, in one embodiment of this application, user credit data from multiple dimensions is obtained through Kafka, and the user credit data is divided into credit data to be trained and credit data to be predicted according to the topic, including:

[0016] Use DataX, a tool that supports heterogeneous data source synchronization, to synchronize user credit data from multiple dimensions. Perform ETL processing on the user credit data and then aggregate the ETL-processed user credit data into a unified Kafka message queue.

[0017] The user credit data in the Kafka message queue is divided into credit data to be trained and credit data to be predicted according to the topic.

[0018] The credit data to be trained is synchronized to the Hive repository in real time for storage.

[0019] Optionally, in one embodiment of this application, the credit assessment model is dynamically updated, including:

[0020] Incremental training data is obtained by adding data increments to the original training data using the credit data to be trained.

[0021] When the incremental training data reaches the preset quantity requirement or the timing condition is triggered, the incremental training data is preprocessed and the preprocessed incremental training data is sent to the TensorFlow server. At the same time, the credit assessment model is sent to the TensorFlow server.

[0022] Based on incremental training data, the credit assessment model is trained and updated using a TensorFlow server to obtain the trained credit assessment model.

[0023] The updated credit assessment model is represented as follows:

[0024] M new =train(M) old ,I inc )

[0025] Among them, M new M represents the updated model.old I represents the old model from the previous stage. inc This represents the incremental training data from the previous stage to the current stage.

[0026] Optionally, in one embodiment of this application, the method further includes:

[0027] The raw training data is obtained through Kafka, synchronized to the Hive repository in real time for storage, preprocessed, and then sent to the TensorFlow server via remote RPC to build a credit assessment model based on the raw training data.

[0028] Optionally, in one embodiment of this application, the credit data to be trained and the original training data are synchronized to a Hive repository in real time for storage, including:

[0029] The Two Phase Commit protocol is used to synchronize training data from Kafka to the Hive repository in real time for storage. The training data includes credit data to be trained and raw training data.

[0030] Optionally, in one embodiment of this application, the training data is synchronized from Kafka to a Hive repository in real time for storage using the Two Phase Commit protocol, including:

[0031] The Checkpoint module of the job manager periodically sends Checkpoint barriers to the data source nodes, so that the Checkpoint barriers flow sequentially into the Window and Data Sink nodes in the direction of data flow. After the Window and Data Sink nodes receive the Checkpoint barrier, they stop the calculation and save the state snapshot to the highly available persistent storage component. Then, the Window and Data Sink nodes send an ACK confirmation to the Checkpoint module of the job manager.

[0032] Once all nodes have completed saving their state snapshots, they pre-commit external transactions to the Hive repository. The Hive repository then writes the data to the specified location and sends an ACK confirmation to the job manager.

[0033] Optionally, in one embodiment of this application, the training data is synchronized from Kafka to a Hive repository in real time using the Two Phase Commit protocol for storage, and the method further includes:

[0034] Once the job manager receives all ACK confirmations, it sends a status snapshot completion notification to each node. Simultaneously, the data sink node sends a commit command to the Hive repository via RPC. The Hive repository's transaction module executes the commit operation, completing the actual data writing.

[0035] To achieve the above objectives, a second aspect of this application proposes a credit assessment device based on a batch-stream integrated computing engine, comprising: an acquisition module and an assessment module, wherein...

[0036] The acquisition module is used to acquire user credit data from multiple dimensions through Kafka, and divide the user credit data into credit data to be trained and credit data to be predicted according to the topic.

[0037] The evaluation module is used to obtain a pre-trained credit assessment model, input the credit data to be predicted into the credit assessment model for real-time evaluation, and obtain the evaluation result. The credit assessment model is dynamically updated by incrementing the original training data with the credit data to be trained and then using the incremented training data.

[0038] To achieve the above objectives, a third aspect of this application proposes a credit assessment system based on a batch-stream integrated computing engine, comprising: a Kafka module, a Hive repository module, a TensorFlow engine module, a model and data management module, a stream computing module, and an assessment service module, wherein...

[0039] The Kafka module is used to access user credit data from multiple dimensions in real time, and divide the user credit data into credit data to be trained and credit data to be predicted according to the topic. The credit data to be trained is synchronized to the Hive repository for storage in real time, and the credit data to be predicted is sent to the stream computing module.

[0040] The Hive repository module is used to store training data, send training data to the TensorFlow engine module, store incremental result data sent by the model and data management modules, and perform data analysis based on the incremental result data.

[0041] The TensorFlow engine module is used to receive training data and a credit assessment model, train and update the credit assessment model, and send the updated credit assessment model to the model and data management module.

[0042] The model and data management module is used to periodically synchronize incremental result data to the Hive repository, send the credit assessment model to the TensorFlow engine, and also send the updated credit assessment model to the stream computing module.

[0043] The stream computing module is used to periodically pull pre-trained credit assessment models from the model and data management module, and simultaneously obtain the credit data to be predicted provided by the Kafka module in real time, perform DAG calculation, obtain the assessment results, and send the assessment results to the assessment service module; at the same time, the assessment results are sent as incremental result data to the model and data management module.

[0044] The evaluation service module is used to query the evaluation results obtained by the stream computing module, and can also query historical data to perform relevant statistical analysis.

[0045] To achieve the above objectives, a fourth aspect of this application proposes a non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by a processor, can perform a credit intelligent assessment method based on a batch-stream integrated computing engine.

[0046] The credit scoring intelligent assessment method, apparatus, and system based on a batch-stream integrated computing engine, as well as the non-transitory computer-readable storage medium, as described in this application, solve the technical problems of inaccurate and unstable credit scoring in existing credit scoring intelligent assessment methods. They utilize classic neural network model algorithms to dynamically predict credit scoring and can achieve online learning of the assessment model based on real-time streaming credit data. Furthermore, they can update the model's assessment effect by combining historical patterns with real-time changes, eliminating model instability and thus improving the accuracy of credit scoring intelligent assessment.

[0047] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0048] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0049] Figure 1 A flowchart of a credit assessment method based on a batch-stream integrated computing engine provided in Embodiment 1 of this application;

[0050] Figure 2 This is a Kafka-Flink-Hive end-to-end consistency flowchart of the credit assessment method based on a batch-stream integrated computing engine, as described in an embodiment of this application.

[0051] Figure 3 This is a diagram illustrating the overall framework of the credit intelligent assessment method based on a batch-stream integrated computing engine, as described in this application.

[0052] Figure 4This is a schematic diagram illustrating the model training and predictive analysis of the credit intelligent assessment method based on the batch-stream integrated computing engine in this application embodiment.

[0053] Figure 5 This is a schematic diagram of the structure of a credit assessment device based on a batch-stream integrated computing engine, provided in Embodiment 2 of this application.

[0054] Figure 6 This is a schematic diagram of the structure of a credit assessment system based on a batch-stream integrated computing engine, as provided in Embodiment 3 of this application.

[0055] Figure 7 This is a flowchart of the credit intelligent assessment system based on the batch-stream integrated computing engine, which is an embodiment of this application. Detailed Implementation

[0056] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0057] In current research on intelligent credit assessment, most researchers either use machine learning methods to train credit assessment models based on credit data, or use static models trained with neural networks in deep learning to predict dynamic credit data. Both approaches have limited or unstable impact on intelligent credit assessment scores.

[0058] Current research simply trains models on initial credit data and directly applies them to intelligent credit assessment. While this may achieve the desired results in the early stages of a credit system, as the system receives more and more data, these data characteristics will inevitably change. If a static model is used to predict new credit data, the assessment effectiveness will deteriorate with the growth of credit data. Therefore, a feedback mechanism is needed to continuously update the model to adapt to changes in new data and prevent the assessment from becoming biased.

[0059] This application proposes a dynamic model training and prediction method that enables the model to stably and efficiently achieve the intelligent credit assessment process.

[0060] The following description, with reference to the accompanying drawings, describes a credit assessment method, apparatus, and system based on a batch-stream integrated computing engine, according to embodiments of this application.

[0061] Figure 1 This is a flowchart of a credit assessment method based on a batch-stream integrated computing engine, provided in Embodiment 1 of this application.

[0062] like Figure 1 As shown, this intelligent credit assessment method based on a batch-stream integrated computing engine includes the following steps:

[0063] Step 101: Obtain user credit data from multiple dimensions through Kafka, and divide the user credit data into credit data to be trained and credit data to be predicted according to the topic;

[0064] Step 102: Obtain the pre-trained credit assessment model, input the credit data to be predicted into the credit assessment model for real-time evaluation, and obtain the evaluation result. The credit assessment model is dynamically updated by incrementing the original training data using the credit data to be trained and then using the incremented training data.

[0065] The intelligent credit assessment method based on a batch-stream integrated computing engine in this application acquires user credit data from multiple dimensions through Kafka and divides the user credit data into training data and prediction data according to topics. A pre-trained credit assessment model is obtained, and the prediction data is input into the model for real-time assessment to obtain the assessment result. The credit assessment model is dynamically updated by incrementally adding data to the original training data using the training data. This solves the technical problems of inaccurate and unstable credit scoring in existing intelligent credit assessment methods. It dynamically predicts intelligent credit scores using classic neural network model algorithms and can achieve online learning of the assessment model based on real-time streaming credit data. Furthermore, it can update the model's assessment effect by combining historical patterns and real-time changes, eliminating model instability and thus improving the accuracy of intelligent credit assessment.

[0066] The credit scoring intelligent assessment model training and predictive analysis method proposed in this application effectively solves the problem that the model's performance deteriorates as credit scoring data grows. Addressing the characteristic of credit scoring data distribution changing over time, this application utilizes newly added data with changing distribution to retrain the existing model, obtaining a new model that can then be applied to subsequent predictions. This ensures stable predictive performance and may even surpass that of the old model.

[0067] Credit data is mainly obtained through third-party credit reporting agencies, web crawlers, bank systems, or manual records, and it is characterized by being multi-sourced, unstructured, and massive.

[0068] Furthermore, in this embodiment of the application, user credit data from multiple dimensions is obtained through Kafka, and the user credit data is divided into credit data to be trained and credit data to be predicted according to the topic, including:

[0069] The user credit data from multiple dimensions is synchronized using DataX, a tool that supports heterogeneous data source synchronization. The user credit data is then processed by ETL and aggregated into a Kafka message queue. The user credit data in the Kafka message queue is divided into credit data to be trained and credit data to be predicted according to the topic.

[0070] Since Kafka does not persist the received credit data, the credit data to be trained in Kafka needs to be synchronized and stored in a Hive repository.

[0071] When new credit data arrives for training, or when there is dissatisfaction with the model's prediction results and correct "guidance" is provided, the exchange of model and data between prediction and training can be realized, enabling dynamic updates of the model.

[0072] Furthermore, in this embodiment of the application, the credit assessment model is dynamically updated, including:

[0073] Incremental training data is obtained by adding data increments to the original training data using the credit data to be trained.

[0074] When the incremental training data reaches the preset quantity requirement or the timing condition is triggered, the incremental training data is preprocessed and the preprocessed incremental training data is sent to the TensorFlow server. At the same time, the credit assessment model is sent to the TensorFlow server.

[0075] Based on incremental training data, the credit assessment model is trained and updated using a TensorFlow server to obtain the trained credit assessment model.

[0076] The updated credit assessment model is represented as follows:

[0077] M new =train(M) old ,I inc )

[0078] Among them, M new M represents the updated model. old I represents the old model from the previous stage. inc This represents the incremental training data from the previous stage to the current stage.

[0079] The Tensorflow server receives incremental training data preprocessed by Flink. inc And the model M of the model and data exchange module old This allows training a new model M. new .

[0080] Furthermore, in the embodiments of this application, the method further includes:

[0081] The raw training data is obtained through Kafka. Since Kafka does not persist the received credit data, the raw training data needs to be synchronized to the Hive repository in real time for storage. Flink's batch processing function is used to preprocess the raw training data, and the preprocessed raw training data is sent to the TensorFlow server via remote RPC (Remote Procedure Call Protocol) to build a credit assessment model based on the raw training data.

[0082] During the synchronization of credit scoring training data from Kafka to the Hive repository, to ensure the training effect of the credit scoring assessment model, it is necessary to ensure the integrity of the training data, that is, to ensure end-to-end consistency of the Kafka to Hive synchronization process. To ensure that this process conforms to the exactly-once semantics, the Flink stream processing engine is used to implement this synchronization process. Since the Flink engine only implements end-to-end consistency between Kafka-Flink-Kafka, and credit scoring data requires a persistent and stable data warehouse, this application utilizes the Two Phase Commit (2PC) protocol to implement end-to-end consistency between Kafka-Flink-Hive. Here, Flink defines an abstract interface TwoPhaseCommitSinkFunction for the Two Phase Commit protocol. To achieve consistency at the Hive output end, this application defines a sink class FlinkHiveProducerSink to implement the Two Phase Commit process.

[0083] Furthermore, in this embodiment of the application, the credit data to be trained and the original training data are synchronized to the Hive repository in real time for storage, including:

[0084] The training data is synchronized from Kafka to a Hive repository in real time using the Two Phase Commit protocol for storage. Figure 2 As shown, it includes a pre-submission stage and a submission stage, wherein the training data includes credit data to be trained and original training data.

[0085] Furthermore, in this embodiment, the training data is synchronized from Kafka to the Hive repository in real time for storage using the Two Phase Commit protocol, including:

[0086] During the pre-commit phase, the Checkpoint module of the job manager periodically sends Checkpoint barriers to the data source node (DataSource), so that the Checkpoint barriers flow sequentially into the Window and Data Sink nodes along the data flow direction. After the Window and Data Sink nodes receive the Checkpoint barrier, they stop the calculation and save the state snapshot to the highly available persistent storage (State Backend) component. That is, the current calculation state is saved to the highly available persistent storage (State Backend) component. Afterwards, the Window and Data Sink nodes send an ACK confirmation to the Checkpoint module of the job manager.

[0087] Once all nodes have completed saving their state snapshots (i.e., after the Data Sink node has completed saving its state snapshot), it pre-commits the external transaction to the Hive repository. Then, Hive writes the data to the specified location but does not commit it. Finally, it sends an ACK confirmation to the job manager.

[0088] Furthermore, in this embodiment of the application, the training data is synchronized from Kafka to the Hive repository in real time using the Two Phase Commit protocol for storage, and the method further includes:

[0089] Once the job manager receives all ACK confirmations, it enters the commit phase. The job manager sends a status snapshot completion notification to each node, indicating that the data saving is complete. At the same time, the Data Sink node sends a commit command to the Hive repository via RPC. The Hive repository's transaction module executes the commit operation, completing the actual writing of the data.

[0090] This application proposes a consistency algorithm for the credit reporting data synchronization process. The algorithm utilizes the classic Two Phase Commit protocol, and achieves end-to-end consistency of the entire system through the FlinkHiveProducerSink with the help of the Checkpoint mechanism and the transaction support module of the Hive warehouse. This ensures the consistency of credit reporting data synchronized from Kafka to the Hive data warehouse through the Flink component, providing data security, accuracy support capabilities, and a complete data support environment for the credit reporting intelligent assessment system.

[0091] Figure 3 This is a diagram illustrating the overall framework of the intelligent credit assessment method based on a batch-stream integrated computing engine, as described in this application.

[0092] like Figure 3 As shown, the intelligent credit assessment method includes: a Kafka component for unified access to multi-source credit data such as social, income, consumption, and credit data; Hive for storing standardized raw credit data; a Flink engine batch processing module for building intelligent credit assessment models from the raw data and updating the models online using new data; and a stream processing module for consuming credit prediction data from Kafka in real time, performing real-time intelligent credit assessment predictions, and displaying the intelligent credit assessment results.

[0093] Figure 4 This diagram illustrates the model training and predictive analysis of the credit intelligent assessment method based on the batch-stream integrated computing engine, as described in this application embodiment.

[0094] like Figure 4 As shown, the model training and predictive analysis process of this intelligent credit assessment method is as follows: New training data provided by the Kafka data source is synchronized to the Hive repository in real time. When the timed triggering conditions are met or the amount of new training data reaches a certain requirement, Flink is used to preprocess the new training data. The preprocessed data participates in the online learning and dynamic parameter update of the TensorFlow model through the model and data management module. The data to be evaluated from the Kafka data source is sent to the Flink real-time prediction module. Flink completes the real-time evaluation of the data to be evaluated by the TensorFlow model through the model and data management module and obtains the prediction results, and then displays the evaluation results.

[0095] Figure 5 This is a schematic diagram of the structure of a credit assessment device based on a batch-stream integrated computing engine, as provided in Embodiment 2 of this application.

[0096] like Figure 5 As shown, the intelligent credit assessment device based on a batch-stream integrated computing engine includes: an acquisition module and an assessment module, wherein...

[0097] The acquisition module 10 is used to acquire user credit data from multiple dimensions through Kafka, and divide the user credit data into credit data to be trained and credit data to be predicted according to the topic.

[0098] The evaluation module 20 is used to obtain a pre-trained credit assessment model, input the credit data to be predicted into the credit assessment model for real-time evaluation, and obtain the evaluation result. The credit assessment model is dynamically updated by incrementing the original training data with the credit data to be trained and then using the incremented training data.

[0099] This application's embodiment of the intelligent credit assessment device based on a batch-stream integrated computing engine includes: an acquisition module and an assessment module. The acquisition module acquires user credit data from multiple dimensions via Kafka and divides the user credit data into training data and prediction data according to topics. The assessment module acquires a pre-trained credit assessment model, inputs the prediction data into the model for real-time assessment, and obtains the assessment result. The credit assessment model is dynamically updated by incrementally adding training data to the original training data using the training data. This addresses the technical problems of inaccurate and unstable credit scoring in existing intelligent credit assessment methods. It dynamically predicts intelligent credit scores using classic neural network model algorithms and enables online learning of the assessment model based on real-time streaming credit data. Furthermore, it combines historical patterns with real-time changes to update the model's assessment effect, eliminating model instability and thus improving the accuracy of intelligent credit assessment.

[0100] Figure 6 This is a schematic diagram of the structure of a credit assessment system based on a batch-stream integrated computing engine, as provided in Embodiment 3 of this application.

[0101] like Figure 6 As shown, this credit scoring intelligent assessment system based on a batch and stream computing engine includes: a Kafka module, a Hive repository module, a TensorFlow engine module, a model and data management module, a stream computing module, and an assessment service module.

[0102] The Kafka module 100 is used to access user credit data from multiple dimensions in real time, and divide the user credit data into credit data to be trained and credit data to be predicted according to the topic. The credit data to be trained is synchronized to the Hive repository for storage in real time, and the credit data to be predicted is sent to the stream computing module.

[0103] The Hive repository module 200 is used to store training data, send training data to the TensorFlow engine module, store incremental result data sent by the model and data management modules, and perform data analysis based on the incremental result data.

[0104] The TensorFlow engine module 300 is used to receive training data and a credit assessment model, train and update the credit assessment model, and send the updated credit assessment model to the model and data management module.

[0105] The model and data management module 400 is used to periodically synchronize incremental result data to the Hive repository, send the credit assessment model to the TensorFlow engine, and also send the updated credit assessment model to the stream computing module.

[0106] The stream computing module 500 is used to periodically pull pre-trained credit assessment models from the model and data management module, and simultaneously obtain the credit data to be predicted provided by the Kafka module in real time, perform DAG calculation, obtain the assessment results, and send the assessment results to the assessment service module; at the same time, the assessment results are sent as incremental result data to the model and data management module.

[0107] The evaluation service module 600 is used to query the evaluation results obtained by the stream computing module, and can also query historical data to perform relevant statistical analysis.

[0108] The credit scoring intelligent assessment system based on a batch-stream integrated computing engine, as described in this application embodiment, includes: a Kafka module, a Hive repository module, a TensorFlow engine module, a model and data management module, a stream computing module, and an assessment service module. The Kafka module is used to access user credit scoring data from multiple dimensions in real time, and divide the user credit scoring data into training data and prediction data according to topics. The training data is synchronized to the Hive repository for storage in real time, and the prediction data is sent to the stream computing module. The Hive repository module stores training data and sends the training data to the TensorFlow engine module. It also stores incremental result data sent by the model and data management module and performs data analysis based on the incremental result data. The TensorFlow engine module receives... The system trains and updates the credit assessment model, sending the updated model to the model and data management module. The model and data management module periodically synchronizes incremental results to the Hive repository, sends the credit assessment model to the TensorFlow engine, and sends the updated model to the stream computing module. The stream computing module periodically pulls the pre-trained credit assessment model from the model and data management module, while simultaneously acquiring real-time credit data to be predicted from the Kafka module, performing DAG calculations to obtain assessment results, and sending these results as incremental results to the assessment service module. The assessment results are also sent to the model and data management module. The assessment service module queries the assessment results obtained from the stream computing module and can also query historical data for relevant statistical analysis. Therefore, this method can solve the technical problems of inaccurate and unstable credit scoring in existing intelligent credit assessment methods. It uses classic neural network model algorithms to dynamically predict intelligent credit scores and can achieve online learning of the assessment model based on real-time streaming credit data. At the same time, it can update the model assessment effect by combining historical patterns and real-time changes, eliminating model instability and thus improving the accuracy of intelligent credit assessment.

[0109] This application proposes a credit scoring intelligent assessment system based on the Flink batch-stream integrated engine. The system uses Kafka for unified real-time access of credit scoring data and Hive warehouse for offline storage of credit scoring data. At the same time, a model and data management module is designed. By leveraging the characteristics of Flink batch-stream integrated semantics, the dynamic updating of the model is guaranteed.

[0110] Figure 7 This is a flowchart of the credit intelligent assessment system based on the batch-stream integrated computing engine, which is an embodiment of this application.

[0111] like Figure 7 As shown, the credit assessment system process includes: real-time access of relevant credit data to the Kafka module, with data uniformly divided by topic, including training data and credit data to be predicted; real-time synchronization of training data in the Kafka module to the Hive repository module to ensure the integrity of the training data; the raw training set in the Hive repository module is preprocessed by Flink and then fed into the TensorFlow engine module for deep learning training to update the credit assessment model, while the Hive repository stores the incremental result data sent by the model and data management module, and performs data analysis based on the incremental result data; the Flink stream processing module periodically pulls the model from the management module (for updates), and simultaneously receives data in real time. The system receives real-time credit prediction data from the Kafka module, performs DAG calculations, and finally sends the results to the evaluation service module. Simultaneously, it outputs the predicted results as incremental data to the model and data management module. The evaluation service module can query historical data for statistical analysis and also queries the credit assessment scores generated by the Flink stream computing module. The model and data management module periodically synchronizes incremental results data to the Hive repository module and sends the current model to the TensorFlow engine module. After receiving the training data and model, the TensorFlow engine module begins training the new model, and the trained model is then pushed back to the model and data management module, ultimately achieving dynamic model updates.

[0112] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the credit intelligent assessment method based on a batch-stream integrated computing engine as described in the above embodiments.

[0113] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0114] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0115] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0116] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0117] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0118] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0119] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0120] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A credit assessment method based on a batch-stream integrated computing engine, characterized in that, Includes the following steps: User credit data from multiple dimensions is obtained through Kafka, and the user credit data is divided into credit data to be trained and credit data to be predicted according to the topic. A pre-trained credit assessment model is obtained, and the credit data to be predicted is input into the credit assessment model for real-time assessment to obtain the assessment result. The credit assessment model is dynamically updated by incrementing the original training data using the credit data to be trained and then using the incremented training data. The dynamic updating of the credit assessment model includes: Incremental training data is obtained by adding data increments to the original training data using the credit data to be trained. When the incremental training data reaches the preset quantity requirement or a timing condition is triggered, the incremental training data is preprocessed and the preprocessed incremental training data is sent to the TensorFlow server, and the credit assessment model is sent to the TensorFlow server at the same time. Based on incremental training data, the credit assessment model is trained and updated using a TensorFlow server to obtain the trained credit assessment model. The updated credit assessment model is represented as follows: M new =train(M old ,I inc ) Among them, M new M represents the updated model. old I represents the old model from the previous stage. inc This represents the incremental training data from the previous stage to the current stage.

2. The method as described in claim 1, characterized in that, The process involves acquiring user credit data from multiple dimensions using Kafka and then dividing this data into training data and prediction data based on topics. The user credit data from multiple dimensions is synchronized using DataX, a tool that supports heterogeneous data source synchronization. The user credit data is then processed by ETL, and the ETL-processed user credit data is uniformly aggregated into a Kafka message queue. The user credit data in the Kafka message queue is divided into credit data to be trained and credit data to be predicted according to the topic. The credit data to be trained is synchronized to the Hive repository in real time for storage.

3. The method as described in claim 2, characterized in that, The method further includes: The raw training data is obtained through Kafka, and then synchronized to the Hive repository in real time for storage. The raw training data is preprocessed, and then sent to the TensorFlow server via remote RPC to build a credit assessment model based on the raw training data.

4. The method as described in claim 3, characterized in that, The training credit data and the original training data are synchronized to the Hive repository in real time for storage, including: The training data is synchronized from Kafka to the Hive repository in real time using the Two Phase Commit protocol. The training data includes credit data to be trained and original training data.

5. The method as described in claim 4, characterized in that, The process of using the Two Phase Commit protocol to synchronize training data from Kafka to a Hive repository in real time for storage includes: The Checkpoint module of the job manager periodically sends Checkpoint barriers to the data source nodes, so that the Checkpoint barriers flow sequentially into the Window and Data Sink nodes in the direction of data flow. After the Window and Data Sink nodes receive the Checkpoint barrier, they stop the calculation and save the state snapshot to the highly available persistent storage component. Then, the Window and Data Sink nodes send an ACK confirmation to the Checkpoint module of the job manager. Once all nodes have completed saving their state snapshots, they pre-commit external transactions to the Hive repository. The Hive repository then writes the data to the specified location and sends an ACK confirmation to the job manager.

6. The method as described in claim 5, characterized in that, The method of using the Two Phase Commit protocol to synchronize training data from Kafka to a Hive repository in real time for storage also includes: Once the job manager receives all ACK confirmations, it sends a status snapshot completion notification to each node. Simultaneously, the data sink node sends a commit command to the Hive repository via RPC. The Hive repository's transaction module executes the commit operation, completing the actual data writing.

7. A credit assessment device based on a batch-stream integrated computing engine, characterized in that, This includes an acquisition module and an evaluation module, among which... The acquisition module is used to acquire user credit data from multiple dimensions through Kafka, and divide the user credit data into credit data to be trained and credit data to be predicted according to the topic. An evaluation module is used to obtain a pre-trained credit assessment model, input the credit data to be predicted into the credit assessment model for real-time evaluation, and obtain the evaluation result. The credit assessment model is dynamically updated by incrementing the original training data with the credit data to be trained and then using the incremented training data. The dynamic updating of the credit assessment model includes: Incremental training data is obtained by adding data increments to the original training data using the credit data to be trained. When the incremental training data reaches the preset quantity requirement or a timing condition is triggered, the incremental training data is preprocessed and the preprocessed incremental training data is sent to the TensorFlow server, and the credit assessment model is sent to the TensorFlow server at the same time. Based on incremental training data, the credit assessment model is trained and updated using a TensorFlow server to obtain the trained credit assessment model. The updated credit assessment model is represented as follows: M new =train(M old ,I inc ) Among them, M new M represents the updated model. old I represents the old model from the previous stage. inc This represents the incremental training data from the previous stage to the current stage.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images