Credit investigation data integrated acquisition, processing and display system and method and storage medium

Through the integrated collection, processing and display system of credit reporting data, the automated collection, processing and modeling of data is realized, the problem of inefficient traditional data processing is solved, the efficiency and security of data integration and sharing are improved, and the value of data is realized. Maximize utilization.

CN119046362APending Publication Date: 2024-11-29GUANGDONG YUECAI CREDIT INFORMATION CO LTD

Patent Information

Application Number
CN202411153288.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

Traditional data processing methods are inefficient, prone to errors, difficult to integrate and mine decentralized data from governments or group enterprises, lack automation and security sharing mechanisms, affecting the value utilization of data.

Method used

A integrated collection, processing and display system for credit reporting data is designed, including data acquisition, processing, modeling and sharing modules. Through automated collection, memory processing, machine learning modeling and secure sharing, unified data standardization and multi-source integration are realized. , supports a variety of machine learning algorithms and secure sharing mechanisms.

Benefits of technology

It improves data collection and processing efficiency, reduces manual intervention, ensures data accuracy and security, maximizes data value, provides online monitoring and display, and solves the problems of data integration and sharing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046362A_ABST
    Figure CN119046362A_ABST
Patent Text Reader

Abstract

The invention provides a credit investigation data integrated acquisition, processing and display system and method and a storage medium, and relates to the technical field of data processing, the credit investigation data integrated acquisition, processing and display system comprises a data acquisition module, a data processing module, a data modeling module, a data sharing module and a task scheduling module; the system provides centralized management and automatic implementation of data, has a self-adaptive adjustment mechanism for multi-source data acquisition, can automatically optimize an acquisition strategy according to the characteristics of different data sources, has an intelligent error recognition and correction mechanism in a data processing process, reduces manual intervention, improves the accuracy of data processing, and improves the working efficiency. A flexible data modeling tool is provided, various machine learning algorithms are supported, a data model can be rapidly constructed and adjusted according to requirements, a safe data sharing mechanism is provided, the integrity and privacy protection of data in the sharing process are ensured, flow visualization display is achieved, and the problems occurring in the data implementation process are solved; and an unsupervised condition in an automatic process is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a credit data integrated collection, processing and display system, method and storage medium. Background Art

[0002] With the advent of the big data era, data collection, processing, modeling and sharing have become particularly important. Traditional data processing methods rely on manual operations, which are inefficient and prone to errors. For government departments or group companies, data is scattered across various data sites, making it difficult to effectively integrate, mine and develop data.

[0003] Therefore, an automated solution is needed to improve the efficiency and accuracy of data collection and processing, while ensuring the security and shareability of data, solving the problem of data resource integration, and mining more data value through data development and utilization to assist the digital transformation of governments or group enterprises. Therefore, the present invention proposes a credit data integrated collection, processing and display system, method and storage medium to solve the problems existing in the prior art. Summary of the invention

[0004] In response to the above problems, the present invention proposes a system, method and storage medium for the integrated collection, processing and display of credit data. The system, method and storage medium for the integrated collection, processing and display of credit data can effectively collect credit industry data from heterogeneous data sources, process them into a unified credit data standard, and enable the credit data to be effectively applied in models.

[0005] To achieve the purpose of the present invention, the present invention is implemented through the following technical solutions: a credit data integrated collection, processing and display system, including a data collection module, a data processing module, a data modeling module, a data sharing module and a task scheduling module, the data collection module is used to adjust the collection task according to preset rules and data source parameters to automatically collect credit data from different data sources; the data processing module is used to perform memory processing and dumping on the collected raw data, identify erroneous data therein, and integrate the data that meets the requirements into standardized data;

[0006] The data modeling module uses machine learning technology to build a credit data model for various application scenarios based on the results of data processing; the data sharing module is used to provide display services and multi-dimensional sharing of the data that flows through the above modules through a data sharing platform, and perform security processing during data transmission and access; the task scheduling module is used to schedule and manage the operations of the above modules, and complete the automated operation of the above module operations through trigger timing or manual scheduling.

[0007] Further improvements are as follows: the data collection module automatically collects data from different data sources through databases, web crawlers, and API interfaces, including credit data from government data sharing channels, public credit information data pages on government websites, shared documents, and data sharing interfaces. The collection tasks are adjusted according to preset rules and data source parameters to automatically complete data collection.

[0008] The further improvement is that: the acquisition process of the data acquisition module is as follows: confirm the data acquisition method, including database, website, data document, interface document; configure the data source, and configure the corresponding parameters according to the confirmed data acquisition method, including database connection information and account password, website address, data document, interface access address and account password; configure the database connection information and account password of the target library, and establish the corresponding acquisition data storage table according to the required data structure; create acquisition information, describe the content overview of this acquisition, and select the configured database information as the target library; configure the mapping relationship between the acquisition source data table and the target library data table, and its parameters include field name, assignment type, assignment, processing script, and field security level division; create an acquisition task, fill in the trigger type, including manual and automatic, when automatic, configure the task scheduling time parameters, and the scheduling time is expressed in the form of Cron expression; configure based on the created acquisition task, add multiple acquisition information to the same task, and configure its stop mechanism, and its parameter values ​​include all execution failure and partial execution failure; manually execute the task for testing or automatically schedule it through the system according to the cron expression configured by the acquisition task; check the acquisition log to confirm the storage acquisition result.

[0009] A further improvement is that the data processing module uses data cleaning, conversion, and normalization techniques to perform memory processing and dump the collected raw data, automatically identifies erroneous data through preset rules, and integrates data that meets the requirements through governance procedures to form standardized data.

[0010] Further improvement is that: the processing process of the data processing module is as follows: create a data verification job, and specify the target library specified by the data acquisition module as the data source of the data verification job; select the source data table corresponding to the configured data source as the verification table; configure verification rules for the fields in the selected verification table, and the rules include: date verification, value verification, non-empty verification, primary key verification, special character verification, identity card compliance verification, unified social credit code compliance verification, double publicity-administrative license verification, double publicity-administrative penalty verification; create a governance job, and specify the target library specified by the data acquisition module as the data source of the data verification job; configure the governance job, and specify the configured verification table as the governance table, and the configuration parameters include table mapping configuration, such as target character verification, Segment, field value, special processing of source table fields, source table master table, related table, sorting sequence number, sorting method, whether it is primary key, whether it is valid, and filter condition configuration, including filter type, source table field, comparison operation, content input; create governance tasks, fill in the trigger type, including manual and automatic. When automatic, configure the task scheduling time parameters, and the scheduling time is expressed in the form of Cron expression; based on the created governance tasks, configure, add multiple verification and governance job information to the same task, and configure its stop mechanism, and its parameter values ​​include complete execution failure and partial execution failure; manually execute tasks for testing or automatically schedule them through the system according to the cron expression configured by the collection task; view the scheduling log to confirm the governance results.

[0011] Further improvements are as follows: the data modeling module uses machine learning technology to build a credit data model based on the results of data processing. The model is used for predictive analysis, classification, clustering and other application scenarios. The specific process is as follows: create a model and read the model type from the storage medium, including scorecard, classification, regression, clustering, strategy mining, and fill in the model name and description field information; enter the reading and configuration of the model data set, obtain the database and data table to which the data set belongs from the storage device, and select and configure the model Y value; perform data rule configuration processing on the selected data table, and the data processing rule components include missing rate cleaning, encoding, missing value filling, duplicate value removal, outlier processing, and feature deletion. The rule component configuration is performed according to the credit model business and the selected data situation, and different components correspond to different parameter configuration formats; enter the credit model feature engineering component, and the component configuration includes normalization, standardization, feature selection, feature cross-combination, feature grouping aggregation, feature binning, WOE coding, IV value screening, correlation screening, principal component analysis, and feature contribution screening. Different components are selected according to the credit model business, and the configuration is different The same parameter configuration format; divide the credit model data into data sets, including the division of training sets and validation sets. For data with unbalanced samples, upsampling and downsampling are performed to adjust the ratio of positive and negative samples. According to the set training set ratio, the data size of the training set and validation set is automatically calculated and statistically analyzed; according to the business needs of the credit model, algorithm rules are selected from the memory. The algorithm rules include logistic regression, decision tree, XGB classification, K-means clustering, linear regression, decision tree, bp neural network, DBSCAN clustering, and different types of models select different algorithm components; save the model configuration data to the memory database, obtain the memory model information through the program trigger, and train the model. During the training process, the trigger returns the model execution log in real time according to the program running status and displays it on the user's display. After the model training is completed, the trigger returns the accuracy, confusion matrix, auc and roc curves, and ks curve data to the display; read the model list in the model management memory, select one of the models in the display interface for data prediction and analysis, and save the structure data to the database memory.

[0012] Further improvements are as follows: the data sharing module is used to provide external display services for the data passing through the data acquisition module, data processing module, and data modeling module through the data sharing platform, and provides sharing methods such as API calls, data exports, and library table docking, while simultaneously maintaining the security of data during transmission and access.

[0013] The further improvement lies in: the specific operation process of the data sharing module is as follows: create a shared display job, read the shared data source from the memory, select the sharing type and fill in the basic configuration information of the shared job name, where the sharing type includes data file export, interface API call, and library table docking; when the sharing type selects the data export method, according to the security regulations, the table data is encrypted and the file password is set, the encryption method includes SM4 encryption, symmetric encryption and asymmetric encryption, and after the data file is exported, it is sent to the other party through email or other methods; when the sharing type selects API call, the interface API service is provided, including interface authorization, interface data format encoding, and according to the security regulations, the interface transmission data is encrypted and signed and verified, the encryption method includes SM4 encryption, symmetric encryption and asymmetric encryption; when the sharing type selects library table docking, the data in the memory is first encrypted through a trigger, and then the data is pushed to the other party's memory.

[0014] A method for integrated collection, processing and display of credit information data, comprising the following steps:

[0015] S1: According to the preset rules and data source parameter adjustments, set the collection tasks, and automatically collect data from different data sources through databases, web crawlers, and API interfaces. The scope includes credit data from government data sharing channels, public credit information data pages on government websites, shared documents, and data sharing interfaces;

[0016] S2: Use data cleaning, conversion, and normalization technologies to process and dump the collected raw data in memory, automatically identify erroneous data through preset rules, and integrate the data that meets the requirements into standardized data through governance procedures;

[0017] S3: Based on the results of data processing, use machine learning technology to build a credit data model and apply the model to various scenarios, including predictive analysis, classification, and clustering;

[0018] S4: Based on the sharing method of API call, data export, and database table docking, the data that flows through the above steps is provided to the outside through the data sharing platform for display services, and is securely encrypted during the data transmission and access process;

[0019] S5: Set triggers and complete the automation of the above operations through trigger timing or manual execution scheduling.

[0020] A storage medium, wherein a computer program stored in the storage medium, when executed by one or more processors, implements the aforementioned method for integrated collection, processing and display of credit information data.

[0021] The beneficial effects of the present invention are:

[0022] 1. The present invention provides centralized management and automated implementation of data sources, data processing, data modeling, and data applications. It has an adaptive adjustment mechanism for multi-source data collection, can automatically optimize collection strategies according to the characteristics of different data sources, has an intelligent error recognition and correction mechanism in the data processing process, reduces manual intervention, and improves the accuracy of data processing. It has flexible data modeling tools, supports a variety of machine learning algorithms, can quickly build and adjust data models according to needs, has a secure data sharing mechanism, ensures the integrity and privacy protection of data during the sharing process, has a process visualization display, solves problems arising in the data implementation process, and avoids unsupervised automation processes.

[0023] 2. The present invention provides unified management of data sources, supports multi-data source integration technology, unified processing of data according to service requirements, in-depth analysis and modeling training applications for data mining, and automated management of unified authorization and distribution services for data applications. It also provides online display technology to monitor data flows and operations and solve the black box problem in the data implementation process.

[0024] 3. The present invention improves the efficiency and quality of data collection and processing, and develops and utilizes data through data mining and machine learning analysis, thereby maximizing the value of data, while ensuring the security and convenient sharing of data, and providing the function of online real-time monitoring of data collection and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart for the automated data task operation of the present invention;

[0026] Figure 2 A flow chart of machine learning modeling for the present invention;

[0027] Figure 3 It is a system configuration block diagram of the present invention. DETAILED DESCRIPTION

[0028] In order to deepen the understanding of the present invention, the present invention will be further described in detail below in conjunction with examples. The examples are only used to explain the present invention and do not constitute a limitation on the protection scope of the present invention.

[0029] Embodiment 1

[0030] according to Figure 1 , 2As shown in , 3, this embodiment proposes a credit data integrated collection, processing and display system, including a data collection module, a data processing module, a data modeling module, a data sharing module and a task scheduling module, the data collection module is used to adjust the collection task according to the preset rules and data source parameters to automatically collect credit data from different data sources; the data processing module is used to perform memory processing and dump the collected raw data, identify the erroneous data therein, and integrate the data that meets the requirements into standardized data;

[0031] The data modeling module uses machine learning technology to build a credit data model for various application scenarios based on the results of data processing; the data sharing module is used to provide display services and multi-dimensional sharing of the data that flows through the above modules through a data sharing platform, and perform security processing during data transmission and access; the task scheduling module is used to schedule and manage the operations of the above modules, and complete the automated operation of the above module operations through trigger timing or manual scheduling.

[0032] The data collection module automatically collects data from different data sources through databases, web crawlers, and API interfaces. The scope includes credit data from government data sharing channels, public credit information data pages on government websites, shared documents, and data sharing interfaces. The collection tasks are adjusted according to preset rules and data source parameters to automatically complete data collection.

[0033] The acquisition process of the data acquisition module is as follows: confirm the data acquisition method, including database, website, data document, interface document; configure the data source, and configure the corresponding parameters according to the confirmed data acquisition method, including database connection information and account password, website address, data document, interface access address and account password; configure the database connection information and account password of the target library, and establish the corresponding acquisition data storage table according to the required data structure; create acquisition information, describe the content overview of this acquisition, and select the configured database information as the target library; configure the mapping relationship between the acquisition source data table and the target library data table, and its parameters include field name, assignment type, assignment, processing script, and field security level classification; create an acquisition task, fill in the trigger type, including manual and automatic. When automatic, configure the task scheduling time parameters, and the scheduling time is expressed in the form of Cron expression; configure based on the created acquisition task, add multiple acquisition information to the same task, and configure its stop mechanism, and its parameter values ​​include all execution failure and partial execution failure; manually execute the task for testing or automatically schedule it through the system according to the cron expression configured by the acquisition task; check the acquisition log to confirm the storage acquisition result.

[0034] The data processing module uses data cleaning, conversion, and normalization technologies to perform memory processing and dump the collected raw data, automatically identifies erroneous data through preset rules, and integrates the data that meets the requirements through governance procedures to form standardized data.

[0035] The processing process of the data processing module is as follows: create a data verification job, and specify the target library specified by the data acquisition module as the data source of the data verification job; select the source data table corresponding to the configured data source as the verification table; configure verification rules for the fields in the selected verification table, and the rules include: date verification, value verification, non-empty verification, primary key verification, special character verification, ID card compliance verification, unified social credit code compliance verification, double publicity-administrative license verification, double publicity-administrative penalty verification; create a governance job, and specify the target library specified by the data acquisition module as the data source of the data verification job; configure the governance job, and specify the configured verification table as the governance table, and the configuration parameters include table mapping configuration, such as target field, field Value extraction, special processing of source table fields, source table master table, related table, sorting sequence number, sorting method, whether it is primary key, whether it is valid, and filter condition configuration, including filter type, source table fields, comparison operations, and content input; create a governance task, fill in the trigger type, including manual and automatic. When automatic, configure the task scheduling time parameters, and the scheduling time is expressed in the form of Cron expressions; based on the created governance task, configure it, add multiple verification and governance job information to the same task, and configure its stop mechanism, and its parameter values ​​include complete execution failure and partial execution failure; manually execute the task for testing or automatically schedule it through the system according to the cron expression configured by the collection task; view the scheduling log to confirm the governance result.

[0036] The data modeling module uses machine learning technology to build a credit data model based on the results of data processing. The model is used for predictive analysis, classification, clustering and other application scenarios. The specific process is as follows: create a model and read the model type from the storage medium, including scorecard, classification, regression, clustering, strategy mining, and fill in the model name and description field information; enter the reading and configuration of the model data set, obtain the database and data table to which the data set belongs from the memory, and select and configure the model Y value; perform data rule configuration processing on the selected data table. The data processing rule components include missing rate cleaning, encoding, missing value filling, duplicate value removal, outlier processing, and feature deletion. The rule component configuration is performed according to the credit model business and the selected data situation. Different components correspond to different parameter configuration formats; enter the credit model feature engineering component. The component configuration includes normalization, standardization, feature selection, feature cross-combination, feature grouping aggregation, feature binning, WOE coding, IV value screening, correlation screening, principal component analysis, and feature contribution screening. Different components are selected according to the credit model business and different parameters are configured. Configuration format; divide the credit model data into data sets, including training sets and validation sets. For data with unbalanced samples, upsampling and downsampling are performed to adjust the ratio of positive and negative samples. According to the set training set ratio, the data size of the training set and validation set is automatically calculated and statistically analyzed; according to the business needs of the credit model, algorithm rules are selected from the memory. The algorithm rules include logistic regression, decision tree, XGB classification, K-means clustering, linear regression, decision tree, bp neural network, DBSCAN clustering, and different types of models select different algorithm components; save the model configuration data to the memory database, obtain the memory model information through the program trigger, and train the model. During the training process, the trigger returns the model execution log in real time according to the program running status and displays it on the user's display. After the model training is completed, the trigger returns the accuracy, confusion matrix, auc and roc curves, and ks curve data to the display; read the model list in the model management memory, select one of the models in the display interface for data prediction and analysis, and save the structural data to the database memory.

[0037] The data sharing module is used to provide external display services for the data passing through the data acquisition module, data processing module, and data modeling module through the data sharing platform, and provide sharing methods such as API calls, data exports, and library table docking, while simultaneously maintaining the security of data during transmission and access.

[0038] The specific operation process of the data sharing module is as follows: create a shared display job, read the shared data source from the memory, select the sharing type and fill in the basic configuration information of the shared job name, where the sharing types include data file export, interface API call, and library table docking; when the sharing type selects the data export method, according to the security regulations, the table data is encrypted and the file password is set, the encryption method includes SM4 encryption, symmetric encryption and asymmetric encryption, and after the data file is exported, it is sent to the other party through email or other methods; when the sharing type selects API call, the interface API service is provided, including interface authorization, interface data format encoding, and according to the security regulations, the interface transmission data is encrypted, signed and verified, and the encryption methods include SM4 encryption, symmetric encryption and asymmetric encryption; when the sharing type selects library table docking, the data in the memory is first encrypted through a trigger, and then the data is pushed to the other party's memory.

[0039] A method for integrated collection, processing and display of credit information data, comprising the following steps:

[0040] According to the preset rules and data source parameter adjustments, set the collection tasks, and automatically collect data from different data sources through databases, web crawlers, and API interfaces, including credit data from government data sharing channels, public credit information data pages on government websites, shared documents, and data sharing interfaces;

[0041] Use data cleaning, conversion, and normalization technologies to process and dump the collected raw data in memory, automatically identify erroneous data through preset rules, and integrate the data that meets the requirements into standardized data through governance procedures;

[0042] Based on the results of data processing, we use machine learning technology to build a credit data model and apply the model to various scenarios, including predictive analysis, classification, and clustering;

[0043] Based on the sharing method of API call, data export, and database table docking, the data that flows through the above steps is provided to the outside through the data sharing platform for display services, and is securely encrypted during the data transmission and access process;

[0044] Set triggers and complete the automation of the above operations through trigger timing or manual scheduling.

[0045] A storage medium, wherein a computer program stored in the storage medium, when executed by one or more processors, implements the aforementioned method for integrated collection, processing and display of credit information data.

[0046] Embodiment 2

[0047] according to Figure 1 , 2As shown in Figure 3, this embodiment proposes a credit data integrated collection, processing and display system, method and storage medium to improve the efficiency and quality of data collection and processing, and to develop and utilize data through data mining and machine learning analysis to maximize the value of data, while ensuring the security and convenient sharing of data, and monitoring the data collection and processing status online in real time.

[0048] 1. Data collection module: This module automatically collects data from different data sources through various collection methods such as databases, web crawlers, API interfaces, etc. Its scope includes credit data from government data sharing channels, public credit information data pages on government websites, shared documents, data sharing interfaces, etc. The collection task can be adjusted according to preset rules and data source parameters to automatically complete data collection.

[0049] The collection process is as follows:

[0050] 1) Confirm the data collection method, such as database, website, data document, interface document;

[0051] 2) Configure the data source. According to the data collection method confirmed in step 1), configure the corresponding parameters, such as database connection information and account password, website address, data document, interface access address and account password, etc.;

[0052] 3) Configure the database connection information and account password of the target database, and establish the corresponding collection data storage table according to the required data structure;

[0053] 4) Create collection information, describe the content overview of this collection, and select the database information configured in step 3) as the target database;

[0054] 5) Configure the one-to-one mapping relationship between the source data table and the target database data table. Its parameters include at least field name, assignment type, assignment, processing script, field security level classification, etc.

[0055] 6) Create a collection task and fill in the trigger type, which can be manual or automatic. If it is automatic, you also need to configure the task scheduling time parameters. The scheduling time is expressed in the form of Cron expression.

[0056] 7) Based on the collection task created in step 6), configure the collection task. Multiple collection information can be added to the same task, and its stop mechanism can be configured. Its parameter values ​​include complete execution failure, partial execution failure, etc.

[0057] 8) Manually execute tasks for testing or automatically schedule them through the system based on the cron expressions configured for the collection tasks.

[0058] 9) Check the acquisition log to confirm the storage acquisition results.

[0059] 2. Data processing module: This module uses data cleaning, conversion, normalization and other technologies to process and dump the collected raw data in memory, automatically identify erroneous data through preset rules, and integrate the data that meets the requirements through governance procedures to form standardized data.

[0060] The main steps are as follows:

[0061] 1) Create a data verification job and specify the target library specified by the data acquisition module as the data source of the data verification job;

[0062] 2) Select the source data table corresponding to the data source configured in step 1) as the verification table;

[0063] 3) Configure verification rules for the fields in the verification table selected in step 2) one by one, including: date verification, value verification, non-empty verification, primary key verification, special character verification, ID card compliance verification, unified social credit code compliance verification, double publicity-administrative license verification, double publicity-administrative penalty verification, etc.;

[0064] 4) Create a governance job and specify the target library specified by the data acquisition module as the data source for the data verification job;

[0065] 5) Configure the governance job and specify the validation table configured in step 3) as the governance table. The configuration parameters include table mapping configuration, such as target fields, field values, special processing of source table fields, source table master table, related tables, sorting sequence number, sorting method, whether it is a primary key, whether it is valid, etc., as well as filtering condition configuration, such as filtering type, source table fields, comparison operations, content input, etc.

[0066] 6) Create a governance task and fill in the trigger type, which can be manual or automatic. If it is automatic, you also need to configure the task scheduling time parameters. The scheduling time is expressed in the form of Cron expression.

[0067] 7) Based on the governance task created in step 6), configure the governance task. You can add multiple verification and governance job information to the same task, and configure its stop mechanism. Its parameter values ​​include complete execution failure, partial execution failure, etc.

[0068] 8) Manually execute tasks for testing or automatically schedule them through the system based on the cron expressions configured for the collection tasks.

[0069] 9) Check the scheduling log to confirm the governance results.

[0070] 3. Data modeling module: Based on the results of data processing, this module uses machine learning technology to build a credit data model. The model can be used in a variety of application scenarios such as predictive analysis, classification, and clustering.

[0071] The steps are as follows:

[0072] 1) Create a model and read the model type from the storage medium, such as scorecard, classification, regression, clustering, strategy mining, and fill in field information such as model name and description.

[0073] 2) After completing step 1), enter the reading and configuration of the model data set, obtain the database and data table to which the data set belongs from the memory, and select and configure the model Y value.

[0074] 3) After completing step 2), the selected data table is processed by data rule configuration. The data processing rule components include missing rate cleaning, encoding, missing value filling, duplicate value removal, outlier processing, and feature deletion. The rule components are configured according to the credit model business and the selected data. Different components correspond to different parameter configuration formats.

[0075] 4) After completing step 3), enter the credit model feature engineering component. The component configuration includes normalization, standardization, feature selection, feature cross-combination, feature grouping aggregation, feature binning, WOE coding, IV value screening, correlation screening, principal component analysis, and feature contribution screening. Select different components according to the credit model business and configure different parameter configuration formats.

[0076] 5) After completing step 4), the credit model data is divided into data sets, including training sets and validation sets. In addition, for data with unbalanced samples, upsampling and downsampling can be performed to adjust the ratio of positive and negative samples to better adapt to the training and learning of the model. According to the set training set ratio, the data size of the training set and validation set is automatically calculated.

[0077] 6) After completing step 5), select algorithm rules from the storage according to the business requirements of the credit model. The algorithm rules include logistic regression, decision tree, XGB classification, K-means clustering, linear regression, decision tree, bp neural network, and DBSCAN clustering. Different types of models select different algorithm components.

[0078] 7) After completing the above 6 steps, the configuration data of the model is saved in the memory database. The memory model information can be obtained through the program trigger to train the model. During the training process, the trigger will return the model execution log in real time according to the program running status and display it on the user's monitor. After the model training is completed, the trigger returns the accuracy, confusion matrix, AUC and ROC curves, and KS curve data to the display.

[0079] 8) Model prediction: Read the model list in the model management memory, select one of the models in the display interface to perform data prediction and analysis, and save the structural data to the database memory.

[0080] 4. Data sharing module: The modeled, collected or processed data is provided to the outside world through a secure data sharing platform. The platform supports multiple sharing methods such as API calls, data export, and database table docking, and ensures the security of data during transmission and access.

[0081] The steps are as follows:

[0082] 1) Create a shared display job, read the shared data source from the storage, select the sharing type and fill in the basic configuration information such as the sharing job name. The sharing types include data file export, interface API call, and library table docking. The display methods include: API interface display, data export display, and online visualization display.

[0083] 2) If you select "Data Export" as the sharing type in step 1), the system will encrypt the table data and set the file password according to the security regulations. The encryption methods include SM4 encryption, symmetric encryption and asymmetric encryption. After the data file is exported, it will be sent to the other party through email or other means.

[0084] 3) If you select "API call" as the sharing type in step 1), the system provides interface API services, such as interface authorization, interface data format encoding, etc., and encrypts and verifies the data transmitted through the interface according to security regulations. The encryption methods include SM4 encryption, symmetric encryption and asymmetric encryption.

[0085] 4) If you select "library-table connection" as the sharing type in step 1), the system will first encrypt the data in the storage through a trigger, and then push the data to the other party's storage.

[0086] 5. Task scheduling module: The above 1-4 module jobs are scheduled and managed through the task scheduling module, and the automatic operation of 1-4 module jobs can be completed through trigger timing or manual execution scheduling.

[0087] Data flow description: External comprehensive data flows into the data acquisition module from the external database or interface. After being stored in the data acquisition module and filtered and classified through triggers, it flows to the data acquisition processing module. After being stored in the data processing module and verified and filtered through data processing rules, it flows to the data modeling module. After being stored in the data modeling module and trained and analyzed through models, it flows to the data sharing module. After being stored in the data sharing module and encrypted through data security, it flows to the outside.

[0088] The system trigger program automatically identifies the type of external data source by collecting configuration information. During the data collection process, the program automatically optimizes the data collection frequency, concurrency number, and data reading and writing speed strategy to ensure that the data is stored smoothly in the memory.

[0089] Principle of data processing error recognition: Brush through configured rule keywords, data value upper and lower limits, encoding generation rules, etc.

[0090] Intelligent correction principle: First, the system filters out non-compliant data according to data processing rules, and then removes non-compliant data through triggers or manually removes non-compliant data through the operation interface.

[0091] The present invention provides centralized management and automated implementation of data sources, data processing, data modeling, and data applications, and has an adaptive adjustment mechanism for multi-source data acquisition, which can automatically optimize the acquisition strategy according to the characteristics of different data sources, and has an intelligent error recognition and correction mechanism in the data processing process, which reduces manual intervention and improves the accuracy of data processing. It has flexible data modeling tools, supports a variety of machine learning algorithms, can quickly build and adjust data models according to needs, has a secure data sharing mechanism, ensures the integrity and privacy protection of data during the sharing process, has a process visualization display, solves problems that arise in the data implementation process, and avoids the situation where the automation process is unsupervised. In addition, the present invention provides unified management of data sources, supports multi-data source integration technology, uniformly processes data according to service requirements, performs in-depth analysis and modeling training applications on data mining, and uniformly authorizes and distributes data applications. It also provides online display technology to monitor data flows and operations and solve the black box problem in the data implementation process. At the same time, the present invention improves the efficiency and quality of data collection and processing, and develops and utilizes data through data mining and machine learning analysis, etc., to maximize the value of data, while ensuring the security and convenient sharing of data, and provides online real-time monitoring of data collection and processing.

[0092] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A credit data integrated collection, processing and display system, including a data collection module, a data processing module, a data modeling module, a data sharing module and a task scheduling module, characterized in that: The data collection module is used to adjust the collection tasks according to preset rules and data source parameters to automatically collect credit data from different data sources; the data processing module is used to perform memory processing and dump the collected raw data, identify erroneous data therein, and integrate the data that meets the requirements into standardized data; The data modeling module uses machine learning technology to build a credit data model for various application scenarios based on the results of data processing; the data sharing module is used to provide display services and multi-dimensional sharing of the data that flows through the above modules through a data sharing platform, and perform security processing during data transmission and access; the task scheduling module is used to schedule and manage the operations of the above modules, and complete the automated operation of the above module operations through trigger timing or manual scheduling.

2. The integrated credit data collection, processing and display system according to claim 1, characterized in that: The data collection module automatically collects data from different data sources through databases, web crawlers, and API interfaces. The scope includes credit data from government data sharing channels, public credit information data pages on government websites, shared documents, and data sharing interfaces. The collection tasks are adjusted according to preset rules and data source parameters to automatically complete data collection.

3. The integrated credit data collection, processing and display system according to claim 2 is characterized by: The data acquisition module has the following collection process: confirm the data collection method, including database, website, data document, interface document; configure the data source, and configure the corresponding parameters according to the confirmed data collection method, including database connection information and account password, website address, data document, interface access address and account password; configure the database connection information and account password of the target library, and establish the corresponding collection data storage table according to the required data structure; Create collection information, describe the content overview of this collection, and select the configured database information as the target database; Configure the mapping relationship between the source data table and the target database table. The parameters include field name, value type, value assignment, processing script, and field security level classification. Create a collection task and fill in the trigger type, including manual and automatic. When automatic, configure the task scheduling time parameters. The scheduling time is expressed in the form of Cron expression. Based on the created collection task, configure it, add multiple collection information to the same task, and configure its stop mechanism. Its parameter values ​​include complete execution failure and partial execution failure. Manually execute the task for testing or automatically schedule it through the system according to the cron expression configured for the collection task. View the collection log to confirm the storage collection results.

4. The credit data integrated collection, processing and display system according to claim 1, characterized in that: The data processing module uses data cleaning, conversion, and normalization technologies to perform memory processing and dump the collected raw data, automatically identifies erroneous data through preset rules, and integrates the data that meets the requirements through governance procedures to form standardized data.

5. The integrated credit data collection, processing and display system according to claim 4 is characterized by: The processing process of the data processing module is as follows: create a data verification job, and specify the target library specified by the data acquisition module as the data source of the data verification job; select the source data table corresponding to the configured data source as the verification table; Configure validation rules for the fields in the selected validation table. The rules include: date validation, value validation, non-null validation, primary key validation, special character validation, ID card compliance validation, unified social credit code compliance validation, double disclosure - administrative license validation, double disclosure - administrative penalty validation; Create a governance job and specify the target library specified by the data acquisition module as the data source for the data verification job; configure the governance job and specify the configured verification table as the governance table. The configuration parameters include table mapping configuration, such as target fields, field values, special processing of source table fields, source table master tables, related tables, sorting numbers, sorting methods, whether it is a primary key, whether it is valid, and filter condition configuration, including filter type, source table fields, comparison operations, and content input; create a governance task and fill in the trigger type, including manual and automatic. When automatic, configure the task scheduling time parameters, and the scheduling time is expressed in the form of Cron expressions; configure based on the created governance task, add multiple verification and governance job information to the same task, and configure its stop mechanism. Its parameter values ​​include complete execution failure and partial execution failure; manually execute the task for testing or automatically schedule it through the system according to the cron expression configured for the acquisition task; view the scheduling log to confirm the governance result.

6. The credit data integrated collection, processing and display system according to claim 1, characterized in that: The data modeling module uses machine learning technology to build a credit data model based on the results of data processing. The model is used for predictive analysis, classification, clustering and other application scenarios. The specific process is as follows: create a model and read the model type from the storage medium, including scorecard, classification, regression, clustering, strategy mining, and fill in the model name and description field information; enter the reading and configuration of the model data set, obtain the database and data table to which the data set belongs from the memory, and select and configure the model Y value; perform data rule configuration processing on the selected data table. The data processing rule components include missing rate cleaning, encoding, missing value filling, duplicate value removal, outlier processing, and feature deletion. The rule component configuration is performed according to the credit model business and the selected data situation. Different components correspond to different parameter configuration formats; enter the credit model feature engineering component. The component configuration includes normalization, standardization, feature selection, feature cross-combination, feature grouping aggregation, feature binning, WOE coding, IV value screening, correlation screening, principal component analysis, and feature contribution screening. Different components are selected according to the credit model business and different parameters are configured. Configuration format; divide the credit model data into data sets, including training sets and validation sets. For data with unbalanced samples, upsampling and downsampling are performed to adjust the ratio of positive and negative samples. According to the set training set ratio, the data size of the training set and validation set is automatically calculated and statistically analyzed; according to the business needs of the credit model, algorithm rules are selected from the memory. The algorithm rules include logistic regression, decision tree, XGB classification, K-means clustering, linear regression, decision tree, bp neural network, DBSCAN clustering, and different types of models select different algorithm components; save the model configuration data to the memory database, obtain the memory model information through the program trigger, and train the model. During the training process, the trigger returns the model execution log in real time according to the program running status and displays it on the user's display. After the model training is completed, the trigger returns the accuracy, confusion matrix, auc and roc curves, and ks curve data to the display; read the model list in the model management memory, select one of the models in the display interface for data prediction and analysis, and save the structural data to the database memory.

7. The credit data integrated collection, processing and display system according to claim 1, characterized in that: The data sharing module is used to provide external display services for the data passing through the data acquisition module, data processing module, and data modeling module through the data sharing platform, and provide sharing methods such as API calls, data exports, and library table docking, while simultaneously maintaining the security of data during transmission and access.

8. The integrated credit data collection, processing and display system according to claim 7, characterized in that: The specific operation process of the data sharing module is as follows: create a shared display job, read the shared data source from the memory, select the sharing type and fill in the basic configuration information of the shared job name, where the sharing types include data file export, interface API call, and library table docking; when the sharing type selects the data export method, according to the security regulations, the table data is encrypted and the file password is set, the encryption method includes SM4 encryption, symmetric encryption and asymmetric encryption, and after the data file is exported, it is sent to the other party through email or other methods; when the sharing type selects API call, the interface API service is provided, including interface authorization, interface data format encoding, and according to the security regulations, the interface transmission data is encrypted, signed and verified, and the encryption methods include SM4 encryption, symmetric encryption and asymmetric encryption; when the sharing type selects library table docking, the data in the memory is first encrypted through a trigger, and then the data is pushed to the other party's memory.

9. A method for integrated collection, processing and display of credit information data, using a system for integrated collection, processing and display of credit information data as described in any one of claims 1 to 8, characterized in that: The following steps are involved: S1: According to the preset rules and data source parameter adjustments, set the collection tasks, and automatically collect data from different data sources through databases, web crawlers, and API interfaces. The scope includes credit data from government data sharing channels, public credit information data pages on government websites, shared documents, and data sharing interfaces; S2: Use data cleaning, conversion, and normalization technologies to process and dump the collected raw data in memory, automatically identify erroneous data through preset rules, and integrate the data that meets the requirements into standardized data through governance procedures; S3: Based on the results of data processing, use machine learning technology to build a credit data model and apply the model to various scenarios, including predictive analysis, classification, and clustering; S4: Based on the sharing method of API call, data export, and database table docking, the data that flows through the above steps is provided to the outside through the data sharing platform for display services, and is securely encrypted during the data transmission and access process; S5: Set triggers and complete the automation of the above operations through trigger timing or manual execution scheduling.

10. A storage medium, characterized in that: The computer program stored in the storage medium, when executed by one or more processors, implements the method for integrated collection, processing and display of credit information data as described in claim 9.

Citation Information

Patent Citations

  • Government affair data resource integrated management system and method based on big data analysis

    CN117472874A

  • Cross-service system data fusion sharing platform, method, device and medium

    CN117992415A

  • Data sharing and co-processing platform based on cloud computing

    CN118474166A

Cited By

  • Credit scoring system for quantitative model service capability based on artificial intelligence

    CN120765025A