Enterprise intelligent question and answer screening method and system based on large model technology

Through the enterprise intelligent question-and-answer screening method and system of large-model technology, the data processing problems of financial institutions in enterprise batch screening are solved, standardized processing and intelligent screening of enterprise data are realized, and data processing efficiency and accuracy are improved.

CN118939768AInactive Publication Date: 2024-11-12天元大数据信用管理有限公司

Patent Information

Application Number
CN202410969970.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Financial institutions lack effective large-scale model technology in enterprise batch screening, making it difficult to quickly and accurately mine valuable data and features from massive and complex enterprise data.

Method used

Using intelligent enterprise question-and-answer screening methods and systems based on big model technology, the standardized processing of enterprise data and the steps of data collection, data governance, feature indicator extraction and derivation, construction of enterprise label rules and models, Q&A content analysis and screening enterprises, and the steps of enterprise data are realized. Intelligent filtering.

Benefits of technology

An automated data cleaning and processing system has been established, and a corporate portrait and feature label library has been built, which has realized rapid and real-time screening of the enterprise, and has improved data processing efficiency and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939768A_ABST
    Figure CN118939768A_ABST
Patent Text Reader

Abstract

The invention discloses an enterprise intelligent question and answer screening method and system based on a large model technology, belongs to the technical field of deep learning, and aims to solve the technical problem of how to realize enterprise batch screening based on the large model technology. Comprising the following steps: establishing different data acquisition modes, and establishing a data acquisition management tool; performing data governance on the acquired enterprise data based on a large model technology; taking the original indexes and the new indexes as feature indexes required by training, and storing the feature indexes in an established index library; establishing enterprise portrait labels, constructing label rules and models, and storing the generated enterprise portraits and feature labels to an enterprise portrait and feature label library; related screening conditions and rules are generated based on the enterprise features and the requirement conditions, and an SQL screening program is generated according to the screening conditions and the rules; and based on the enterprise portrait and the feature tag library, executing an SQL screening program to screen the enterprise required by the client and returning the enterprise to the client.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and specifically to an enterprise intelligent question and answer screening method and system based on big model technology. Background Art

[0002] With the accelerated development and evolution of big model technology, all walks of life are actively developing related industry and industrial big models and applications based on big models, and have launched various industry and industrial big models. Large models are based on large data sets, and use relevant big data, semantic analysis, machine learning and other related comprehensive big model technologies to train data to form a deep learning model with a large number of parameters, which can train rich data representations and patterns, thereby performing well in various task processing and analysis.

[0003] At present, financial institutions attach more and more importance to the batch screening of enterprises, especially the ability to quickly screen out batches of enterprises according to their different credit conditions and needs, so that they can effectively expand their customer base. However, there are relatively few studies on enterprise big models for financial institutions. There is a lack of big models that can process massive enterprise data, and it is not possible to accurately and quickly discover valuable data and enterprises from massive and complex enterprise data. Through big model training technology, massive data of enterprises can be mined and trained, and enterprise portraits and feature labels can be constructed, such as enterprise scale, enterprise type, credit rating, organizational type, etc., to form a comprehensive portrait and feature label library of enterprises; at the same time, based on the needs of financial institutions for enterprise screening, the use of big model technology can convert the demand content into screening label rules and related screening procedures, so as to screen out the required enterprises.

[0004] How to achieve batch screening for enterprises based on large model technology is a technical problem that needs to be solved. Summary of the invention

[0005] The technical task of the present invention is to address the above shortcomings and provide an enterprise intelligent question and answer screening method and system based on big model technology to solve the technical problem of how to achieve batch screening of enterprises based on big model technology.

[0006] In a first aspect, the present invention provides an enterprise intelligent question-answering screening method based on a large model technology, comprising the following steps:

[0007] Enterprise data collection: Establish different data collection methods according to different data sources, data types and data volumes, and establish data collection management tools to manage the collected data;

[0008] Data governance: Based on the big model technology, the collected enterprise data is governed to obtain standardized enterprise data, and a standardized enterprise database is established and stored in the standardized enterprise database. Data governance includes data screening and verification, subject identification, data format and missing value processing, association and fusion, standardization, and flow identification;

[0009] Feature index extraction and derivation: formulate original indicators according to business needs, and derive indicators from the original indicators based on the large model algorithm to obtain new indicators. The original indicators and new indicators are used as feature indicators required for training, and the feature indicators are stored in the established indicator library;

[0010] Construct enterprise labeling rules and models: Based on enterprise data and characteristic indicators, analyze the importance of indicators through large model training technology, screen out important characteristic indicators, describe enterprise characteristics based on important characteristic indicators, establish enterprise portrait labels and construct labeling rules and models, generate enterprise portraits and characteristic labels based on labeling rules and models, and store the generated enterprise portraits and characteristic labels in the enterprise portrait and characteristic label library;

[0011] Question and answer content analysis: Through the semantic analysis in the big model technology, the enterprise screening requirements proposed by the customer are semantically analyzed to obtain the enterprise characteristics and requirements, and then the relevant screening conditions and rules are generated based on the enterprise characteristics and requirements, and the SQL screening program is generated according to the screening conditions and rules;

[0012] Filter enterprises: Based on enterprise portraits and feature tag libraries, execute SQL filtering programs to filter out the enterprises required by customers and return them to customers.

[0013] As a preferred method, there are two data collection methods, namely:

[0014] For large-volume enterprise data collection that needs to be updated regularly, offline distributed collection is used, and in the context of the data warehouse, ETL is used to extract, transform and load enterprise data;

[0015] For small batches of enterprise data that need to be updated regularly, a real-time collection method is adopted, and Flume / Kafka tools are used to implement stream processing to achieve the extraction, conversion, loading and warehousing of enterprise data.

[0016] As a preferred method, data governance is performed on the collected enterprise data based on big model technology, including:

[0017] Original data collection: The collected enterprise data is used as original data and stored in the original data layer;

[0018] Data screening and verification: Based on the big model technology, the enterprise data is screened and classified, and the enterprise data is divided into multiple types. Based on the big model technology, data verification rules are generated according to the needs, and the data fields and contents are verified according to the data verification rules. The enterprise data that passes the verification is loaded into the central database. For the enterprise data that fails, the enterprise data is returned to the data collection layer, and the enterprise data is re-collected through real-time collection. For the re-collected enterprise data, if it fails to pass the verification or the data is wrong, the re-collected enterprise data is discarded and stored in the original data layer;

[0019] Cleaning: Enterprise data is cleaned through large model technology. The cleaning content includes subject identification, filling missing data, removing duplicate data, eliminating noise data, and associative fusion;

[0020] Standard data: Store cleaned enterprise data in a standardized database by category, and identify the data source and related flow of enterprise data;

[0021] Monitoring and analysis: Monitor and analyze the process and quality of data governance, monitor the cleaned standardized database, and perform quality control on the enterprise data stored in the standardized database.

[0022] Preferably, the feature index extraction and derivation includes the following steps:

[0023] Business indicator definition: Based on the enterprise data in the standardized enterprise database, extract the data fields, and define the relevant business indicators as the original indicators according to the characteristics and meaning of the data fields and the business;

[0024] Derived indicator generation: Derived indicators are derived based on the defined indicators and related data fields and based on the big model algorithm. Indicator derivation is to combine existing indicators or fields to generate new meaningful indicators. The big model algorithm includes indicator engineering derivation, crossover, GBDT, decision tree, and random forest.

[0025] Indicator storage: Indicators are divided into multiple physical tables through segmentation. A predetermined number of indicators are divided into one physical table. Indicators are stored in the form of wide tables and vertical tables. An indicator library is built and stored through the indicator library.

[0026] Preferably, building enterprise tag rules and models includes the following steps:

[0027] Construction of portrait statistical labels: Based on enterprise data and indicators, mining and analysis are carried out through large model technologies including statistical analysis, association analysis, correlation, WOE indicator binning, IV value screening and stepwise regression, and indicator importance analysis is carried out through large model algorithms including XGBoost and decision trees, and important feature indicators are screened out. Based on the important feature indicators, enterprise characteristics are described and fact labels are established. Based on the fact labels, fact labels are segmented and clustered through large model technologies including WOE indicator binning, unsupervised clustering and association rules to establish statistical labels;

[0028] Label rules and model construction: Based on enterprise data and indicators, relevant label rules and models are constructed through large model algorithms including statistical analysis, association analysis, correlation, WOE indicator binning, IV value screening, cluster analysis, neural network, stepwise regression, decision tree, XGBoost and random forest to form a model layer;

[0029] Construction of enterprise portraits and feature label library: Label rules and models generate enterprise portraits and feature labels, and build a feature label library to store the generated enterprise portraits and feature labels.

[0030] In a second aspect, the present invention provides an enterprise intelligent question and answer screening system based on big model technology, which is used to screen enterprises through an enterprise intelligent question and answer screening method based on big model technology as described in any one of the first aspects, and the system includes an enterprise data collection module, a data governance module, a feature index extraction and derivation module, an enterprise label rule and model construction module, a question and answer content analysis module and an enterprise screening module;

[0031] The enterprise data collection module is used to perform the following: establish different data collection methods according to different data sources, data types and data volumes, and establish data collection management tools, which are used to manage the collected data;

[0032] The data governance module is used to perform the following: data governance of the collected enterprise data based on the big model technology to obtain standardized enterprise data, establish a standardized enterprise database, and store the standardized enterprise data in the standardized enterprise database. Data governance includes data screening and verification, subject identification, data format and missing value processing, association and fusion, standardization, and flow identification;

[0033] The feature index extraction and derivation module is used to perform the following: formulate original indicators according to business needs, and derive indicators from the original indicators based on the large model algorithm to obtain new indicators, use the original indicators and the new indicators as feature indicators required for training, and store the feature indicators in the established indicator library;

[0034] The enterprise labeling rules and model building module is used to perform the following: based on enterprise data and characteristic indicators, perform indicator importance analysis through large model training technology, screen out important characteristic indicators, describe enterprise characteristics based on important characteristic indicators, establish enterprise portrait labels and build labeling rules and models, generate enterprise portraits and characteristic labels based on labeling rules and models, and store the generated enterprise portraits and characteristic labels in the enterprise portrait and characteristic label library;

[0035] The question and answer content parsing module is used to perform the following: semantic analysis of the enterprise screening requirements proposed by the customer through semantic analysis in the big model technology, obtain enterprise characteristics and requirements, and generate relevant screening conditions and rules based on the enterprise characteristics and requirements, and generate SQL screening programs according to the screening conditions and rules;

[0036] The enterprise screening module is used to perform the following: based on the enterprise portrait and feature tag library, execute the SQL screening program to screen out the enterprises required by the customer and return them to the customer.

[0037] As a preferred method, there are two data collection methods, namely:

[0038] For large-volume enterprise data collection that needs to be updated regularly, offline distributed collection is used, and in the context of the data warehouse, ETL is used to extract, transform and load enterprise data;

[0039] For small batches of enterprise data that need to be updated regularly, a real-time collection method is adopted, and Flume / Kafka tools are used to implement stream processing to achieve the extraction, conversion, loading and warehousing of enterprise data.

[0040] Preferably, the data quality module is configured with an original data layer, a data screening and verification layer, a cleaning layer, a standard data layer, and a monitoring and analysis layer;

[0041] The original data layer is used to perform the following: Data governance of collected enterprise data based on big model technology, including:

[0042] Original data collection: The collected enterprise data is used as original data and stored in the original data layer;

[0043] Data screening and verification: Based on the big model technology, the enterprise data is screened and classified, and the enterprise data is divided into multiple types. Based on the big model technology, data verification rules are generated according to the needs, and the data fields and contents are verified according to the data verification rules. The enterprise data that passes the verification is loaded into the central database. For the enterprise data that fails, the enterprise data is returned to the data collection layer, and the enterprise data is re-collected through real-time collection. For the re-collected enterprise data, if it fails to pass the verification or the data is wrong, the re-collected enterprise data is discarded and stored in the original data layer;

[0044] Cleaning: Enterprise data is cleaned through large model technology. The cleaning content includes subject identification, filling missing data, removing duplicate data, eliminating noise data, and associative fusion;

[0045] Standard data: Store cleaned enterprise data in a standardized database by category, and identify the data source and related flow of enterprise data;

[0046] Monitoring and analysis: Monitor and analyze the process and quality of data governance, monitor the cleaned standardized database, and perform quality control on the enterprise data stored in the standardized database.

[0047] The enterprise intelligent question-answering screening method and system based on large model technology of the present invention have the following advantages:

[0048] 1. The big model technology is used to solve the problems of data screening and verification, subject identification, data format and missing value processing, association and fusion, standardization, and flow identification of enterprise information, and a complete set of automatic cleaning and processing systems and standardized enterprise databases based on big model technology are established;

[0049] 2. It provides a method to further derive indicators based on business characteristic indicators by using large model technology, thus providing a way of thinking for the application of large model technology in derived indicators;

[0050] 3. An automated sample extraction technology was built, which uses large model technology to automatically extract different training samples, providing a solution for automated sample construction and improving the efficiency of sample construction;

[0051] 4. Based on the automatically constructed training samples, we analyze and train a large number of enterprises based on large model training technologies such as Woe, GBDT, NLP, neural network, ensemble learning, etc., build portrait statistical labels, label rules and models, establish enterprise portraits and feature label libraries, and support the upper-level intelligent question and answer screening function;

[0052] 5. An intelligent question-and-answer system is built using big model technology. The system can automatically analyze the enterprise screening needs and requirements proposed by customers and generate relevant screening programs. At the same time, based on enterprise portraits and label libraries, it can screen out the enterprises required by customers, realizing rapid and real-time screening of enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] The present invention is further described below in conjunction with the accompanying drawings.

[0055] Figure 1 This is a flowchart of an enterprise intelligent question and answer screening method based on big model technology in Example 1. DETAILED DESCRIPTION

[0056] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0057] The embodiments of the present invention provide an enterprise intelligent question and answer screening method and system based on big model technology, which are used to solve the technical problem of how to achieve batch screening of enterprises based on big model technology.

[0058] Embodiment 1:

[0059] The present invention provides an enterprise intelligent question and answer screening method based on big model technology, including six steps of enterprise data collection, data governance, feature indicator extraction and derivation, building enterprise label rules and models, question and answer content analysis, and enterprise screening.

[0060] Step S100: Enterprise data collection: Different data collection methods are established according to different data sources, data types and data volumes, and a data collection management tool is established. The data collection management tool is used to manage the collected data.

[0061] As a specific implementation of step S100, there are two data collection methods, namely:

[0062] (1) For large-scale enterprise data collection that needs to be updated regularly, offline distributed collection is used, and ETL is used in the context of data warehouse to extract, transform and load enterprise data;

[0063] (2) For small batches of enterprise data that need to be updated regularly, a real-time collection method is adopted, and Flume / Kafka tools are used to implement stream processing to achieve the extraction, conversion, loading and storage of enterprise data.

[0064] In this embodiment, the above two methods realize the automatic collection of data, and at the same time establish a collection management tool to achieve efficient management of the collection.

[0065] Step S200 Data Governance: Based on the big model technology, the collected enterprise data is governed to obtain standardized enterprise data, and a standardized enterprise database is established, and the standardized enterprise data is stored in the standardized enterprise database. Among them, data governance includes data screening and verification, subject identification, data format and missing value processing, association and fusion, standardization, and flow identification.

[0066] As a specific implementation, data governance is performed on the collected enterprise data based on big model technology, including:

[0067] (1) Raw data collection: The collected enterprise data is used as raw data and stored in the raw data layer;

[0068] (2) Data screening and verification: Based on the big model technology, the enterprise data is screened and classified into multiple types. Based on the big model technology, data verification rules are generated according to the needs, and the data fields and contents are verified according to the data verification rules. The enterprise data that passes the verification is loaded into the central database. For the enterprise data that fails, the enterprise data is returned to the data collection layer and re-collected in real time. For the re-collected enterprise data, if it fails the verification or the data is wrong, the re-collected enterprise data is discarded and stored in the original data layer;

[0069] (3) Cleaning: Enterprise data is cleaned through large model technology. The cleaning content includes subject identification, filling missing data, removing duplicate data, eliminating noise data, and associative fusion;

[0070] (4) Standard data: The cleaned enterprise data is stored in a standardized database according to categories, and the data source and related flow of the enterprise data are marked;

[0071] (5) Monitoring and Analysis: Monitor and analyze the process and quality of data governance, monitor the cleaned standardized database, and perform quality control on the enterprise data stored in the standardized database.

[0072] Among them, step (3) mainly analyzes the causes and forms of "dirty data", uses existing big model data mining means and methods to clean "dirty data", and converts "dirty data" into data that meets data quality requirements, thereby improving the quality of the data set and meeting the needs of data mining analysis. Subject identification mainly defines relevant subject identification identifiers and rules based on different subjects and data types. Enterprise subjects are mainly identified by unified social credit codes, enterprise names, types, etc., among which unified social credit codes and enterprise names are mainly identified; in addition, for data whose subject identification is not cleaned, the correlation between the two subject features is analyzed by NLP analysis and matching algorithm models based on the subject characteristics, and the relevant subjects are identified and matched based on the correlation. The missing data filling process is based on the K-NN nearest neighbor filling algorithm in the big model technology to fill the missing data: first, the values ​​of each field of the data set are non-dimensionalized to eliminate the influence of inconsistent units between fields and obtain a standardized data matrix; then, the data records containing missing values ​​"-" are filled with K-NN nearest neighbors, and the distance matrix between the data record and other data records that do not contain missing values ​​is calculated, and the data with the closest Euclidean distance is selected to fill the missing data. Deduplication of duplicate data refers to the use of field similarity in large model technology to identify and judge duplicate values. Clustering methods are used to identify and eliminate noise data in massive enterprise data. Data association fusion is performed based on subject identification, and data of different dimensions are integrated with data of the same dimension to process relevant indicators of the model across tables.

[0073] Step (4) mainly stores the cleaned standardized data, and stores it according to data categories such as basic enterprise information, change information, shareholder information, foreign investment, intellectual property information, judicial case filing information, announcement information, judgment information, execution information, dishonest executors, restrictions on high consumption, e-commerce business information, store information, financial information, customs information, administrative penalty information, loan information, legal representative information, etc., so that it can be used and called by the upper layer. At the same time, the data source and related circulation are marked for data tracing.

[0074] Step S300: Feature indicator extraction and derivation: formulate original indicators according to business needs, and derive indicators from the original indicators based on the large model algorithm to obtain new indicators, use the original indicators and the new indicators as feature indicators required for training, and store the feature indicators in the established indicator library.

[0075] In this embodiment, the feature index extraction and derivation includes the following steps:

[0076] (1) Business indicator definition: Based on the enterprise data in the standardized enterprise database, extract the data fields, and define the relevant business indicators as the original indicators based on the characteristics and meaning of the data fields and the business;

[0077] (2) Derived indicator generation: Derived indicators are derived based on the defined indicators and related data fields and based on the big model algorithm. Indicator derivation is to combine existing indicators or fields to generate new meaningful indicators. Big model algorithms include indicator engineering derivation, crossover, GBDT, decision tree, and random forest.

[0078] (3) Index storage: The indexes are divided into multiple physical tables by means of segmentation. A predetermined number of indexes are divided into one physical table. The indexes are stored in the form of wide tables and vertical tables. An index library is constructed and the indexes are stored in the index library.

[0079] Step S400 constructs enterprise labeling rules and models: Based on enterprise data and characteristic indicators, the indicator importance analysis is performed through large model training technology, and important characteristic indicators are screened out. Enterprise characteristics are described based on the important characteristic indicators, enterprise portrait labels are established, and labeling rules and models are constructed. Enterprise portraits and characteristic labels are generated based on the labeling rules and models, and the generated enterprise portraits and characteristic labels are stored in the enterprise portrait and characteristic label library.

[0080] As a specific implementation of step S400, building enterprise tag rules and models includes the following steps:

[0081] (1) Construction of portrait statistical labels: Based on enterprise data and indicators, mining and analysis are carried out through large model technologies including statistical analysis, association analysis, correlation, WOE indicator binning, IV value screening and stepwise regression, and indicator importance analysis is carried out through large model algorithms including XGBoost and decision trees to screen out important feature indicators, describe enterprise characteristics based on important feature indicators, establish fact labels, and segment and cluster fact labels based on fact labels through large model technologies including WOE indicator binning, unsupervised clustering and association rules to establish statistical labels;

[0082] (2) Labeling rules and model construction: Based on enterprise data and indicators, relevant labeling rules and models are constructed through large-scale model algorithms including statistical analysis, association analysis, correlation, WOE indicator binning, IV value screening, cluster analysis, neural network, stepwise regression, decision tree, XGBoost and random forest to form a model layer;

[0083] (3) Construction of enterprise portrait and feature label library: Label rules and models are used to generate enterprise portraits and feature labels, and a feature label library is constructed to store the generated enterprise portraits and feature labels.

[0084] Among them, in step (2), based on massive enterprise data and indicators, statistical analysis, association analysis, correlation, WOE indicator binning, IV value screening, cluster analysis, neural network, stepwise regression, decision tree, XGBoost, random forest and other large model technologies are used to build relevant label rules and models to form a model layer. The specific implementation steps are as follows:

[0085] First, statistical analysis is performed on the relevant massive enterprise characteristics to construct relevant fact labels;

[0086] Second, based on fact labels, we use association analysis, correlation, WOE index binning, IV value screening, cluster analysis, neural networks, stepwise regression, decision trees, XGBoost, random forests and other large model technologies to automatically screen the company's fact labels and build relevant prediction features. We use neural networks, stepwise regression, decision trees, XGBoost, random forests and other large model technologies to train and adjust model parameters, and finally establish massive association rules and models.

[0087] Step S500 Question and answer content analysis: semantic analysis is performed on the enterprise screening requirements proposed by the customer through semantic analysis in the big model technology to obtain enterprise characteristics and requirements, and relevant screening conditions and rules are generated based on the enterprise characteristics and requirements, and SQL screening programs are generated according to the screening conditions and rules.

[0088] In this embodiment, step S500 uses a question-and-answer method and the semantic analysis in the big model technology to conduct an in-depth semantic analysis on the enterprise screening needs proposed by the customer, and extracts the enterprise characteristics and corresponding requirements required by the customer; according to the extracted enterprise characteristics and corresponding requirements, relevant screening conditions and rules are automatically combined, and a SQL screening program is generated according to the screening conditions and rules.

[0089] Step S600: Filter enterprises: Based on enterprise portraits and feature tag libraries, execute SQL filtering procedures to filter out enterprises required by customers and return them to customers.

[0090] The method of this embodiment is based on the multi-dimensional massive data of the enterprise, and uses the big model technology to build a new big model data processing technology and system to achieve rapid data processing and accurate cleaning; at the same time, the big model technology is introduced to build the portrait and feature label model of massive enterprises, expand the application field of the big model technology, and make the enterprise portrait and feature label better describe the real situation of the enterprise. According to the customer's demand for enterprise screening, the big model technology is used to implement the analysis of the demand content, and the analysis content is converted into screening label rules and conditions, so as to automatically screen out batches of enterprises that meet the conditions.

[0091] Embodiment 2:

[0092] The present invention discloses an enterprise intelligent question and answer screening system based on big model technology, comprising an enterprise data collection module, a data governance module, a feature index extraction and derivation module, an enterprise label rule and model building module, a question and answer content parsing module and an enterprise screening module.

[0093] The enterprise data collection module is used to perform the following: establish different data collection methods according to different data sources, data types and data volumes, and establish a data collection management tool, which is used to manage the collected data.

[0094] As the specific implementation of the enterprise data collection module, there are two data collection methods:

[0095] (1) For large-scale enterprise data collection that needs to be updated regularly, offline distributed collection is used, and ETL is used in the context of data warehouse to extract, transform and load enterprise data;

[0096] (2) For small batches of enterprise data that need to be updated regularly, a real-time collection method is adopted, and Flume / Kafka tools are used to implement stream processing to achieve the extraction, conversion, loading and storage of enterprise data.

[0097] In this embodiment, the above two methods realize the automatic collection of data, and at the same time establish a collection management tool to achieve efficient management of the collection.

[0098] The data governance module is used to perform the following: perform data governance on the collected enterprise data based on big model technology to obtain standardized enterprise data, establish a standardized enterprise database, and store the standardized enterprise data in the standardized enterprise database. Among them, data governance includes data screening and verification, subject identification, data format and missing value processing, association and fusion, standardization, and flow identification.

[0099] As a specific implementation, the data governance module is used to implement data governance as follows:

[0100] (1) Raw data collection: The collected enterprise data is used as raw data and stored in the raw data layer;

[0101] (2) Data screening and verification: Based on the big model technology, the enterprise data is screened and classified into multiple types. Based on the big model technology, data verification rules are generated according to the needs, and the data fields and contents are verified according to the data verification rules. The enterprise data that passes the verification is loaded into the central database. For the enterprise data that fails, the enterprise data is returned to the data collection layer and re-collected in real time. For the re-collected enterprise data, if it fails the verification or the data is wrong, the re-collected enterprise data is discarded and stored in the original data layer;

[0102] (3) Cleaning: Enterprise data is cleaned through large model technology. The cleaning content includes subject identification, filling missing data, removing duplicate data, eliminating noise data, and associative fusion;

[0103] (4) Standard data: The cleaned enterprise data is stored in a standardized database according to categories, and the data source and related flow of the enterprise data are marked;

[0104] (5) Monitoring and Analysis: Monitor and analyze the process and quality of data governance, monitor the cleaned standardized database, and perform quality control on the enterprise data stored in the standardized database.

[0105] Among them, step (3) mainly analyzes the causes and forms of "dirty data", uses existing big model data mining means and methods to clean "dirty data", and converts "dirty data" into data that meets data quality requirements, thereby improving the quality of the data set and meeting the needs of data mining analysis. Subject identification mainly defines relevant subject identification identifiers and rules based on different subjects and data types. Enterprise subjects are mainly identified by unified social credit codes, enterprise names, types, etc., among which unified social credit codes and enterprise names are mainly identified; in addition, for data whose subject identification is not cleaned, the correlation between the two subject features is analyzed by NLP analysis and matching algorithm models based on the subject characteristics, and the relevant subjects are identified and matched based on the correlation. The missing data filling process is based on the K-NN nearest neighbor filling algorithm in the big model technology to fill the missing data: first, the values ​​of each field of the data set are non-dimensionalized to eliminate the influence of inconsistent units between fields and obtain a standardized data matrix; then, the data records containing missing values ​​"-" are filled with K-NN nearest neighbors, and the distance matrix between the data record and other data records that do not contain missing values ​​is calculated, and the data with the closest Euclidean distance is selected to fill the missing data. Deduplication of duplicate data refers to the use of field similarity in large model technology to identify and judge duplicate values. Clustering methods are used to identify and eliminate noise data in massive enterprise data. Data association fusion is performed based on subject identification, and data of different dimensions are integrated with data of the same dimension to process relevant indicators of the model across tables.

[0106] Step (4) mainly stores the cleaned standardized data, and stores it according to data categories such as basic enterprise information, change information, shareholder information, foreign investment, intellectual property information, judicial case filing information, announcement information, judgment information, execution information, dishonest executors, restrictions on high consumption, e-commerce business information, store information, financial information, customs information, administrative penalty information, loan information, legal representative information, etc., so that it can be used and called by the upper layer. At the same time, the data source and related circulation are marked for data tracing.

[0107] The feature indicator extraction and derivation module is used to perform the following: formulate original indicators according to business needs, and derive indicators from the original indicators based on the large model algorithm to obtain new indicators, use the original indicators and new indicators as feature indicators required for training, and store the feature indicators in the established indicator library.

[0108] In this embodiment, the feature index extraction and derivation module is used to perform the following operations:

[0109] (1) Business indicator definition: Based on the enterprise data in the standardized enterprise database, extract the data fields, and define the relevant business indicators as the original indicators based on the characteristics and meaning of the data fields and the business;

[0110] (2) Derived indicator generation: Derived indicators are derived based on the defined indicators and related data fields and based on the big model algorithm. Indicator derivation is to combine existing indicators or fields to generate new meaningful indicators. Big model algorithms include indicator engineering derivation, crossover, GBDT, decision tree, and random forest.

[0111] (3) Index storage: The indexes are divided into multiple physical tables by means of segmentation. A predetermined number of indexes are divided into one physical table. The indexes are stored in the form of wide tables and vertical tables. An index library is constructed and the indexes are stored in the index library.

[0112] The enterprise labeling rules and model building module is used to perform the following: based on enterprise data and feature indicators, perform indicator importance analysis through large model training technology, screen out important feature indicators, describe enterprise characteristics based on important feature indicators, establish enterprise portrait labels and build labeling rules and models, generate enterprise portraits and feature labels based on labeling rules and models, and store the generated enterprise portraits and feature labels in the enterprise portrait and feature label library.

[0113] As a specific implementation of the enterprise labeling rules and model building module, it is used to perform the following operations:

[0114] (1) Construction of portrait statistical labels: Based on enterprise data and indicators, mining and analysis are carried out through large model technologies including statistical analysis, association analysis, correlation, WOE indicator binning, IV value screening and stepwise regression, and indicator importance analysis is carried out through large model algorithms including XGBoost and decision trees to screen out important feature indicators, describe enterprise characteristics based on important feature indicators, establish fact labels, and segment and cluster fact labels based on fact labels through large model technologies including WOE indicator binning, unsupervised clustering and association rules to establish statistical labels;

[0115] (2) Labeling rules and model construction: Based on enterprise data and indicators, relevant labeling rules and models are constructed through large-scale model algorithms including statistical analysis, association analysis, correlation, WOE indicator binning, IV value screening, cluster analysis, neural network, stepwise regression, decision tree, XGBoost and random forest to form a model layer;

[0116] (3) Construction of enterprise portrait and feature label library: Label rules and models are used to generate enterprise portraits and feature labels, and a feature label library is constructed to store the generated enterprise portraits and feature labels.

[0117] Among them, in step (2), based on massive enterprise data and indicators, statistical analysis, association analysis, correlation, WOE indicator binning, IV value screening, cluster analysis, neural network, stepwise regression, decision tree, XGBoost, random forest and other large model technologies are used to build relevant label rules and models to form a model layer. The specific implementation steps are as follows:

[0118] First, statistical analysis is performed on the relevant massive enterprise characteristics to construct relevant fact labels;

[0119] Second, based on fact labels, we use association analysis, correlation, WOE index binning, IV value screening, cluster analysis, neural networks, stepwise regression, decision trees, XGBoost, random forests and other large model technologies to automatically screen the company's fact labels and build relevant prediction features. We use neural networks, stepwise regression, decision trees, XGBoost, random forests and other large model technologies to train and adjust model parameters, and finally establish massive association rules and models.

[0120] The question and answer content parsing module is used to perform the following: perform semantic analysis on the enterprise screening requirements proposed by the customer through semantic analysis in the big model technology, obtain enterprise characteristics and requirements, and generate relevant screening conditions and rules based on the enterprise characteristics and requirements, and generate SQL screening programs according to the screening conditions and rules.

[0121] In this embodiment, the question and answer content analysis module is used to conduct in-depth semantic analysis on the enterprise screening needs proposed by the customer through question and answer, using the semantic analysis in the big model technology, and extract the enterprise characteristics and corresponding requirements required by the customer; according to the extracted enterprise characteristics and corresponding requirements, relevant screening conditions and rules are automatically combined, and a SQL screening program is generated according to the screening conditions and rules.

[0122] The enterprise screening module is used to perform the following: based on the enterprise portrait and feature tag library, execute the SQL screening program to screen out the enterprises required by the customer and return them to the customer.

[0123] The system of this embodiment can execute the method disclosed in Example 1 to realize intelligent question and answer screening for the enterprise.

[0124] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. An enterprise intelligent question-answering screening method based on large model technology, It is characterized in that The steps include: Enterprise data collection: Establish different data collection methods according to different data sources, data types and data volumes, and establish data collection management tools to manage the collected data; Data governance: Based on the big model technology, the collected enterprise data is governed to obtain standardized enterprise data, and a standardized enterprise database is established and stored in the standardized enterprise database. Data governance includes data screening and verification, subject identification, data format and missing value processing, association and fusion, standardization, and flow identification; Feature index extraction and derivation: formulate original indicators according to business needs, and derive indicators from the original indicators based on the large model algorithm to obtain new indicators. The original indicators and new indicators are used as feature indicators required for training, and the feature indicators are stored in the established indicator library; Construct enterprise labeling rules and models: Based on enterprise data and characteristic indicators, analyze the importance of indicators through large model training technology, screen out important characteristic indicators, describe enterprise characteristics based on important characteristic indicators, establish enterprise portrait labels and construct labeling rules and models, generate enterprise portraits and characteristic labels based on labeling rules and models, and store the generated enterprise portraits and characteristic labels in the enterprise portrait and characteristic label library; Question and answer content analysis: Through the semantic analysis in the big model technology, the enterprise screening requirements proposed by the customer are semantically analyzed to obtain the enterprise characteristics and requirements, and then the relevant screening conditions and rules are generated based on the enterprise characteristics and requirements, and the SQL screening program is generated according to the screening conditions and rules; Filter enterprises: Based on enterprise portraits and feature tag libraries, execute SQL filtering programs to filter out the enterprises required by customers and return them to customers.

2. According to claim 1, the enterprise intelligent question and answer screening method based on large model technology, It is characterized in that There are two ways to collect data: For large-volume enterprise data collection that needs to be updated regularly, offline distributed collection is used, and in the context of the data warehouse, ETL is used to extract, transform and load enterprise data; For small batches of enterprise data that need to be updated regularly, a real-time collection method is adopted, and Flume / Kafka tools are used to implement stream processing to achieve the extraction, conversion, loading and warehousing of enterprise data.

3. According to claim 1, the enterprise intelligent question and answer screening method based on large model technology, It is characterized in that Data governance of collected enterprise data based on big model technology, including: Original data collection: The collected enterprise data is used as original data and stored in the original data layer; Data screening and verification: Based on the big model technology, the enterprise data is screened and classified, and the enterprise data is divided into multiple types. Based on the big model technology, data verification rules are generated according to the needs, and the data fields and contents are verified according to the data verification rules. The enterprise data that passes the verification is loaded into the central database. For the enterprise data that fails, the enterprise data is returned to the data collection layer, and the enterprise data is re-collected through real-time collection. For the re-collected enterprise data, if it fails to pass the verification or the data is wrong, the re-collected enterprise data is discarded and stored in the original data layer; Cleaning: Enterprise data is cleaned through large model technology. The cleaning content includes subject identification, filling missing data, removing duplicate data, eliminating noise data, and associative fusion; Standard data: Store cleaned enterprise data in a standardized database by category, and identify the data source and related flow of enterprise data; Monitoring and analysis: Monitor and analyze the process and quality of data governance, monitor the cleaned standardized database, and perform quality control on the enterprise data stored in the standardized database.

4. The enterprise intelligent question-answering screening method based on large model technology according to claim 1, It is characterized in that The extraction and derivation of characteristic indicators include the following steps: Business indicator definition: Based on the enterprise data in the standardized enterprise database, extract the data fields, and define the relevant business indicators as the original indicators according to the characteristics and meaning of the data fields and the business; Derived indicator generation: Derived indicators are derived based on the defined indicators and related data fields and based on the big model algorithm. Indicator derivation is to combine existing indicators or fields to generate new meaningful indicators. The big model algorithm includes indicator engineering derivation, crossover, GBDT, decision tree, and random forest. Indicator storage: Indicators are divided into multiple physical tables through segmentation. A predetermined number of indicators are divided into one physical table. Indicators are stored in the form of wide tables and vertical tables. An indicator library is built and stored through the indicator library.

5. The enterprise intelligent question-answering screening method based on large model technology according to claim 1, It is characterized in that Building enterprise tagging rules and models includes the following steps: Construction of portrait statistical labels: Based on enterprise data and indicators, mining and analysis are carried out through large model technologies including statistical analysis, association analysis, correlation, WOE indicator binning, IV value screening and stepwise regression, and indicator importance analysis is carried out through large model algorithms including XGBoost and decision trees, and important feature indicators are screened out. Based on the important feature indicators, enterprise characteristics are described and fact labels are established. Based on the fact labels, fact labels are segmented and clustered through large model technologies including WOE indicator binning, unsupervised clustering and association rules to establish statistical labels; Label rules and model construction: Based on enterprise data and indicators, relevant label rules and models are constructed through large model algorithms including statistical analysis, association analysis, correlation, WOE indicator binning, IV value screening, cluster analysis, neural network, stepwise regression, decision tree, XGBoost and random forest to form a model layer; Construction of enterprise portraits and feature label library: Label rules and models generate enterprise portraits and feature labels, and build a feature label library to store the generated enterprise portraits and feature labels.

6. An enterprise intelligent question and answer screening system based on large model technology, It is characterized in that Used to screen enterprises through an enterprise intelligent question and answer screening method based on big model technology as described in any one of claims 1 to 5, the system includes an enterprise data collection module, a data governance module, a feature index extraction and derivation module, an enterprise label rule and model building module, a question and answer content parsing module and an enterprise screening module; The enterprise data collection module is used to perform the following: establish different data collection methods according to different data sources, data types and data volumes, and establish data collection management tools, which are used to manage the collected data; The data governance module is used to perform the following: data governance of the collected enterprise data based on the big model technology to obtain standardized enterprise data, establish a standardized enterprise database, and store the standardized enterprise data in the standardized enterprise database. Data governance includes data screening and verification, subject identification, data format and missing value processing, association and fusion, standardization, and flow identification; The feature index extraction and derivation module is used to perform the following: formulate original indicators according to business needs, and derive indicators from the original indicators based on the large model algorithm to obtain new indicators, use the original indicators and the new indicators as feature indicators required for training, and store the feature indicators in the established indicator library; The enterprise labeling rules and model building module is used to perform the following: based on enterprise data and characteristic indicators, perform indicator importance analysis through large model training technology, screen out important characteristic indicators, describe enterprise characteristics based on important characteristic indicators, establish enterprise portrait labels and build labeling rules and models, generate enterprise portraits and characteristic labels based on labeling rules and models, and store the generated enterprise portraits and characteristic labels in the enterprise portrait and characteristic label library; The question and answer content parsing module is used to perform the following: semantic analysis of the enterprise screening requirements proposed by the customer through semantic analysis in the big model technology, obtain enterprise characteristics and requirements, and generate relevant screening conditions and rules based on the enterprise characteristics and requirements, and generate SQL screening programs according to the screening conditions and rules; The enterprise screening module is used to perform the following: based on the enterprise portrait and feature tag library, execute the SQL screening program to screen out the enterprises required by the customer and return them to the customer.

7. The enterprise intelligent question and answer screening system based on large model technology according to claim 6, It is characterized in that There are two ways to collect data: For large-volume enterprise data collection that needs to be updated regularly, offline distributed collection is used, and in the context of the data warehouse, ETL is used to extract, transform and load enterprise data; For small batches of enterprise data that need to be updated regularly, a real-time collection method is adopted, and Flume / Kafka tools are used to implement stream processing to achieve the extraction, conversion, loading and warehousing of enterprise data.

8. The enterprise intelligent question and answer screening system based on large model technology according to claim 6, It is characterized in that The data quality module is configured with the original data layer, data screening and verification layer, cleaning layer, standard data layer, and monitoring and analysis layer; The original data layer is used to perform the following: Data governance of collected enterprise data based on big model technology, including: Original data collection: The collected enterprise data is used as original data and stored in the original data layer; Data screening and verification: Based on the big model technology, the enterprise data is screened and classified, and the enterprise data is divided into multiple types. Based on the big model technology, data verification rules are generated according to the needs, and the data fields and contents are verified according to the data verification rules. The enterprise data that passes the verification is loaded into the central database. For the enterprise data that fails, the enterprise data is returned to the data collection layer, and the enterprise data is re-collected through real-time collection. For the re-collected enterprise data, if it fails to pass the verification or the data is wrong, the re-collected enterprise data is discarded and stored in the original data layer; Cleaning: Enterprise data is cleaned through large model technology. The cleaning content includes subject identification, filling missing data, removing duplicate data, eliminating noise data, and associative fusion; Standard data: Store cleaned enterprise data in a standardized database by category, and identify the data source and related flow of enterprise data; Monitoring and analysis: Monitor and analyze the process and quality of data governance, monitor the cleaned standardized database, and perform quality control on the enterprise data stored in the standardized database.

Citation Information

Patent Citations

  • Enterprise intelligent risk control method and equipment

    CN113537807A

  • Enterprise financing fund matching method based on data mining

    CN114266492A

  • Quantitative statistical method for risk formation based on enterprise characteristics and operation behaviors

    CN114723287A

  • Method for constructing enterprise portrait model for assisting government investment decision

    CN114819618A

  • Digital intelligent matching method for financial products of small and micro enterprises

    CN117611318A

Cited By

  • Method for extracting spatial distribution characteristics of different types of enterprises in combination with large model

    CN120492881A

  • Data governance quality intelligent detection system and method based on large model

    CN121255793A