Model verification method and apparatus, credit evaluation model verification method and apparatus, and computing device

Abstract

The embodiment of the invention provides a model verification method and device, a credit evaluation model verification method and device and computing equipment, and the model verification method comprises the steps that a verification data set from a target project is obtained, and the verification data set comprises user data of multiple dimensions in the target project and project results corresponding to the user data; performing feature analysis on the user data, and extracting a target dimension from a plurality of dimensions; generating at least one project scene of the target project based on the target dimension; respectively inputting the user data in the at least one project scene into a machine learning model applied to the target project to obtain a predicted project result corresponding to the user data in the at least one project scene; according to the invention, project scene matching is carried out based on the predicted project result and the project result corresponding to the user data, the matching degree of the machine learning model and the target project is determined, the accuracy and efficiency of model verification are improved, and the cost of model verification is reduced.

Description

Technical Field

[0001] The embodiments of this specification relate to the technical field of machine learning, and more particularly to a method, apparatus, and computing device for model verification and credit assessment model verification. Background Art

[0002] With the development of artificial intelligence and machine learning technologies, machine learning models have been widely used in projects in various industries and application fields, greatly improving the efficiency of projects and the quality of decision-making. For example, in the field of healthcare, these models can assist in disease diagnosis and improve the accuracy and speed of diagnosis. For another example, in the e-commerce industry, machine learning models can optimize inventory management and personalized recommendation systems, enhance user experience and improve recommendation accuracy. For another example, in the field of intelligent customer service, models can provide more natural and smooth human-computer dialogue and improve customer service experience. For another example, in the field of content delivery, algorithms can accurately push content based on user preferences to improve user engagement. For another example, in the credit industry, machine learning models can be used for credit assessment to improve the speed and accuracy of credit assessment.

[0003] At present, in order to continuously adapt to the changing project environment and project requirements, frequent iterative updates of machine learning models have become the norm. However, each model iteration update may introduce uncertainty and risk, and the matching degree between the machine learning model and the actual project needs to be verified.

[0004] However, due to the limitation of human imagination, it is impossible to directly construct a verification scenario suitable for the real world from massive data to verify the validity of the machine learning model. As a result, the verification results often do not conform to the real world situation, and the accuracy of model verification is insufficient. In addition, the manual construction process seriously affects the efficiency of model verification and increases the verification cost. Therefore, a model verification method with high accuracy, high efficiency and low cost is needed. Summary of the invention

[0005] In view of this, an embodiment of this specification provides a model verification method. One or more embodiments of this specification also relate to a verification method of a credit assessment model, a model verification device, a verification device of a credit assessment model, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0006] According to a first aspect of an embodiment of this specification, a model checking method is provided, including:

[0007] Acquire a verification data set from a target project, wherein the verification data set includes user data of multiple dimensions in the target project and project results corresponding to the user data;

[0008] Perform feature analysis on user data and extract target dimensions from multiple dimensions;

[0009] Based on the target dimension, generating at least one project scenario for the target project;

[0010] Inputting user data in at least one project scenario into a machine learning model applied to a target project, respectively, to obtain a predicted project result corresponding to the user data in at least one project scenario;

[0011] The project scenarios are matched based on the predicted project results and project results corresponding to the user data to determine the matching degree between the machine learning model and the target project.

[0012] According to a second aspect of the embodiments of this specification, a method for verifying a credit assessment model is provided, comprising:

[0013] Acquire a verification data set from a credit project, wherein the verification data set includes user attribute data and user behavior data of multiple dimensions in the credit project, and project results corresponding to the user attribute data and the user behavior data;

[0014] Perform feature analysis on user attribute data and user behavior data, and extract target dimensions from multiple dimensions;

[0015] Based on the target dimension, generate at least one project scenario for the credit project;

[0016] Inputting user data in at least one project scenario into a credit assessment model applied to a credit project, respectively, to obtain a predicted project result corresponding to user attribute data and user behavior data in at least one project scenario;

[0017] The project scenario is matched based on the predicted project results and project results corresponding to the user attribute data and user behavior data to determine the matching degree between the credit assessment model and the credit project.

[0018] According to a third aspect of the embodiments of this specification, a model checking device is provided, including:

[0019] A first acquisition module is configured to acquire a verification data set from a target project, wherein the verification data set includes user data of multiple dimensions in the target project and project results corresponding to the user data;

[0020] A first extraction module is configured to perform feature analysis on user data and extract a target dimension from multiple dimensions;

[0021] A first generating module is configured to generate at least one project scenario of a target project based on a target dimension;

[0022] A first prediction module is configured to input user data in at least one project scenario into a machine learning model applied to a target project, respectively, to obtain a prediction project result corresponding to the user data in at least one project scenario;

[0023] The first matching module is configured to perform project scenario matching based on the predicted project results and project results corresponding to the user data, and determine the matching degree between the machine learning model and the target project.

[0024] According to a fourth aspect of the embodiments of this specification, a verification device for a credit assessment model is provided, including:

[0025] A second acquisition module is configured to acquire a verification data set from a credit project, wherein the verification data set includes user attribute data and user behavior data of multiple dimensions in the credit project, and project results corresponding to the user attribute data and the user behavior data;

[0026] A second extraction module is configured to perform feature analysis on the user attribute data and the user behavior data, and extract a target dimension from the multiple dimensions;

[0027] A second generating module is configured to generate at least one project scenario of a credit project based on the target dimension;

[0028] The second prediction module is configured to input the user data in at least one project scenario into the credit assessment model applied to the credit project, and obtain the prediction project result corresponding to the user attribute data and the user behavior data in at least one project scenario;

[0029] The second matching module is configured to perform project scenario matching based on the predicted project results and project results corresponding to the user attribute data and the user behavior data, and determine the matching degree between the credit assessment model and the credit project.

[0030] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0031] Memory and processor;

[0032] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above method are implemented.

[0033] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and the instructions implement the steps of the above method when executed by a processor.

[0034] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, wherein when the computer program product is executed in a computer, the computer is caused to execute the steps of the above method.

[0035] In one embodiment of the present specification, a verification data set from a target project is obtained, wherein the verification data set includes user data of multiple dimensions in the target project and project results corresponding to the user data, and real user data and corresponding project results are obtained from the actual project, thereby avoiding the limitations of artificially constructed data, improving the quality and authenticity of the data, and making the verification results more consistent with the real target project. Feature analysis is performed on the user data, and the target dimension that has a great impact on the project results is extracted from multiple dimensions. Based on the target dimension, at least one project scenario of the target project is generated, ensuring that the generated project scenario can target the target dimension with a great impact, and achieving comprehensive coverage of real project scenarios under different conditions. The user data under at least one project scenario is respectively input into the machine learning model applied to the target project, and the predicted project results corresponding to the user data under at least one project scenario are obtained. The project scenario is matched based on the predicted project results corresponding to the user data and the project results, and the matching degree between the machine learning model and the target project is determined. In the generated project scenario under multiple conditions that meet the real world, a comprehensive automatic verification of the machine learning model is completed, which improves the accuracy and efficiency of the model verification and reduces the cost of the model verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of a model verification method provided by an embodiment of this specification;

[0037] Figure 2 It is a flow chart of a verification method of a credit assessment model provided by an embodiment of this specification;

[0038] Figure 3 It is a process flow chart of a verification method of a credit assessment model applied to user rating provided by an embodiment of this specification;

[0039] Figure 4 It is a structural schematic diagram of a model verification device provided by an embodiment of this specification;

[0040] Figure 5 It is a structural schematic diagram of a verification device for a credit assessment model provided by an embodiment of this specification;

[0041] Figure 6 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0042] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0043] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0044] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0045] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0046] First, the terms involved in one or more embodiments of this specification are explained.

[0047] Model validation: The process of evaluating the ability of a machine learning model to accurately predict or classify in a specific application scenario.

[0048] Metric calculation: The process of evaluating model performance through quantitative indicators, such as accuracy, recall, etc.

[0049] Scenario modeling: Building models based on specific conditions or environments to simulate scenarios where events occur in the real world in order to test and validate the performance of machine learning models.

[0050] Feature analysis: The process of identifying variables or attributes from a dataset that have an impact on model predictions.

[0051] Decision tree algorithm: A supervised learning method that recursively splits the data set into subsets to form a tree structure, where each internal node represents a test on a feature, each branch represents a test result, and each leaf node represents a category or output value. Decision trees classify or regress data through a series of rules.

[0052] Random field algorithm: A type of probabilistic graphical model used to describe and model complex data distributions and dependencies. Common random field algorithms include Markov Random Field (MRF) and Conditional Random Field (CRF). These models represent the interdependencies between variables through probability distribution.

[0053] Clustering algorithm: An unsupervised learning method used to group objects in a data set into multiple clusters so that objects within the same cluster are more similar, while objects in different clusters are less similar. Common clustering algorithms include K-means clustering (K-means), hierarchical clustering (Hierarchical Clustering) and DBSCAN (Density-Based Spatial Clustering of Applications with Noise).

[0054] Genetic Algorithm: A global optimization search algorithm based on natural selection and genetic mechanisms. It gradually optimizes the population of solutions by simulating the selection, crossover and mutation operations in the biological evolution process to find the best solution to the problem. Genetic algorithms are often used in optimization problems, scheduling problems and machine learning hyperparameter tuning.

[0055] Bayesian algorithm: A probabilistic statistical method based on Bayes' theorem, used to infer the probability distribution of unknown parameters given observed data. The Bayesian algorithm updates the posterior probability through the prior probability and likelihood function, and is often used in classification, regression, prediction, and decision analysis. Common Bayesian models include the Naive Bayesian classifier and the Bayesian network.

[0056] Wide table: A data table with a large number of columns. These columns can come from different sources and may include various aggregated and summarized data. Wide tables are used to store lengthy data to support complex queries such as analysis and mining.

[0057] Narrow table: A data table that contains only a small number of columns (such as a primary key and a few attributes). This design concept is used to solve data storage and access problems in a single project scenario, thereby improving query efficiency and reducing storage space.

[0058] Consistent coding: A coding method adopted to ensure that data from different sources or environments can be accurately compared, ensuring that the same data structure corresponds to the same coding.

[0059] Hash coding: A method of converting data into a fixed-length string, usually used for fast lookup and data integrity verification. In machine learning, hash coding can be used to convert high-dimensional sparse features into low-dimensional dense features to reduce storage space and improve computational efficiency.

[0060] MD5 (Message-Digest Algorithm 5): A widely used hash function that can accept input of any length and generate a 128-bit (16-byte) hash value. MD5 is often used for data integrity verification.

[0061] A / B testing: A statistical experimental method that randomly assigns users to two or more groups (Group A and Group B), displays different versions (such as website pages, application interfaces, etc.), and then collects and analyzes user behavior data to evaluate which version performs better. In the verification of machine learning models, A / B testing can be used to compare the performance of different models or model versions in actual projects, so as to select a better model.

[0062] At present, the model verification method represented by A / B testing uses artificially constructed data and subjective judgment. Not only does it consume a lot of resources and have a long cycle in the artificial construction process, but also due to the limitations of human imagination, it is impossible to directly construct a verification scenario suitable for the real world from massive data to verify the effectiveness of the machine learning model. As a result, the verification results often do not conform to the real-world situation and the accuracy of the model verification is insufficient.

[0063] In response to the above problems, a model verification method is provided in this specification. This specification also involves a verification method for a credit assessment model, a model verification device, a verification device for a credit assessment model, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0064] See also Figure 1 , Figure 1 A flow chart of a model verification method provided according to an embodiment of this specification is shown, comprising the following specific steps:

[0065] Step 102: Acquire a verification data set from a target project, wherein the verification data set includes user data of multiple dimensions in the target project and project results corresponding to the user data.

[0066] The embodiments of this specification are applied to data processing applications, platforms or systems with model checking functions.

[0067] The target project is a specific application or project field to which the machine learning model is applied. The target field includes a large amount of user data and project results corresponding to the user data. For example, the medical and health field, the e-commerce industry, the intelligent customer service field, the content delivery field, and the credit field. It should be noted that the specific applications or project fields exemplified in the embodiments of this specification do not limit the target projects and the data of the target projects. When it is impossible to enumerate all applications and project fields, they only serve to facilitate the explanation.

[0068] The verification data set is a basic data set used to verify the fitness of the machine learning model and the target project. The verification data set is obtained from the target project and contains a data set of user data of multiple dimensions and project results corresponding to the user data. For example, the verification data set consists of patient data and diagnosis results in the medical and health field. Another example is the verification data set consisting of user data and purchase results in the e-commerce industry. Another example is the verification data set consisting of user query content and reply results in the field of intelligent customer service. Another example is the verification data set consisting of user content browsing records and click behaviors in the field of content delivery. Another example is the verification data set between user data and credit assessment results in the credit field.

[0069] Dimensions are attribute dimensions to which user data belongs, and each dimension represents the attributes of a specific type of user data. For example, in the field of medical health, multiple dimensions may include the patient's medical history, test results, family medical history, etc. For another example, in the e-commerce industry, multiple dimensions may include the user's browsing history, purchase history, search keywords, user reviews, etc. For another example, in the field of intelligent customer service, multiple dimensions may include the user's query content, emotional state, service history, user feedback, etc. For another example, in the field of content delivery, multiple dimensions may include the user's interest preferences, browsing history, content exposure times, etc. For another example, in the field of credit, multiple dimensions may include the user's education, occupation, credit record, etc.

[0070] User data is information data related to users in the target project. User data is used as model input for verifying machine learning models, including at least one of static user attribute data and dynamic user behavior data. For example, in the field of medical health, user data may include patient medical history data, examination result data, family medical history data, etc. For another example, in the e-commerce industry, user data may include user browsing history data, purchase history data, search keyword data, user evaluation data, etc. For another example, in the field of intelligent customer service, user data may include user query content data, emotional state data, service history data, user feedback data, etc. For another example, in the field of content delivery, user data may include user interest preference data, browsing history data, content exposure times data, etc. For another example, in the field of credit, user data may include user education data, occupation data, credit record data, etc.

[0071] The project result is the project output result corresponding to the user data in the target project. The project result is the real output result in the target project. The project result is used to verify the label output of the machine learning model and is compared with the predicted project result output by the model to complete the verification of the machine learning model. The project results can be binary (such as whether the disease is serious, whether the content is clicked, whether it meets the information level) or multi-category (such as the type of disease, the category of content, and the credit rating). For example, in the field of medical health, the project results can be whether the disease is serious and the type of disease. For another example, in the e-commerce industry, the project results can be whether the product is purchased. For another example, in the field of intelligent customer service, the project results can be the reply results. For another example, in the field of content delivery, the project results can be whether the click is made and the category of the content. For another example, in the field of credit, the project results can be whether the information level and credit rating are met.

[0072] An optional way to obtain a verification data set from the target project is to use big data processing technology to extract user data of multiple dimensions and project results corresponding to the user data from the target project, and to construct a verification data set based on the user data of multiple dimensions and project results corresponding to the user data.

[0073] For example, before a new version of a product recommendation model replaces an old version of a product recommendation model and is deployed on an e-commerce platform, the validity of the new version of the product recommendation model needs to be verified.

[0074] Using big data processing tools such as Hadoop or Spark, we extract 200 dimensions of user data (registration time, geographic location, occupation, education level, browsing history, purchase history, search keywords, user reviews, items in the shopping cart, user feedback, user interaction behavior, user device information, etc.) from the log database, transaction record database, and user database of the e-commerce platform, and the four project results corresponding to the user data:

[0075] User data: registration time data, geographic location data, occupation data, education level data, browsing history data, purchase history data, search keyword data, user evaluation data, shopping cart product data, user feedback data, user interaction behavior data, user device information data...

[0076] Project results: whether the user clicked on the recommended product (clicked / not clicked); whether the user purchased the recommended product (purchased / not purchased); the user's evaluation of the recommended product (good / medium / bad); the user's satisfaction score for the recommendation system (e.g., 1-5 points).

[0077] Based on the above 200 dimensions of user data and 4 project results, a verification data set is constructed.

[0078] Obtain a verification data set from the target project, where the verification data set includes user data of multiple dimensions in the target project and project results corresponding to the user data. Obtaining real user data and corresponding project results from the actual project avoids the limitations of artificially constructed data, improves the quality and authenticity of the data, and makes the verification results more consistent with the real target project.

[0079] Step 104: Perform feature analysis on the user data and extract the target dimension from multiple dimensions.

[0080] The target dimension is a dimension extracted from multiple dimensions that has a significant impact on the model decision and is a key dimension that needs to be focused on during the model verification process. For example, in the field of medical health, the target dimension may include the patient's medical history and test results. For another example, in the e-commerce industry, the target dimension may include the user's browsing history and purchase history. For another example, in the field of intelligent customer service, the target dimension may include the user's query content, emotional state, service history, and user feedback. For another example, in the field of content delivery, the target dimension may include interest preferences, browsing history, and content exposure times. For another example, in the field of credit, the target dimension may include the user's occupation and credit record.

[0081] Exemplarily, statistical algorithms and machine learning algorithms are used to perform feature analysis on user data (registration time data, geographic location data, occupation data, education level data, browsing history data, purchase history data, search keyword data, user evaluation data, shopping cart item data, user feedback data, user interaction behavior data, user device information data...), and 20 target dimensions are extracted from 200 dimensions: geographic location, occupation, browsing history, purchase history, search keywords...

[0082] By performing feature analysis on user data and extracting target dimensions that have a great impact on project results from multiple dimensions, the scientific nature and pertinence of data dimension selection are improved, and data dimension support is provided for the subsequent generation of real project scenarios that fully cover different conditions.

[0083] Step 106: Generate at least one project scenario of the target project based on the target dimension.

[0084] The project scenario of the target project is a real project scenario used to simulate different conditions of the target project. The generation of at least one project scenario ensures that the model can cover the real situation under different conditions during the verification process, so as to comprehensively evaluate the performance of the machine learning model. For example, in the field of medical health, the generated project scenarios include: Project Scenario 1: A young male patient with a history of hypertension, recent examination results show that the blood pressure is high, whether he suffers from cardiovascular disease; Project Scenario 2: A middle-aged female patient with no obvious medical history, recently developed headache symptoms, whether he suffers from brain disease. For another example, in the e-commerce industry, the generated project scenarios include: Project Scenario 1: A young female user who often browses clothing products, recently searched for "dress", and purchased clothing consumer goods; Project Scenario 2: A middle-aged male user, whose purchase history is mainly concentrated in electronic products, recently searched for "laptop computer", and purchased electronic consumer goods. For another example, in the field of intelligent customer service, the generated project scenarios include: Project Scenario 1: The user asks how to reset the password and is anxious; Project Scenario 2: The user feedback is dissatisfied with a certain service and requires reprocessing. For example, in the field of content delivery, the generated project scenarios include: Project scenario 1: young users, whose interests are travel, have recently browsed many travel articles; Project scenario 2: middle-aged users, whose interests are technology, have recently browsed many technology news. For example, in the field of credit, the generated project scenarios include: Project scenario 1: young users, who have no credit record, apply for small credit projects; Project scenario 2: middle-aged users, who have good credit records, apply for large credit projects.

[0085] Based on the target dimension, at least one project scenario of the target project is generated. An optional method is: using a decision tree algorithm to generate at least one project scenario of the target project based on the target dimension. Specifically, use the target dimension as a feature and the project result as a label to train a decision tree model, extract rules from the decision tree, each leaf node represents a project scenario, and generate a specific project scenario based on the extracted rules.

[0086] Based on the target dimension, generate at least one project scenario of the target project. Another optional method is: use a random field algorithm to generate at least one project scenario of the target project based on the target dimension. Specifically, use the target dimension as a feature and the project result as a label to train a random field model (such as conditional random field CRF), and use the model to generate project results under different dimensional combinations to form a project scenario.

[0087] Based on the target dimension, generate at least one project scenario of the target project. Another optional method is: use a clustering algorithm to generate at least one project scenario of the target project based on the target dimension. Specifically, use the target dimension as a feature to train a clustering model (such as K-means, DBSCAN, etc.). Each cluster center and its nearby points can represent a project scenario.

[0088] Based on the target dimension, at least one project scenario of the target project is generated. Another optional method is to use a genetic algorithm to generate at least one project scenario of the target project based on the target dimension. Specifically, a set of initial project scenarios is randomly generated. A fitness function is defined to evaluate the rationality of each project scenario. Through operations such as selection, crossover, and mutation, the project scenarios are gradually optimized to finally obtain a set of optimized project scenarios.

[0089] Based on the target dimension, at least one project scenario of the target project is generated. Another optional method is: using a Bayesian algorithm to generate at least one project scenario of the target project based on the target dimension. Specifically, a probabilistic dependency relationship between the target dimension and the project result is defined, and a Bayesian network is used to generate different samples, each sample representing a project scenario.

[0090] For example, a genetic algorithm is used to generate five scenarios for product recommendations on an e-commerce platform based on 20 target dimensions (geographic location, occupation, browsing history, purchase history, search keywords, etc.):

[0091] Project scenario 1: Young female users who frequently browse fashion products, have recent purchase records, and search for keywords related to fashion.

[0092] Project scenario 2: middle-aged male users, who frequently browse electronic products, have a purchase history focused on high-priced items, and search for keywords related to electronic products.

[0093] Project scenario 3: Elderly users, who frequently browse health products and daily necessities, have low purchase frequency, and search for health-related keywords.

[0094] Project scenario 4: Teenagers who frequently browse anime peripherals and game products, have frequent purchase histories, and search for keywords related to entertainment.

[0095] Project scenario 5: middle-aged female users, often browse household items and children's products, purchase history is concentrated on household items, and search keywords are related to family life.

[0096] Based on the target dimension, at least one project scenario for the target project is generated, ensuring that the generated project scenario can target the target dimension with great influence, achieving comprehensive coverage of real project scenarios under different conditions, and providing scenario support for subsequent prediction and matching processes.

[0097] Step 108: Input the user data in at least one project scenario into the machine learning model applied to the target project to obtain the predicted project results corresponding to the user data in at least one project scenario.

[0098] The machine learning model is a machine learning model specifically applied to the target project. The model has learned the patterns and rules in the historical data of the target project through pre-training, and can predict or classify new data to adapt to the project environment and project requirements of the target project. Machine learning models include but are not limited to: linear regression model, logistic regression model, decision tree model, random forest model, clustering model, support vector machine model, neural network model, large language model and generative Transformer model.

[0099] The predicted project result is the project output result corresponding to the user data output by the machine learning model. The predicted project result is the output result of the simulation under at least one generated project scenario. The predicted output result is used to verify the predicted output of the machine learning model and is compared with the project result to complete the verification of the machine learning model. The predicted project result can be binary (such as whether the disease is serious, whether the content is clicked, whether it meets the information level) or multi-category (such as the type of disease, the category of content, and the credit rating). For example, in the field of medical health, the predicted project result can be whether the disease is serious and the type of disease. For another example, in the e-commerce industry, the predicted project result can be whether to purchase goods. For another example, in the field of intelligent customer service, the predicted project result can be a reply result. For another example, in the field of content delivery, the predicted project result can be whether to click and the category of content. For another example, in the field of credit, the predicted project result can be whether to meet the information level and credit rating.

[0100] Input the user data in at least one project scenario into the machine learning model applied to the target project, and obtain the predicted project results corresponding to the user data in at least one project scenario. An optional method is: according to at least one project scenario, input the user data into the machine learning model applied to the target project, and obtain the predicted project results corresponding to the user data in at least one project scenario.

[0101] Exemplarily, according to five project scenarios (project scenario 1, project scenario 2, project scenario 3, project scenario 4, and project scenario 5), user data are respectively input into the retrieval recall model applied to product recommendation to obtain four prediction project results corresponding to the user data under the five project scenarios: whether the user clicks on the recommended product (clicked / not clicked); whether the user purchases the recommended product (purchased / not purchased); the user's evaluation of the recommended product (good / medium / bad); and the user's satisfaction score for the recommendation system (e.g., 1-5 points).

[0102] The user data in at least one project scenario are respectively input into the machine learning model applied to the target project to obtain the predicted project results corresponding to the user data in at least one project scenario, thereby ensuring the comprehensiveness and accuracy of the predicted project results and improving the effect of subsequent model testing.

[0103] Step 110: Perform project scenario matching based on the predicted project results and project results corresponding to the user data to determine the matching degree between the machine learning model and the target project.

[0104] The matching degree between the machine learning model and the target project is a quantitative indicator of the applicability and performance of the machine learning model in the target project. The matching degree evaluates the effectiveness and risk of the machine learning model in the target project. The matching degree can also be used to guide the future verification process and further optimize the verification process. Including but not limited to at least one of the following: coding coverage, accuracy and recall.

[0105] Based on the predicted project results and project results corresponding to the user data, project scenarios are matched to determine the matching degree between the machine learning model and the target project. An optional method is to compare the predicted project results and project results corresponding to the user data in at least one project scenario to determine the matching degree between the machine learning model and the target project.

[0106] Exemplarily, by comparing 4 predicted project results and 4 project results corresponding to user data in 5 project scenarios (whether the user clicked on the recommended product (clicked / not clicked); whether the user purchased the recommended product (purchased / not purchased); the user's evaluation of the recommended product (good / medium / bad); the user's satisfaction score for the recommendation system (e.g., 1-5 points)), it is determined that the match degree between the retrieval recall model and the product recommendation is 65%.

[0107] In the embodiment of this specification, a verification data set from a target project is obtained, wherein the verification data set includes user data of multiple dimensions in the target project and project results corresponding to the user data, and real user data and corresponding project results are obtained from the actual project, thereby avoiding the limitations of artificially constructed data, improving the quality and authenticity of the data, and making the verification results more consistent with the real target project. Feature analysis is performed on the user data, and the target dimension that has a great impact on the project results is extracted from multiple dimensions. Based on the target dimension, at least one project scenario of the target project is generated, ensuring that the generated project scenario can target the target dimension with a great impact, and achieving comprehensive coverage of real project scenarios under different conditions. The user data under at least one project scenario is respectively input into the machine learning model applied to the target project, and the predicted project results corresponding to the user data under at least one project scenario are obtained. The project scenario is matched based on the predicted project results and project results corresponding to the user data, and the matching degree of the machine learning model and the target project is determined. In the generated project scenario under multiple conditions that meet the real world, a comprehensive automatic verification of the machine learning model is completed, which improves the accuracy and efficiency of the model verification and reduces the cost of the model verification.

[0108] In an optional embodiment of the present specification, step 104 includes the following specific steps:

[0109] Counting data features of user data, and extracting target dimensions from multiple dimensions based on the data features, wherein the data features include at least one of discrete features, distribution features, and correlation features; and / or, using a pre-trained feature analysis model to perform feature analysis on user data, and extracting target dimensions from multiple dimensions, wherein the feature analysis model is trained based on sample user data of multiple sample dimensions, and the multiple sample dimensions are annotated with label dimensions.

[0110] The data features of user data are statistical and attribute features that describe user data. The data features include at least one of discreteness features, distribution features, and correlation features. They are used to analyze the distribution, discreteness, and correlation between dimensions of user data, and then extract dimensions that have a significant impact on model decisions.

[0111] The dispersion feature describes the degree of dispersion of data. The dispersion feature can include variance, standard deviation, etc. For example, in the e-commerce industry, the standard deviation of the purchase quantity data of goods can reflect the fluctuation of the purchase quantity of goods. If the standard deviation is large, it means that the purchase quantity of goods fluctuates greatly; if the standard deviation is small, it means that the purchase quantity of goods is relatively stable.

[0112] Distribution characteristics describe the distribution of data, and can include mean, median, mode, skewness, kurtosis, etc. For example, in the field of medical health, the mean and median of a patient's medical history data can reflect the central trend of the patient's medical history. If the mean and median are close, it means that the distribution of the patient's medical history is relatively uniform; if the mean and median are significantly different, it means that the distribution of the patient's medical history may be skewed.

[0113] The correlation feature is used to describe the correlation between different dimensions. The correlation feature may include correlation coefficient, mutual information, etc. For example, in the field of intelligent customer service, the correlation coefficient between the user's query content and emotional state can reflect the relationship between the two. If the correlation coefficient is high, it means that the user's query content and his emotional state are strongly correlated.

[0114] The feature analysis model is a machine learning model used to automatically extract key features from user data. The model is trained based on sample user data of multiple sample dimensions. Since multiple sample dimensions are annotated with label dimensions, the model learns which dimensions have a significant impact on model predictions, thereby extracting target dimensions from complex and massive dimensions. Feature analysis models include but are not limited to: decision tree model, random forest model, support vector machine model, neural network model.

[0115] Exemplarily, the discreteness characteristics (variance, standard deviation), distribution characteristics (mean, median, mode, skewness, kurtosis) and correlation (correlation coefficient, mutual information) characteristics of user data (registration time data, geographic location data, occupation data, education level data, browsing record data, purchase history data, search keyword data, user evaluation data, shopping cart product data, user feedback data, user interaction behavior data, user device information data...) are statistically analyzed, and based on these discreteness characteristics, distribution characteristics and correlation characteristics, 14 target dimensions are extracted from 200 dimensions.

[0116] Exemplarily, a pre-trained support vector machine model is used to perform feature analysis on user data (registration time data, geographic location data, occupation data, education level data, browsing record data, purchase history data, search keyword data, user evaluation data, shopping cart item data, user feedback data, user interaction behavior data, user device information data, etc.), and 18 target dimensions are extracted from 200 dimensions.

[0117] In the embodiments of this specification, statistical algorithms and / or machine learning algorithms are used to complete feature analysis, accurately identify target dimensions that have an impact on model predictions, combine the inherent characteristics and logic of user data, improve the scientific nature and pertinence of data dimension selection, and provide more accurate data dimension support for the subsequent generation of real project scenarios that fully cover different conditions.

[0118] In an optional embodiment of the present specification, before step 108, the following specific steps are also included:

[0119] Use the domain knowledge of the target project to filter the target dimensions.

[0120] The domain knowledge of the target project is the expertise, experience and data accumulated in the project domain of the target project, which can help to more accurately understand and process the dimensions to which user data belongs. Domain knowledge includes but is not limited to: knowledge graph, knowledge base, and expert domain knowledge. Among them, the knowledge graph is a structured knowledge representation form that describes the knowledge in the domain through entities, attributes and relationships. For example, in the field of medical health, the knowledge graph can include entities such as diseases, symptoms, drugs, treatment methods and the relationships between them. Among them, the knowledge base is a system for storing and managing professional knowledge in the field, usually containing a large amount of structured and semi-structured data. For example, in the e-commerce industry, the knowledge base can include information such as product classification, user behavior patterns, and market trends. Among them, expert domain knowledge refers to the experience, insights and suggestions provided by experts or professionals in the field. This knowledge can be obtained through interviews, questionnaires, literature reviews, etc. For example, in the field of intelligent customer service, experts can provide knowledge about common user problems, response strategies, etc.

[0121] Exemplarily, expert domain knowledge of product recommendation is used to screen 14+18 target dimensions to obtain 20 target dimensions.

[0122] In the embodiments of this specification, based on dimension extraction, the target dimensions are secondary screened and optimized in combination with the domain knowledge of the target project to ensure that the selected target dimensions are both representative and in line with the project environment and project requirements of the target project, providing more accurate data dimension support for the subsequent generation of real project scenarios that fully cover different conditions.

[0123] In an optional embodiment of the present specification, user data and project results corresponding to the user data are recorded in a first data table, and user data and predicted project results in at least one project scenario are recorded in a second data table;

[0124] Correspondingly, before step 110, the following specific steps are also included:

[0125] Encode the user data and project results of the target dimension in the first data table to obtain a first code of at least one project scenario; encode the user data and predicted project results of the target dimension in the second data table to obtain a second code of at least one project scenario;

[0126] Correspondingly, step 110 includes the following specific steps:

[0127] Compare the first code and the second code to determine the matching degree between the machine learning model and the target project.

[0128] The first data table is structured data that records the user data in the target project and the project results corresponding to the user data. The first data table can be a wide table, used to record user data and corresponding project results in multiple dimensions, suitable for batch processing and statistical analysis, or multiple narrow tables, used to record detailed data of each user data in different dimensions, suitable for fine-grained analysis and dynamic evaluation, which is not limited here.

[0129] The second data table is structured data that records user data and predicted project results in at least one project scenario. The second data table can be a wide table for recording user data of a target dimension and corresponding predicted project results, suitable for batch processing and statistical analysis, or multiple narrow tables for recording detailed data of each user data in different target dimensions, suitable for fine-grained analysis and dynamic evaluation, which is not limited here.

[0130] The first code is the coding result obtained by coding the user data and project results of the target dimension in the first data table. The purpose of coding is to convert the original data into a unified format to facilitate comparison with the data in the verification process.

[0131] The second code is the coding result obtained by coding the user data of the target dimension and the predicted project results in the second data table. The purpose of coding is to convert the predicted data into a unified format to facilitate comparison with the actual data.

[0132] It should be noted that the first code and the second code are unique coding identifiers generated for each project scenario through a consistent coding algorithm, wherein consistent coding methods include but are not limited to: hash coding and Base64 coding.

[0133] Exemplarily, the user data of 20 target dimensions in the first data table (geographic location data, occupational data, browsing record data, purchase history data, search keyword data, etc.) and 4 types of project results (whether the user clicks on the recommended product (clicked / not clicked); whether the user purchases the recommended product (purchased / not purchased); the user's evaluation of the recommended product (good / medium / bad review); the user's satisfaction score for the recommendation system (such as 1-5 points)) are consistently coded to obtain the first codes of 5 project scenarios: a11c240cc... The user data of 20 target dimensions in the second data table (geographic location data, occupational data, browsing record data, purchase history data, search keyword data, etc.) and 4 types of predicted project results (whether the user clicks on the recommended product (clicked / not clicked); whether the user purchases the recommended product (purchased / not purchased); the user's evaluation of the recommended product (good / medium / bad review); the user's satisfaction score for the recommendation system (such as 1-5 points)) are consistently coded to obtain the second codes of 5 project scenarios: a11c2f3cc... By comparing the first and second coding, it was determined that the coding coverage of the machine learning model and the target project was 3%.

[0134] In the embodiments of this specification, by encoding the data table, direct comparability between different project scenarios is ensured, the comprehensiveness and accuracy of model verification are improved, the misjudgment rate in the verification process is reduced, and the reliability and efficiency of model verification are improved.

[0135] In an optional embodiment of the present specification, encoding the user data and project results of the target dimension in the first data table to obtain a first code of at least one project scenario includes the following specific steps:

[0136] Hash-encode the user data and project results of the target dimension in the first data table to obtain a first hash value of at least one project scenario;

[0137] Encoding the user data of the target dimension and the predicted project results in the second data table to obtain a second code of at least one project scenario includes the following specific steps:

[0138] Hash encoding is performed on the user data of the target dimension and the predicted project results in the second data table to obtain a second hash value of at least one project scenario;

[0139] Comparing the first code with the second code to determine the matching degree between the machine learning model and the target project includes the following specific steps:

[0140] Comparing the first hash value and the second hash value to determine the coverage of at least one project scenario;

[0141] Based on the coverage of at least one project scenario, determine the match between the machine learning model and the target project.

[0142] The first hash value is a unique identifier obtained by hashing the user data and project results of the target dimension in the first data table. Hash coding converts the original data into a string of fixed length to ensure that the coded identifier of each project scenario is unique. The way to obtain the first hash value by coding includes but is not limited to: using the MD5 hash function and the SHA-256 hash function. The purpose of coding is to convert the original data into a unified hash value format to facilitate comparison with the data in the verification process.

[0143] The second hash value is a unique identifier obtained by hashing the user data of the target dimension in the second data table and the predicted project results. Hash coding converts the predicted data into a string of fixed length to ensure that the coded identifier of each project scenario is unique. The method of encoding to obtain the second hash value includes but is not limited to: using the MD5 hash function and the SHA-256 hash function. The purpose of encoding is to convert the predicted data into a unified hash value to facilitate comparison with the actual data.

[0144] The coverage of at least one project scenario is the consistency ratio between the model prediction results and the actual project results after comparing the first hash value and the second hash value. The coverage reflects the accuracy and reliability of the model in a specific project scenario. The coverage is used to evaluate the matching degree between the machine learning model and the target project. The higher the coverage, the closer the model's prediction results are to the actual results, and the better the model's performance.

[0145] Exemplarily, the user data (geographic location data, occupation data, browsing record data, purchase history data, search keyword data, etc.) of 20 target dimensions in the first data table and 4 project results (whether the user clicks on the recommended product, whether the user purchases the recommended product, the user's evaluation of the recommended product, and the user's satisfaction score with the recommendation system) are MD5 hashed to generate the first hash value: a11c240cc. The user data (geographic location data, occupation data, browsing record data, purchase history data, search keyword data, etc.) of 20 target dimensions in the second data table and 4 prediction project results (whether the user clicks on the recommended product, whether the user purchases the recommended product, the user's evaluation of the recommended product, and the user's satisfaction score with the recommendation system) are MD5 hashed to generate the second hash value: a11c2f3cc. Comparing the first hash value and the second hash value of the five project scenarios, it is found that the hash value of one project scenario is completely consistent, and the hash values ​​of the other four project scenarios are inconsistent. Therefore, the coverage rate of the five project scenarios is 1 / 5=20%. Based on the coverage of the five project scenarios, the matching degree between the machine learning model and the target project was determined to be 0.2.

[0146] In the embodiments of this specification, by hashing the data table, the uniqueness and comparability of the codes between different project scenarios are ensured, the comprehensiveness and accuracy of the model verification are further improved, the misjudgment rate in the verification process is further reduced, and the reliability and efficiency of the model verification are further improved.

[0147] In an optional embodiment of the present specification, before encoding the user data and project results of the target dimension in the first data table to obtain the first code of at least one project scenario, the following specific steps are also included:

[0148] Preprocessing the user data and the project results in the first data table, wherein the preprocessing includes at least one of modifying the order between target dimensions, modifying the naming of the target dimensions, modifying the mapping relationship between the user data, and modifying the mapping relationship between the user data and the project results;

[0149] Correspondingly, before encoding the user data of the target dimension and the predicted project results in the second data table to obtain the second code of at least one project scenario, the following specific steps are also included:

[0150] The user data and predicted project results in the second data table are preprocessed, wherein the preprocessing includes at least one of modifying the order between target dimensions, modifying the naming of target dimensions, modifying the mapping relationship between user data, and modifying the mapping relationship between user data and predicted project results.

[0151] Modifying the order between target dimensions is to adjust the order of target dimensions in the data table to better meet the requirements of model processing or improve the readability of data. Modifying the order between target dimensions ensures that the data is more orderly during processing, which is convenient for model understanding and processing. The order of target dimensions can be rearranged through programming or data processing tools. For example, in the field of medical health, the original order of target dimensions in the first data table is: occupation, medical history, test results, location, family medical history, and the order is changed to: location, occupation, medical history, test results, family medical history.

[0152] Modify the name of the target dimension to change the name of each target dimension in the data table to better reflect its meaning or improve the readability of the data. Modify the name of the target dimension to ensure that the dimension name in the data table is more intuitive and easy to understand, which is convenient for subsequent data processing and analysis. You can rename the target dimension through programming or data processing tools. For example, in the field of healthcare, the original name of the target dimension in the first data table is: work, and the name can be changed to: occupation.

[0153] Modifying the mapping relationship between user data means adjusting the table association method between user data in the data table to better reflect the actual relationship between the data or improve the data processing efficiency. Modifying the mapping relationship between user data ensures that the relationship between the data is more reasonable and convenient for the understanding and processing of the model. The mapping relationship between user data can be redefined through programming or data processing tools. For example, in the field of intelligent customer service, there is a certain mapping relationship between the query content and the emotional state of the user in the first data table. The query content "how to reset the password" can be mapped to the emotional state "anxiety". In order to better reflect the actual emotions of the user, the mapping relationship can be adjusted to "how to reset the password" mapped to the emotional state "eager".

[0154] Modifying the mapping relationship between user data and project results is to adjust the table association between user data and project results in the data table to better reflect the actual relationship between the data or improve the efficiency of data processing. Modifying the mapping relationship between user data and project results ensures that the relationship between the data is more reasonable and convenient for the understanding and processing of the model. The mapping relationship between user data can be redefined through programming or data processing tools. For example, in the e-commerce industry, there is a certain mapping relationship between the browsing history and purchase history of users in the first data table. The browsing record "browsing a certain mobile phone many times" can be mapped to the purchase history "purchasing this mobile phone". In order to better reflect the actual purchasing behavior of users, the mapping relationship can be adjusted to "browsing a certain mobile phone many times" to the purchase history "adding to the shopping cart but not purchasing".

[0155] In the embodiments of the present specification, by preprocessing the user data and project results in the first data table and the second data table, data inconsistency and redundancy can be effectively eliminated, thereby improving data readability and processing efficiency, ensuring that data features can be captured more accurately during the model verification process, and improving the reliability of the model verification.

[0156] Take the target item as an information item as an example. Figure 2 A flowchart of a method for verifying a credit assessment model provided according to an embodiment of this specification is shown, and includes the following specific steps:

[0157] Step 202: Acquire a verification data set from a credit project, wherein the verification data set includes user attribute data and user behavior data of multiple dimensions in the credit project, and project results corresponding to the user attribute data and user behavior data.

[0158] The embodiments of this specification are applied to data processing applications, platforms or systems with model checking functions.

[0159] Credit items are items used in the credit field to assess the credit status of users such as individuals or enterprises.

[0160] The verification data set is a basic data set used to verify the adaptability of the credit assessment model and the credit project. The verification data set is obtained from the credit project and contains user attribute data and user behavior data in multiple dimensions, as well as a data set of project results corresponding to the user attribute data and user behavior data.

[0161] The dimension is the attribute dimension to which the user attribute data and the user behavior data belong, and each dimension represents an attribute of a specific type of user attribute data and user behavior data.

[0162] User attribute data is data that describes the user's basic information and static characteristics. This data is usually fixed and does not change frequently, reflecting the user's inherent attributes. For example, the user's marital status, educational background, occupation, place of residence, etc.

[0163] User behavior data is data that describes the dynamic behavior and activities of users in a specific time period. These data reflect the real-time behavior and habits of users and usually change over time. For example, consumption behavior data, loan and repayment behavior data, search behavior data, browsing behavior data, social activity data, and transaction behavior data.

[0164] The project result is the project output result corresponding to the user attribute data and user behavior data in the credit project. The project result is the real output result in the credit project. The project result is used to verify the label output of the credit assessment model and is compared with the predicted project result output by the model to complete the verification of the credit assessment model. The project result can be binary or multi-category.

[0165] For example, before a new version of a credit assessment model is deployed in a credit application instead of an old version of the credit assessment model, the validity of the new version of the credit assessment model needs to be verified.

[0166] Using big data processing tools such as Hadoop or Spark, extract 50 dimensions of user attribute data, user behavior data and corresponding 5 project results from the log database, transaction record database and user database of the credit application:

[0167] User attribute data: occupation data, education level data...

[0168] User behavior data: consumption behavior data, loan and repayment behavior data, search behavior data, browsing behavior data, social activity data, transaction behavior data...

[0169] Project results: User rating A, User rating B, User rating C, User rating D, User rating E.

[0170] Based on the above 50 dimensions of user attribute data, user behavior data and the corresponding 5 project results, a verification data set is constructed.

[0171] Obtain a verification data set from a credit project, wherein the verification data set includes user attribute data and user behavior data of multiple dimensions in the credit project, as well as project results corresponding to the user attribute data and user behavior data. Obtaining real user attribute data, user behavior data and corresponding project results from actual projects avoids the limitations of artificially constructed data, improves the quality and authenticity of the data, and makes the verification results more consistent with real credit projects.

[0172] Step 204: Perform feature analysis on the user attribute data and the user behavior data, and extract the target dimension from multiple dimensions.

[0173] Exemplarily, statistical algorithms and machine learning algorithms are used to perform feature analysis on user attribute data (occupation data, education data, etc.) and user behavior data (consumption behavior data, loan repayment behavior data, search behavior data, browsing behavior data, social activity data, transaction behavior data, etc.), and 10 target dimensions are extracted from 50 dimensions:

[0174] Occupation, education level, consumption behavior, borrowing and repayment behavior, trading behavior...

[0175] Perform feature analysis on user attribute data and user behavior data, extract target dimensions from multiple dimensions, improve the scientific nature and pertinence of data dimension selection, and provide data dimension support for the subsequent generation of real project scenarios that fully cover different conditions.

[0176] Step 206: Generate at least one project scenario for the credit project based on the target dimension.

[0177] The project scenario of the credit project is a real project scenario used to simulate different conditions of the credit project. The generation of at least one project scenario ensures that the model can cover the real situation under different conditions during the verification process, so as to comprehensively evaluate the performance of the credit assessment model. For example, in the credit field, the generated project scenarios include: Project Scenario 1: Young users with no credit record apply for small credit projects; Project Scenario 2: Middle-aged users with good credit records apply for large credit projects.

[0178] Based on the target dimension, at least one project scenario of the credit project is generated. An optional method is: using a decision tree algorithm, based on the target dimension, to generate at least one project scenario of the credit project. Specifically, use the target dimension as a feature and the project result as a label to train a decision tree model, extract rules from the decision tree, each leaf node represents a project scenario, and generate specific project scenarios based on the extracted rules.

[0179] Based on the target dimension, generate at least one project scenario of the credit project. Another optional method is: use a random field algorithm to generate at least one project scenario of the credit project based on the target dimension. Specifically, use the target dimension as a feature and the project result as a label to train a random field model (such as conditional random field CRF), and use the model to generate project results under different dimensional combinations to form a project scenario.

[0180] Based on the target dimension, generate at least one project scenario of the credit project. Another optional method is: use a clustering algorithm to generate at least one project scenario of the credit project based on the target dimension. Specifically, use the target dimension as a feature to train a clustering model (such as K-means, DBSCAN, etc.). Each cluster center and its nearby points can represent a project scenario.

[0181] Based on the target dimension, at least one project scenario of the credit project is generated. Another optional method is to use a genetic algorithm to generate at least one project scenario of the credit project based on the target dimension. Specifically, a set of initial project scenarios is randomly generated. A fitness function is defined to evaluate the rationality of each project scenario. Through operations such as selection, crossover, and mutation, the project scenarios are gradually optimized to finally obtain a set of optimized project scenarios.

[0182] Based on the target dimension, at least one project scenario of the credit project is generated. Another optional method is: using a Bayesian algorithm to generate at least one project scenario of the credit project based on the target dimension. Specifically, a probabilistic dependency relationship between the target dimension and the project result is defined, and a Bayesian network is used to generate different samples, each sample representing a project scenario.

[0183] For example, the random field algorithm is used to generate five scenarios in the user rating of the credit platform based on 10 target dimensions (occupation, education level, consumption behavior, loan and repayment behavior, transaction behavior, etc.):

[0184] Project scenario 1: young female users, traditional enterprises, secondary and higher education, high consumption, no borrowing behavior, and frequent trading behavior.

[0185] Project scenario 2: middle-aged male users, IT industry practitioners, high graduate education, medium consumption level, borrowing records and timely repayment, active investment transactions.

[0186] Project scenario 3: Elderly female users, retirees, low high school education, low consumption, no borrowing behavior, occasional small transactions.

[0187] Project scenario 4: Young male users, freelancers, low college degree, mainly online consumption, with overdue repayment records, low transaction frequency but large amount.

[0188] Project scenario 5: Adult female users, education industry, stable secondary master's degree, rational consumption, no borrowing needs, fixed deposits and small investments.

[0189] Based on the target dimension, at least one project scenario of the credit project is generated, ensuring that the generated project scenario can target the target dimension with great influence, achieving comprehensive coverage of real project scenarios under different conditions, and providing scenario support for subsequent prediction and matching processes.

[0190] Step 208: Input the user data in at least one project scenario into the credit assessment model applied to the credit project, and obtain the predicted project results corresponding to the user attribute data and the user behavior data in at least one project scenario.

[0191] The credit assessment model is a machine learning model specifically applied to credit projects. Through pre-training, the model has learned the patterns and rules in the historical data of credit projects and can predict or classify new data to adapt to the project environment and project requirements of credit projects. Credit assessment models include but are not limited to: linear regression model, logistic regression model, decision tree model, random forest model, clustering model, support vector machine model, neural network model, large language model and generative Transformer model.

[0192] The predicted project result is the project output result corresponding to the user data output by the credit assessment model. The predicted project result is the output result of the simulation under at least one generated project scenario. The predicted output result is used to verify the predicted output of the credit assessment model and is compared with the project result to complete the verification of the credit assessment model. The predicted project result can be binary (such as whether the person is sick, whether the content is clicked, whether the information level is met) or multi-category (such as the type of disease, the category of content, and the credit rating).

[0193] Input the user attribute data and user behavior data in at least one project scenario into a credit assessment model applied to the credit project, and obtain the predicted project results corresponding to the user attribute data and the user behavior data in at least one project scenario. An optional method is: according to at least one project scenario, input the user attribute data and the user behavior data into a credit assessment model applied to the credit project, and obtain the predicted project results corresponding to the user attribute data and the user behavior data in at least one project scenario.

[0194] Exemplarily, according to five project scenarios (project scenario 1, project scenario 2, project scenario 3, project scenario 4, and project scenario 5), user attribute data and user behavior data are respectively input into a credit assessment model applied to user rating, and five prediction project results corresponding to the user attribute data and user behavior data under the five project scenarios are obtained:

[0195] User rating A, user rating B, user rating C, user rating D, user rating E.

[0196] The user data in at least one project scenario is input into the credit assessment model applied to the credit project to obtain the predicted project results corresponding to the user attribute data and user behavior data in at least one project scenario, thereby ensuring the comprehensiveness and accuracy of the predicted project results and improving the effect of subsequent model testing.

[0197] Step 210: Perform project scenario matching based on the predicted project results and project results corresponding to the user attribute data and the user behavior data, and determine the matching degree between the credit assessment model and the credit project.

[0198] The matching degree between the credit assessment model and the credit project is a quantitative indicator of the applicability and performance of the credit assessment model in the credit project. The matching degree evaluates the effectiveness and risk of the credit assessment model in the credit project. The matching degree can also be used to guide the future verification process and further optimize the verification process. Including but not limited to at least one of the following: coding coverage, accuracy and recall rate.

[0199] Based on the predicted project results and project results corresponding to the user attribute data and the user behavior data, project scenario matching is performed to determine the matching degree between the credit assessment model and the credit project. An optional method is: comparing the predicted project results and project results corresponding to the user attribute data and the user behavior data in at least one project scenario to determine the matching degree between the credit assessment model and the credit project.

[0200] Exemplarily, by comparing the predicted project results and project results (user rating A, user rating B, user rating C, user rating D, user rating E) corresponding to the user attribute data and user behavior data in 5 project scenarios, it is determined that the matching degree between the credit assessment model and the credit project is 35%.

[0201] In the embodiments of this specification, a verification data set from a credit project is obtained, wherein the verification data set includes user attribute data, user behavior data of multiple dimensions in the credit project, and project results corresponding to the user attribute data and user behavior data. The real user attribute data, user behavior data and corresponding project results are obtained from the actual credit project, avoiding the limitations of artificially constructed data, improving the quality and authenticity of the data, and making the verification results more consistent with the real credit project. Feature analysis is performed on the user attribute data and user behavior data, and the target dimension with a large impact on the project results is extracted from multiple dimensions. Based on the target dimension, at least one project scenario of the credit project is generated, ensuring that the generated project scenario can target the target dimension with a large impact, and achieving full coverage of the real project scenarios of the credit project under different conditions. The user data under at least one project scenario is respectively input into the credit assessment model applied to the credit project, and the predicted project results corresponding to the user attribute data and user behavior data under at least one project scenario are obtained. The project scenario is matched based on the predicted project results and project results corresponding to the user attribute data and user behavior data, and the matching degree of the credit assessment model and the credit project is determined. In the generated project scenario under multiple conditions that meet the real world, a comprehensive automatic verification of the credit assessment model is completed, which improves the accuracy and efficiency of the model verification and reduces the cost of the model verification.

[0202] The following combination Figure 3 Taking the application of the verification method of the credit evaluation model provided in this specification in user rating as an example, the verification method of the credit evaluation model is further explained. Figure 3 A flowchart of a processing process of a verification method of a credit assessment model applied to user rating provided by an embodiment of this specification is shown, including the following specific steps:

[0203] Step 302: Using big data processing technology, extract at least one dimension of user attribute data, user behavior data and corresponding project results from the log database, transaction record database and user database of the credit application, and construct a first data wide table that records user data and project results corresponding to the user data.

[0204] In step 302, a verification data set is provided for subsequent model verification.

[0205] Step 304: Count the discreteness characteristics, distribution characteristics and correlation characteristics of the user attribute data and the user behavior data in the first data wide table, and extract the target dimension from multiple dimensions based on the discreteness characteristics, distribution characteristics and correlation characteristics, and use a pre-trained feature analysis model to perform feature analysis on the user attribute data and the user behavior data in the first data wide table to extract the target dimension from multiple dimensions.

[0206] In step 304, based on the discreteness characteristics, distribution characteristics and correlation characteristics of the data, the target dimensions that have a significant impact on the model decision are automatically screened out from the wide table, combining the intrinsic characteristics and logic of the data to improve the scientificity and pertinence of the target dimension selection.

[0207] Step 306: Utilize the expert knowledge of the credit project to screen the target dimension.

[0208] In step 306, based on the preliminary extraction of the target dimensions, the target dimensions are screened and optimized in combination with domain expert knowledge to ensure that the target features are both representative and meet the needs of the credit project.

[0209] Step 308: Sort and modify the mapping relationship of the user attribute data, user behavior data and project results in the first data wide table, generate at least one project scenario for the credit project based on the target dimension, and perform MD5 hash encoding on the user attribute data, user behavior data and project results of the target dimension in the first data wide table to obtain a first hash value of at least one project scenario.

[0210] In step 308, online data is used to construct a diversified scenario model. A unique coding identifier is generated for each scenario through a consistent coding algorithm, and environmental differences are eliminated through sorting and modifying mapping relationships. The MD5 hash algorithm is used to ensure the uniqueness and comparability of the scenario coding.

[0211] Step 310: According to at least one project scenario, the user attribute data and the user behavior data are respectively input into a credit assessment model applied to user rating, and the predicted project results corresponding to the user attribute data and the user behavior data in at least one project scenario are obtained, and a second data wide table recording the user attribute data, the user behavior data and the predicted project results corresponding to the user data is constructed, and MD5 hash encoding is performed on the user attribute data, the user behavior data and the predicted project results of the target dimension in the second data wide table to obtain a second hash value for at least one project scenario.

[0212] In step 310, consistent coding technology is used to ensure the accuracy of cross-environment verification and improve the reliability and practicality of the verification results.

[0213] Step 312: Compare the first hash value and the second hash value to determine the coverage of at least one project scenario, and based on the coverage of at least one project scenario, determine the matching degree between the credit assessment model and the credit project.

[0214] In step 312, a systematic approach is provided to quantify the coverage of validation data and the risk of model changes.

[0215] In the embodiments of this specification, through refined feature analysis and automated scenario construction, the problems of narrow coverage and low efficiency of traditional methods are overcome, and a more comprehensive and efficient model verification is achieved. The consistency coding technology ensures the accuracy of cross-environment verification, improves the reliability and practicality of the verification results, significantly improves the security and efficiency of model change management, and reduces the project risks of credit projects caused by model problems.

[0216] Corresponding to the above method embodiment, this specification also provides a model verification device embodiment, Figure 4 FIG. 1 shows a schematic diagram of a model verification device provided by an embodiment of the present specification. Figure 4 As shown, the device comprises:

[0217] A first acquisition module 402 is configured to acquire a verification data set from a target project, wherein the verification data set includes user data of multiple dimensions in the target project and project results corresponding to the user data;

[0218] A first extraction module 404 is configured to perform feature analysis on the user data and extract a target dimension from multiple dimensions;

[0219] A first generating module 406 is configured to generate at least one project scenario of a target project based on the target dimension;

[0220] The first prediction module 408 is configured to input the user data in at least one project scenario into the machine learning model applied to the target project, and obtain the predicted project result corresponding to the user data in at least one project scenario;

[0221] The first matching module 410 is configured to perform project scenario matching based on the predicted project results and project results corresponding to the user data, and determine the matching degree between the machine learning model and the target project.

[0222] Optionally, the first extraction module 404 is further configured to: count data features of user data, and extract target dimensions from multiple dimensions based on the data features, wherein the data features include at least one of discrete features, distribution features, and correlation features; and / or, use a pre-trained feature analysis model to perform feature analysis on user data and extract target dimensions from multiple dimensions, wherein the feature analysis model is trained based on sample user data of multiple sample dimensions, and the multiple sample dimensions are annotated with label dimensions.

[0223] Optionally, the device further includes: a first screening module configured to screen the target dimension using domain knowledge of the target project.

[0224] Optionally, the user data and the project results corresponding to the user data are recorded in a first data table, and the user data and the predicted project results in at least one project scenario are recorded in a second data table;

[0225] Correspondingly, the device further includes: a first encoding module configured to encode the user data and project results of the target dimension in the first data table to obtain a first encoding of at least one project scenario; encode the user data and predicted project results of the target dimension in the second data table to obtain a second encoding of at least one project scenario;

[0226] Correspondingly, the first matching module 410 is further configured to: compare the first code and the second code to determine the matching degree between the machine learning model and the target project.

[0227] Optionally, the first encoding module is further configured to: perform hash encoding on the user data and project results of the target dimension in the first data table to obtain a first hash value of at least one project scenario; perform hash encoding on the user data and predicted project results of the target dimension in the second data table to obtain a second hash value of at least one project scenario;

[0228] Correspondingly, the first matching module 410 is further configured to: compare the first hash value and the second hash value to determine the coverage of at least one project scenario; and determine the matching degree between the machine learning model and the target project based on the coverage of at least one project scenario.

[0229] Optionally, the device also includes: a first preprocessing module, configured to preprocess the user data and project results in the first data table, wherein the preprocessing includes at least one of modifying the order between target dimensions, modifying the naming of target dimensions, modifying the mapping relationship between user data, and modifying the mapping relationship between user data and project results; and preprocessing the user data and predicted project results in the second data table, wherein the preprocessing includes at least one of modifying the order between target dimensions, modifying the naming of target dimensions, modifying the mapping relationship between user data, and modifying the mapping relationship between user data and predicted project results.

[0230] In the embodiment of this specification, a verification data set from a target project is obtained, wherein the verification data set includes user data of multiple dimensions in the target project and project results corresponding to the user data, and real user data and corresponding project results are obtained from the actual project, thereby avoiding the limitations of artificially constructed data, improving the quality and authenticity of the data, and making the verification results more consistent with the real target project. Feature analysis is performed on the user data, and the target dimension that has a great impact on the project results is extracted from multiple dimensions. Based on the target dimension, at least one project scenario of the target project is generated, ensuring that the generated project scenario can target the target dimension with a great impact, and achieving comprehensive coverage of real project scenarios under different conditions. The user data under at least one project scenario is respectively input into the machine learning model applied to the target project, and the predicted project results corresponding to the user data under at least one project scenario are obtained. The project scenario is matched based on the predicted project results and project results corresponding to the user data, and the matching degree of the machine learning model and the target project is determined. In the generated project scenario under multiple conditions that meet the real world, a comprehensive automatic verification of the machine learning model is completed, which improves the accuracy and efficiency of the model verification and reduces the cost of the model verification.

[0231] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the model verification device, since it is basically similar to the model verification method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the model verification method embodiment.

[0232] Corresponding to the above method embodiment, this specification also provides a verification device embodiment of a credit assessment model. Figure 5 FIG. 1 is a schematic diagram showing a structure of a verification device for a credit evaluation model provided by an embodiment of this specification. Figure 5 As shown, the device comprises:

[0233] A second acquisition module 502 is configured to acquire a verification data set from a credit project, wherein the verification data set includes user attribute data and user behavior data of multiple dimensions in the credit project, and project results corresponding to the user attribute data and the user behavior data;

[0234] The second extraction module 504 is configured to perform feature analysis on the user attribute data and the user behavior data, and extract a target dimension from multiple dimensions;

[0235] A second generating module 506 is configured to generate at least one project scenario of a credit project based on the target dimension;

[0236] The second prediction module 508 is configured to input the user data in at least one project scenario into the credit assessment model applied to the credit project, and obtain the prediction project result corresponding to the user attribute data and the user behavior data in at least one project scenario;

[0237] The second matching module 510 is configured to perform project scenario matching based on the predicted project results and project results corresponding to the user attribute data and the user behavior data, and determine the matching degree between the credit assessment model and the credit project.

[0238] In the embodiments of this specification, a verification data set from a credit project is obtained, wherein the verification data set includes user attribute data, user behavior data of multiple dimensions in the credit project, and project results corresponding to the user attribute data and user behavior data. The real user attribute data, user behavior data and corresponding project results are obtained from the actual credit project, avoiding the limitations of artificially constructed data, improving the quality and authenticity of the data, and making the verification results more consistent with the real credit project. Feature analysis is performed on the user attribute data and user behavior data, and the target dimension with a large impact on the project results is extracted from multiple dimensions. Based on the target dimension, at least one project scenario of the credit project is generated, ensuring that the generated project scenario can target the target dimension with a large impact, and achieving full coverage of the real project scenarios of the credit project under different conditions. The user data under at least one project scenario is respectively input into the credit assessment model applied to the credit project, and the predicted project results corresponding to the user attribute data and user behavior data under at least one project scenario are obtained. The project scenario is matched based on the predicted project results and project results corresponding to the user attribute data and user behavior data, and the matching degree of the credit assessment model and the credit project is determined. In the generated project scenario under multiple conditions that meet the real world, a comprehensive automatic verification of the credit assessment model is completed, which improves the accuracy and efficiency of the model verification and reduces the cost of the model verification.

[0239] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the verification device of the credit assessment model, since it is basically similar to the verification method embodiment of the credit assessment model, the description is relatively simple, and the relevant parts can be referred to the partial description of the verification method embodiment of the credit assessment model.

[0240] Figure 6 The block diagram of a computing device provided by an embodiment of the present specification is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.

[0241] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a Network Interface Controller (NIC)) of wired or wireless, such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, and a Near Field Communication (NFC).

[0242] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0243] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 600 may also be a mobile or stationary server.

[0244] The processor 620 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-mentioned model verification method or the credit assessment model verification method.

[0245] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the model verification method and the verification method of the credit assessment model described above are of the same concept, and the details not described in detail in the technical scheme of the computing device can be found in the description of the technical scheme of the model verification method or the verification method of the credit assessment model described above.

[0246] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned model verification method or credit assessment model verification method when executed by a processor.

[0247] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the model verification method and the verification method of the credit assessment model described above belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be found in the description of the technical scheme of the model verification method or the verification method of the credit assessment model described above.

[0248] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned model verification method or the credit assessment model verification method.

[0249] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the model verification method and the verification method of the credit assessment model described above belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be found in the description of the technical scheme of the model verification method or the verification method of the credit assessment model described above.

[0250] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0251] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the contents contained in the computer readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0252] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present specification.

[0253] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0254] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A model verification method, comprising: Acquire a verification data set from a target project, wherein the verification data set includes user data of multiple dimensions in the target project and project results corresponding to the user data; Performing feature analysis on the user data, and extracting a target dimension from the multiple dimensions; Based on the target dimension, generating at least one project scenario of the target project; Inputting the user data in the at least one project scenario into the machine learning model applied to the target project, respectively, to obtain the predicted project result corresponding to the user data in the at least one project scenario; Project scenario matching is performed based on the predicted project results and project results corresponding to the user data to determine the matching degree between the machine learning model and the target project.

2. According to the method of claim 1, the step of performing feature analysis on the user data and extracting a target dimension from the multiple dimensions comprises: Counting data features of the user data, and extracting a target dimension from the multiple dimensions based on the data features, wherein the data features include at least one of a dispersion feature, a distribution feature, and a correlation feature; or, A pre-trained feature analysis model is used to perform feature analysis on the user data, and a target dimension is extracted from the multiple dimensions, wherein the feature analysis model is trained based on sample user data of multiple sample dimensions, and the multiple sample dimensions are annotated with label dimensions.

3. The method according to claim 1, before generating at least one project scenario for verification based on the target dimension, further comprising: The target dimension is screened using the domain knowledge of the target project.

4. The method according to any one of claims 1 to 3, wherein the user data and the project results corresponding to the user data are recorded in a first data table, and the user data in the at least one project scenario and the predicted project results are recorded in a second data table; Before determining the matching degree between the machine learning model and the target project based on the predicted project results and project results corresponding to the user data, the method further includes: Encode the user data and project results of the target dimension in the first data table to obtain a first code of at least one project scenario; Encoding the user data and the predicted project results of the target dimension in the second data table to obtain a second code of at least one project scenario; The determining the matching degree between the machine learning model and the target project based on the predicted project result and the project result corresponding to the user data includes: Compare the first encoding and the second encoding to determine the degree of match between the machine learning model and the target project.

5. The method according to claim 4, wherein encoding the user data and project results of the target dimension in the first data table to obtain a first code of at least one project scenario comprises: Hash encoding is performed on the user data and project results of the target dimension in the first data table to obtain a first hash value of at least one project scenario; The encoding of the user data and the predicted project results of the target dimension in the second data table to obtain a second code of at least one project scenario includes: Hash encoding is performed on the user data of the target dimension and the predicted project result in the second data table to obtain a second hash value of at least one project scenario; The comparing the first code and the second code to determine the matching degree between the machine learning model and the target project includes: Comparing the first hash value and the second hash value to determine the coverage of the at least one project scenario; Based on the coverage of the at least one project scenario, the matching degree between the machine learning model and the target project is determined.

6. The method according to claim 4, before encoding the user data and project results of the target dimension in the first data table to obtain a first code of at least one project scenario, further comprising: Preprocessing the user data and the project results in the first data table, wherein the preprocessing includes at least one of modifying the order between the target dimensions, modifying the naming of the target dimensions, modifying the mapping relationship between the user data, and modifying the mapping relationship between the user data and the project results; Before encoding the user data and the predicted project results of the target dimension in the second data table to obtain a second code of at least one project scenario, the method further includes: The user data and the predicted project results in the second data table are preprocessed, wherein the preprocessing includes at least one of modifying the order between the target dimensions, modifying the naming of the target dimensions, modifying the mapping relationship between the user data, and modifying the mapping relationship between the user data and the predicted project results.

7. A method for verifying a credit assessment model, comprising: Acquire a verification data set from a credit project, wherein the verification data set includes user attribute data and user behavior data of multiple dimensions in the credit project, and project results corresponding to the user attribute data and the user behavior data; Performing feature analysis on the user attribute data and the user behavior data, and extracting a target dimension from the multiple dimensions; Based on the target dimension, generating at least one project scenario for the credit project; Inputting the user data in the at least one project scenario into the credit assessment model applied to the credit project, respectively, to obtain the predicted project results corresponding to the user attribute data and the user behavior data in the at least one project scenario; Project scenario matching is performed based on the predicted project results and project results corresponding to the user attribute data and the user behavior data to determine the matching degree between the credit assessment model and the credit project.

8. A model verification device, comprising: A first acquisition module is configured to acquire a verification data set from a target project, wherein the verification data set includes user data of multiple dimensions in the target project and project results corresponding to the user data; A first extraction module is configured to perform feature analysis on the user data and extract a target dimension from the multiple dimensions; A first generating module is configured to generate at least one project scenario of the target project based on the target dimension; A first prediction module is configured to input the user data in the at least one project scenario into a machine learning model applied to the target project, respectively, to obtain a prediction project result corresponding to the user data in the at least one project scenario; The first matching module is configured to perform project scenario matching based on the predicted project results and project results corresponding to the user data, and determine the matching degree between the machine learning model and the target project.

9. The apparatus according to claim 8, wherein the first extraction module is further configured to: Counting the data features of the user data, and extracting a target dimension from the multiple dimensions based on the data features, wherein: The data feature includes at least one of a dispersion feature, a distribution feature and a correlation feature; and / or, A pre-trained feature analysis model is used to perform feature analysis on the user data, and a target dimension is extracted from the multiple dimensions, wherein the feature analysis model is trained based on sample user data of multiple sample dimensions, and the multiple sample dimensions are annotated with label dimensions.

10. The device according to claim 8, further comprising: The first screening module is configured to screen the target dimension by utilizing the domain knowledge of the target project.

11. The device according to any one of claims 8 to 10, wherein the user data and the project results corresponding to the user data are recorded in a first data table, and the user data in the at least one project scenario and the predicted project results are recorded in a second data table; The device also includes: A first encoding module is configured to encode the user data and project results of the target dimension in the first data table to obtain a first encoding of at least one project scenario; Encoding the user data and the predicted project results of the target dimension in the second data table to obtain a second code of at least one project scenario; The first matching module is further configured to: Compare the first encoding and the second encoding to determine the degree of match between the machine learning model and the target project.

12. The apparatus according to claim 11, wherein the first encoding module is further configured to: Hash coding the user data and project results of the target dimension in the first data table to obtain a first hash value of at least one project scenario; Hash coding the user data and predicted project results of the target dimension in the second data table to obtain a second hash value of at least one project scenario; The first matching module is further configured to: Compare the first hash value and the second hash value to determine the coverage of the at least one project scenario; based on the coverage of the at least one project scenario, determine the matching degree between the machine learning model and the target project.

13. The device according to claim 11, further comprising: The first preprocessing module is configured to preprocess the user data and the project results in the first data table, wherein the preprocessing includes at least one of modifying the order between the target dimensions, modifying the naming of the target dimensions, modifying the mapping relationship between the user data, and modifying the mapping relationship between the user data and the project results; and preprocess the user data and the predicted project results in the second data table, wherein the preprocessing includes at least one of modifying the order between the target dimensions, modifying the naming of the target dimensions, modifying the mapping relationship between the user data, and modifying the mapping relationship between the user data and the predicted project results.

14. A verification device for a credit assessment model, comprising: A second acquisition module is configured to acquire a verification data set from a credit project, wherein the verification data set includes user attribute data and user behavior data of multiple dimensions in the credit project, and project results corresponding to the user attribute data and the user behavior data; A second extraction module is configured to perform feature analysis on the user attribute data and the user behavior data, and extract a target dimension from the multiple dimensions; A second generating module is configured to generate at least one project scenario of the credit project based on the target dimension; A second prediction module is configured to input the user data in the at least one project scenario into the credit assessment model applied to the credit project, and obtain the prediction project result corresponding to the user attribute data and the user behavior data in the at least one project scenario; The second matching module is configured to perform project scenario matching based on the predicted project results and project results corresponding to the user attribute data and the user behavior data, and determine the matching degree between the credit assessment model and the credit project.

15. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 7 are implemented.

16. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

17. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images