Model verification, credit evaluation model verification method and device, and computing device

Abstract

The embodiment of the specification provides a model verification, a credit evaluation model verification method and device and a computing device, wherein the model verification method comprises: obtaining a verification data set from a target project, wherein the verification data set comprises user data of multiple dimensions in the target project and a project result corresponding to the user data; performing feature analysis on the user data to extract a target dimension from the multiple dimensions; generating at least one project scenario of the target project based on the target dimension; inputting the user data under the at least one project scenario into a machine learning model applied to the target project respectively to obtain a predicted project result corresponding to the user data under the at least one project scenario; and performing project scenario matching based on the predicted project result corresponding to the user data and the project result to determine the matching degree of the machine learning model and the target project, thereby improving the accuracy and efficiency of the model verification and reducing the cost of the model verification.

Description

Technical Field

[0001] The embodiments in this specification relate to the technical field of machine learning, and in particular to a method, apparatus, and computing device for model validation and credit assessment model validation. Background Technology

[0002] With the development of artificial intelligence and machine learning technologies, machine learning models have been widely applied in projects across various industries and application fields, significantly improving project efficiency and decision-making quality. For example, in the healthcare field, these models can assist in disease diagnosis, improving accuracy and speed. In the e-commerce industry, machine learning models can optimize inventory management and personalized recommendation systems, enhancing user experience and improving recommendation accuracy. In intelligent customer service, models can provide more natural and fluent human-computer dialogue, improving customer service experience. In content delivery, algorithms can deliver precise content based on user preferences, increasing user engagement. Finally, in the credit industry, machine learning models can be used for credit assessment, improving the speed and accuracy of credit evaluation.

[0003] Currently, frequent iterations and updates of machine learning models have become the norm in order to adapt to changing project environments and needs. However, each model iteration and update may introduce uncertainties and risks, requiring verification of the matching degree between the machine learning model and the actual project.

[0004] However, model validation through manually constructed data and subjective judgment suffers from limitations in human imagination. It cannot directly construct real-world validation scenarios from massive datasets to verify the effectiveness of machine learning models. This often results in validation results that do not reflect real-world conditions, leading to insufficient accuracy in model validation. Furthermore, the manual construction process significantly impacts the efficiency of model testing and increases validation costs. Therefore, a highly accurate, efficient, and low-cost model validation method is needed. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a model verification method. One or more embodiments of this specification also relate to a method for verifying a credit assessment model, a model verification device, a credit assessment model verification device, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, a model verification method is provided, comprising:

[0007] Obtain the validation dataset from the target project. The validation dataset includes user data from multiple dimensions in the target project and the corresponding project results for the user data.

[0008] Perform feature analysis on user data to extract target dimensions from multiple dimensions;

[0009] Based on the target dimension, generate at least one project scenario for the target project;

[0010] Input user data from at least one project scenario into a machine learning model applied to the target project to obtain predicted project results for the user data in at least one project scenario.

[0011] Based on the predicted project results and project results corresponding to user data, the project scenario is matched to determine the matching degree between the machine learning model and the target project.

[0012] According to a second aspect of the embodiments of this specification, a method for verifying a credit assessment model is provided, comprising:

[0013] Obtain the verification dataset from the credit project. The verification dataset includes user attribute data and user behavior data from multiple dimensions of the credit project, as well as the project results corresponding to the user attribute data and user behavior data.

[0014] Perform feature analysis on user attribute data and user behavior data to extract target dimensions from multiple dimensions;

[0015] Based on the target dimension, generate at least one project scenario for the credit project;

[0016] Input user data from at least one project scenario into a credit assessment model applied to a credit project to obtain predicted project results corresponding to user attribute data and user behavior data for at least one project scenario.

[0017] Based on the predicted project results and project results corresponding to user attribute data and user behavior data, the matching degree between the credit assessment model and the credit project is determined by matching the project scenario.

[0018] According to a third aspect of the embodiments of this specification, a model verification apparatus is provided, comprising:

[0019] The first acquisition module is configured to acquire a validation dataset from the target project, wherein the validation dataset includes user data from multiple dimensions in the target project and the project results corresponding to the user data;

[0020] The first extraction module is configured to perform feature analysis on user data and extract the target dimension from multiple dimensions.

[0021] The first generation module is configured to generate at least one project scenario of the target project based on the target dimension;

[0022] The first prediction module is configured to input user data from at least one project scenario into a machine learning model applied to the target project, and obtain prediction project results corresponding to the user data in at least one project scenario.

[0023] The first matching module is configured to perform project scenario matching based on the predicted project results and project results corresponding to user data, and determine the matching degree between the machine learning model and the target project.

[0024] According to a fourth aspect of the embodiments of this specification, a verification device for a credit assessment model is provided, comprising:

[0025] The second acquisition module is configured to acquire a verification dataset from the credit project. The verification dataset includes user attribute data and user behavior data from multiple dimensions of the credit project, as well as the project results corresponding to the user attribute data and user behavior data.

[0026] The second extraction module is configured to perform feature analysis on user attribute data and user behavior data, and extract the target dimension from multiple dimensions.

[0027] The second generation module is configured to generate at least one project scenario for credit projects based on the target dimension.

[0028] The second prediction module is configured to input user data from at least one project scenario into a credit assessment model applied to a credit project, and obtain prediction project results corresponding to user attribute data and user behavior data in at least one project scenario.

[0029] The second matching module is configured to perform project scenario matching based on the predicted project results and project results corresponding to user attribute data and user behavior data, and determine the matching degree between the credit assessment model and the credit project.

[0030] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0031] Memory and processor;

[0032] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above method.

[0033] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the above-described method.

[0034] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, wherein when the computer program product is executed in a computer, the computer performs the steps of the above-described method.

[0035] In one embodiment of this specification, a validation dataset from a target project is obtained. This dataset includes user data from multiple dimensions within the target project and the corresponding project results. Obtaining authentic user data and project results from a real project avoids the limitations of manually constructed data, improves data quality and authenticity, and makes the validation results more consistent with the actual target project. Feature analysis is performed on the user data to extract target dimensions that have a significant impact on the project results. Based on these target dimensions, at least one project scenario for the target project is generated, ensuring that the generated project scenarios comprehensively cover real-world project scenarios under different conditions, targeting the most influential target dimensions. User data from at least one project scenario is input into a machine learning model applied to the target project to obtain predicted project results for each user data in at least one project scenario. Project scenario matching is performed based on the predicted project results and the actual project results to determine the matching degree between the machine learning model and the target project. This comprehensive automated validation of the machine learning model is achieved within the generated project scenarios that conform to various real-world conditions, improving the accuracy and efficiency of model validation and reducing its cost. Attached Figure Description

[0036] Figure 1 This is a flowchart of a model verification method provided in one embodiment of this specification;

[0037] Figure 2 This is a flowchart of a verification method for a credit assessment model provided in one embodiment of this specification;

[0038] Figure 3 This is a flowchart illustrating the processing procedure of a verification method for a credit assessment model applied to user rating, provided in one embodiment of this specification.

[0039] Figure 4 This is a schematic diagram of the structure of a model verification device provided in one embodiment of this specification;

[0040] Figure 5 This is a schematic diagram of the structure of a verification device for a credit assessment model provided in one embodiment of this specification;

[0041] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0042] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0043] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0044] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0045] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0046] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0047] Model validation: The process of evaluating the ability of a machine learning model to accurately predict or classify in a specific application scenario.

[0048] Metrics calculation: The process of evaluating model performance by quantifying metrics such as accuracy and recall.

[0049] Scene modeling: Building models based on specific conditions or environments to simulate real-world events in order to test and validate the performance of machine learning models.

[0050] Feature analysis: The process of identifying variables or attributes from a dataset that influence model predictions.

[0051] Decision tree algorithm: A supervised learning method that recursively partitions a dataset into subsets, forming a tree structure. Each internal node represents a test on a feature, each branch represents a test result, and each leaf node represents a class or output value. Decision trees classify or regress data using a series of rules.

[0052] Random field algorithms: a class of probabilistic graphical models used to describe and model complex data distributions and dependencies. Common random field algorithms include Markov Random Fields (MRFs) and Conditional Random Fields (CRFs). These models represent the dependencies between variables through probability distributions.

[0053] Clustering algorithms: An unsupervised learning method used to group objects in a dataset into multiple clusters, such that objects within the same cluster have high similarity, while objects in different clusters have low similarity. Common clustering algorithms include K-means clustering, hierarchical clustering, and DBSCAN (Density-Based Spatial Clustering of Applications with Noise).

[0054] Genetic Algorithm: A global optimization search algorithm based on natural selection and genetic mechanisms. It gradually optimizes the population of solutions by simulating selection, crossover, and mutation operations in biological evolution to find the optimal solution to a problem. Genetic algorithms are commonly used in optimization problems, scheduling problems, and hyperparameter tuning in machine learning.

[0055] Bayesian algorithm: A probabilistic and statistical method based on Bayes' theorem, used to infer the probability distribution of unknown parameters given observed data. The Bayesian algorithm updates the posterior probability using prior probabilities and a likelihood function, and is commonly used in classification, regression, prediction, and decision analysis. Common Bayesian models include the Naive Bayes classifier and Bayesian networks.

[0056] Wide tables: Data tables with a large number of columns. These columns can come from different sources and may include various aggregated and summarized data. Wide tables are used to store lengthy data to support complex queries such as analysis and mining.

[0057] Narrow tables: Data tables containing only a few columns (such as a primary key and a few attributes). This design concept is used to solve data storage and access problems in single-project scenarios, thereby improving query efficiency and reducing storage space.

[0058] Consistent coding: A coding method used to ensure that data from different sources or environments can be accurately compared, ensuring that the same data structure corresponds to the same code.

[0059] Hash encoding: a method for converting data into a fixed-length string, typically used for fast lookups and data integrity verification. In machine learning, hash encoding can be used to transform high-dimensional sparse features into low-dimensional dense features, reducing storage space and improving computational efficiency.

[0060] MD5 (Message-Digest Algorithm 5): A widely used hash function that can accept input of arbitrary length and produce a 128-bit (16-byte) hash value. MD5 is often used for data integrity verification.

[0061] A / B testing is a statistical experimental method that involves randomly assigning users to two or more groups (Group A and Group B), showing them different versions of a website (such as a website page or application interface), and then collecting and analyzing user behavior data to evaluate which version performs better. In the validation of machine learning models, A / B testing can be used to compare the performance of different models or model versions in real-world projects, thereby selecting the superior model.

[0062] Currently, the model validation method, represented by A / B testing, which involves manually constructing data and making subjective judgments, is not only resource-intensive and time-consuming in the manual construction process, but also limited by human imagination. It cannot directly construct validation scenarios adapted to the real world from massive amounts of data to verify the effectiveness of machine learning models. As a result, the validation results often do not conform to the real world situation, and the accuracy of model validation is insufficient.

[0063] To address the aforementioned issues, this specification provides a model verification method. This specification also relates to a credit assessment model verification method, a model verification device, a credit assessment model verification device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0064] See Figure 1 , Figure 1 A flowchart of a model validation method according to an embodiment of this specification is shown, including the following specific steps:

[0065] Step 102: Obtain the validation dataset from the target project. The validation dataset includes user data from multiple dimensions in the target project and the corresponding project results for the user data.

[0066] The embodiments in this specification apply to data processing applications, platforms, or systems that have model validation capabilities.

[0067] The target project refers to the specific application or project domain in which the machine learning model is applied. The target domain includes massive amounts of user data and the corresponding project results. Examples include the healthcare field, e-commerce industry, intelligent customer service field, content delivery field, and credit field. It should be noted that the specific applications or project domains exemplified in the embodiments of this specification do not limit the target project or the data of the target project; they are only for illustrative purposes, as it is impossible to exhaustively list all applications and project domains.

[0068] A validation dataset is a foundational dataset used to verify the fit between a machine learning model and a target project. It is a collection of user data across multiple dimensions and the corresponding project results, obtained from the target project. Examples include: a validation dataset in the healthcare field consisting of patient data and diagnostic results; a validation dataset in the e-commerce industry consisting of user data and purchase results; a validation dataset in the field of intelligent customer service consisting of user queries and responses; a validation dataset in the content delivery field consisting of user content browsing history and click behavior; and a validation dataset in the credit field consisting of user data and credit assessment results.

[0069] Dimensions are the attribute dimensions to which user data belongs, with each dimension representing an attribute of a specific type of user data. For example, in the healthcare field, multiple dimensions may include a patient's medical history, examination results, family medical history, etc. In the e-commerce industry, multiple dimensions may include a user's browsing history, purchase history, search keywords, user reviews, etc. In intelligent customer service, multiple dimensions may include a user's query content, emotional state, service history, user feedback, etc. In content delivery, multiple dimensions may include a user's interests and preferences, browsing history, content exposure frequency, etc. In credit, multiple dimensions may include a user's education level, occupation, credit history, etc.

[0070] User data refers to information related to users within a target project. It serves as input for validating machine learning models and includes at least one of the following: static user attribute data and dynamic user behavior data. For example, in the healthcare field, user data may include patient medical history, examination results, and family medical history. In the e-commerce industry, user data may include browsing history, purchase history, search keywords, and user reviews. In intelligent customer service, user data may include user query content, emotional state, service history, and user feedback. In content delivery, user data may include user interests, browsing history, and content exposure frequency. In the credit field, user data may include user education level, occupation, and credit history.

[0071] The project results are the output results corresponding to the user data in the target project. These results represent the actual output of the target project and are used to validate the label output of the machine learning model. The results are compared with the model's predicted project results to validate the machine learning model. Project results can be binary (e.g., whether the user is ill, whether content was clicked, whether the information level is met) or multi-category (e.g., disease type, content category, credit rating). For example, in the healthcare field, project results could be whether the user is ill and the type of disease. In the e-commerce industry, project results could be whether a product was purchased. In the field of intelligent customer service, project results could be the response result. In the field of content delivery, project results could be whether a click was made and the content category. In the credit field, project results could be whether the information level and credit rating are met.

[0072] One possible way to obtain a validation dataset from the target project is to use big data processing technology to extract user data and corresponding project results from multiple dimensions of the target project, and then construct a validation dataset based on the user data and corresponding project results.

[0073] For example, before deploying the new version of the product recommendation model on an e-commerce platform to replace the old version, the effectiveness of the new version of the product recommendation model needs to be verified.

[0074] Using big data processing tools such as Hadoop or Spark, extract user data from the e-commerce platform's log database, transaction record database, and user database, covering 200 dimensions (registration time, geographical location, occupation, education level, browsing history, purchase history, search keywords, user reviews, items in shopping cart, user feedback, user interaction behavior, user device information, etc.) and four corresponding project results:

[0075] User data includes: registration time data, geolocation data, occupation data, education level data, browsing history data, purchase history data, search keyword data, user review data, items in shopping cart data, user feedback data, user interaction behavior data, user device information data, etc.

[0076] Project Results: Whether the user clicked on the recommended product (clicked / not clicked); whether the user purchased the recommended product (purchased / not purchased); user rating of the recommended product (positive / neutral / negative); user satisfaction rating of the recommendation system (e.g., 1-5 points).

[0077] Based on the user data from the above 200 dimensions and the results of the four types of projects, a validation dataset was constructed.

[0078] Obtain a validation dataset from the target project. The validation dataset includes user data from multiple dimensions of the target project and the corresponding project results. Obtaining real user data and corresponding project results from the actual project avoids the limitations of manually constructed data, improves the quality and authenticity of the data, and makes the validation results more consistent with the real target project.

[0079] Step 104: Perform feature analysis on user data and extract the target dimension from multiple dimensions.

[0080] The target dimension is a dimension extracted from multiple dimensions that significantly impacts model decisions and is a key dimension that needs to be focused on during model validation. For example, in the healthcare field, the target dimension might include a patient's medical history and examination results. In the e-commerce industry, it might include a user's browsing history and purchase history. In intelligent customer service, it might include a user's query content, emotional state, service history, and user feedback. In content delivery, it might include interests, browsing history, and content exposure frequency. In credit, it might include a user's occupation and credit history.

[0081] For example, statistical algorithms and machine learning algorithms are used to perform feature analysis on user data (registration time data, geographical location data, occupation data, education level data, browsing history data, purchase history data, search keyword data, user review data, shopping cart item data, user feedback data, user interaction behavior data, user device information data, etc.), and 20 target dimensions are extracted from 200 dimensions: geographical location, occupation, browsing history, purchase history, search keywords, etc.

[0082] By performing feature analysis on user data, we can extract target dimensions that have a significant impact on project results from multiple dimensions, thereby improving the scientific nature and relevance of data dimension selection and providing data dimension support for the subsequent generation of real project scenarios that comprehensively cover different conditions.

[0083] Step 106: Based on the target dimension, generate at least one project scenario for the target project.

[0084] The project scenarios for the target project are real-world scenarios used to simulate different conditions of the target project. Generating at least one project scenario ensures that the model can cover real-world situations under different conditions during validation, thereby comprehensively evaluating the performance of the machine learning model. For example, in the healthcare field, generated project scenarios include: Scenario 1: A young male patient with a history of hypertension; recent test results show elevated blood pressure; is he suffering from cardiovascular disease? Scenario 2: A middle-aged female patient with no obvious medical history; recently experiencing headaches; is she suffering from a brain disease? Another example is in the e-commerce industry, where generated project scenarios include: Scenario 1: A young female user who frequently browses clothing; recently searched for "dress" and purchased apparel; Scenario 2: A middle-aged male user whose purchase history is mainly focused on electronic products; recently searched for "laptop" and purchased electronic consumer goods. Yet another example is in the field of intelligent customer service, where generated project scenarios include: Scenario 1: A user inquires about resetting their password, exhibiting anxiety; Scenario 2: A user expresses dissatisfaction with a previous service and requests a re-processing. For example, in the content delivery field, generated project scenarios include: Scenario 1: A young user with a travel interest who recently browsed multiple travel articles; Scenario 2: A middle-aged user with a technology interest who recently browsed multiple technology news articles. Similarly, in the credit field, generated project scenarios include: Scenario 1: A young user with no credit history applying for a small credit program; Scenario 2: A middle-aged user with a good credit history applying for a large credit program.

[0085] Based on the target dimension, at least one project scenario of the target project is generated. One possible approach is to use a decision tree algorithm to generate at least one project scenario of the target project based on the target dimension. Specifically, the target dimension is used as a feature and the project result is used as a label to train a decision tree model. Rules are extracted from the decision tree, and each leaf node represents a project scenario. Based on the extracted rules, a specific project scenario is generated.

[0086] Based on the target dimension, at least one project scenario of the target project can be generated. Another option is to use a random field algorithm to generate at least one project scenario of the target project based on the target dimension. Specifically, the target dimension is used as a feature and the project result is used as a label to train a random field model (such as a conditional random field CRF). The model is then used to generate project results under different combinations of dimensions to form project scenarios.

[0087] Based on the target dimension, at least one project scenario of the target project can be generated. Alternatively, a clustering algorithm can be used to generate at least one project scenario of the target project based on the target dimension. Specifically, the target dimension is used as a feature to train a clustering model (such as K-means, DBSCAN, etc.), and each cluster center and its nearby points can represent a project scenario.

[0088] Based on the target dimension, at least one project scenario for the target project is generated. Alternatively, a genetic algorithm can be used to generate at least one project scenario for the target project based on the target dimension. Specifically, a set of initial project scenarios is randomly generated. A fitness function is defined to evaluate the rationality of each project scenario. Through operations such as selection, crossover, and mutation, the project scenarios are gradually optimized, ultimately yielding a set of optimized project scenarios.

[0089] Based on the target dimension, at least one project scenario of the target project can be generated. Alternatively, a Bayesian algorithm can be used to generate at least one project scenario of the target project based on the target dimension. Specifically, the probabilistic dependency between the target dimension and the project outcome is defined, and a Bayesian network is used to generate different samples, each representing a project scenario.

[0090] For example, using a genetic algorithm, based on 20 target dimensions (geographic location, occupation, browsing history, purchase history, search keywords, etc.), five scenarios for product recommendations on an e-commerce platform are generated:

[0091] Project Scenario 1: Young female users who frequently browse fashion products, have recent purchase records, and search for fashion-related keywords.

[0092] Project Scenario 2: Middle-aged male user, frequent browser of electronic products, purchase history concentrated on high-priced items, search keywords related to electronic products.

[0093] Project Scenario 3: Elderly users who frequently browse health products and daily necessities, have low purchase frequency, and search for health-related keywords.

[0094] Project Scenario 4: Teen users who frequently browse anime merchandise and game products, have a frequent purchase history, and search for entertainment-related keywords.

[0095] Project Scenario 5: Middle-aged female users who frequently browse home furnishings and children's products, whose purchase history is concentrated on household products, and whose search keywords are related to family life.

[0096] Based on the target dimension, at least one project scenario for the target project is generated, ensuring that the generated project scenarios can comprehensively cover real project scenarios under different conditions for the target dimension with great impact, and providing scenario support for the subsequent prediction and matching process.

[0097] Step 108: Input user data from at least one project scenario into the machine learning model applied to the target project to obtain the predicted project results corresponding to the user data in at least one project scenario.

[0098] A machine learning model is a machine learning model specifically designed for a target project. This model is pre-trained to learn patterns and regularities from the historical data of the target project, enabling it to predict or classify new data to adapt to the project environment and needs. Machine learning models include, but are not limited to: linear regression models, logistic regression models, decision tree models, random forest models, clustering models, support vector machine models, neural network models, large language models, and generative Transformer models.

[0099] The predicted project outcome is the project output corresponding to the user data generated by the machine learning model. It represents the simulated output under at least one generated project scenario. This predicted output is used to validate the machine learning model's predictions by comparing them with the actual project outcome, thus verifying the model's performance. The predicted project outcome can be binary (e.g., whether the user is ill, whether content was clicked, or whether the information level is met) or multi-category (e.g., disease type, content category, credit rating). For example, in the healthcare field, the predicted project outcome could be whether the user is ill and the type of disease. In the e-commerce industry, it could be whether a product was purchased. In intelligent customer service, it could be the response result. In content delivery, it could be whether a click was made and the content category. In the credit field, it could be whether the information level and credit rating are met.

[0100] One possible approach is to input user data from at least one project scenario into a machine learning model applied to the target project to obtain predicted project results for the user data in at least one project scenario.

[0101] For example, according to five project scenarios (project scenario 1, project scenario 2, project scenario 3, project scenario 4, and project scenario 5), user data is input into a retrieval and recall model applied to product recommendation, and four predicted project results are obtained for the user data under the five project scenarios: whether the user clicked on the recommended product (clicked / not clicked); whether the user purchased the recommended product (purchased / not purchased); the user's evaluation of the recommended product (positive / neutral / negative); and the user's satisfaction rating of the recommendation system (e.g., 1-5 points).

[0102] By inputting user data from at least one project scenario into a machine learning model applied to the target project, the predicted project results corresponding to the user data in at least one project scenario are obtained. This ensures the comprehensiveness and accuracy of the predicted project results and improves the effectiveness of subsequent model testing.

[0103] Step 110: Based on the predicted project results and project results corresponding to the user data, perform project scenario matching to determine the matching degree between the machine learning model and the target project.

[0104] The fit between a machine learning model and a target project is a quantitative indicator of the applicability and performance of the machine learning model in the target project. The fit assesses the effectiveness and risk of the machine learning model in the target project and can also be used to guide future verification processes and further optimize the verification workflow. This includes, but is not limited to, at least one of the following: coding coverage, accuracy, and recall.

[0105] Based on the predicted project results and project results corresponding to user data, project scenario matching is performed to determine the matching degree between the machine learning model and the target project. One possible approach is to compare the predicted project results and project results corresponding to user data in at least one project scenario to determine the matching degree between the machine learning model and the target project.

[0106] For example, by comparing the four predicted project results and four project results (whether the user clicked the recommended product (click / not clicked); whether the user purchased the recommended product (purchase / not purchased); the user's evaluation of the recommended product (positive / neutral / negative); and the user's satisfaction rating of the recommendation system (e.g., 1-5 points)) in five project scenarios, the matching degree between the retrieval recall model and the product recommendation is determined to be 65%.

[0107] In this embodiment, a validation dataset from the target project is obtained. This dataset includes user data from multiple dimensions within the target project and the corresponding project results. Obtaining real user data and project results from actual projects avoids the limitations of manually constructed data, improves data quality and authenticity, and makes the validation results more consistent with the real target project. Feature analysis is performed on the user data to extract target dimensions that have a significant impact on the project results. Based on these target dimensions, at least one project scenario for the target project is generated, ensuring that the generated scenario comprehensively covers real project scenarios under different conditions, targeting the most influential dimensions. User data from at least one project scenario is input into a machine learning model applied to the target project to obtain predicted project results for each user data in at least one project scenario. Project scenario matching is performed based on the predicted project results and the actual project results to determine the matching degree between the machine learning model and the target project. This comprehensive automated validation of the machine learning model is achieved within the generated project scenarios that conform to various real-world conditions, improving the accuracy and efficiency of model validation and reducing its cost.

[0108] In one optional embodiment of this specification, step 104 includes the following specific steps:

[0109] The data features of user data are statistically analyzed, and a target dimension is extracted from multiple dimensions based on the data features, wherein the data features include at least one of discrete features, distribution features, and correlation features; and / or, a pre-trained feature analysis model is used to perform feature analysis on user data and extract the target dimension from multiple dimensions, wherein the feature analysis model is trained based on sample user data of multiple sample dimensions, and the multiple sample dimensions are labeled with a dimension.

[0110] The data features of user data are the statistical and attribute features that describe the user data. The data features include at least one of the following: dispersion features, distribution features, and correlation features. These features are used to analyze the distribution, dispersion, and correlation between dimensions of user data, and then extract the dimensions that have a significant impact on model decision-making.

[0111] Dispersion features describe the degree of dispersion of data and can include variance, standard deviation, etc. For example, in the e-commerce industry, the standard deviation of the quantity of goods purchased can reflect the fluctuation in the quantity of goods purchased. If the standard deviation is large, it indicates that the quantity of goods purchased fluctuates greatly; if the standard deviation is small, it indicates that the quantity of goods purchased is relatively stable.

[0112] Distribution characteristics describe the distribution of data and can include the mean, median, mode, skewness, and kurtosis. For example, in the healthcare field, the mean and median of patient medical history data can reflect the central tendency of the patient's medical history. If the mean and median are close, it indicates that the patient's medical history is relatively evenly distributed; if the mean and median differ significantly, it indicates that the distribution of the patient's medical history may be skewed.

[0113] Relevance features describe the correlation between different dimensions and can include correlation coefficients, mutual information, etc. For example, in the field of intelligent customer service, the correlation coefficient between a user's query content and their emotional state can reflect the relationship between the two. If the correlation coefficient is high, it indicates that the user's query content is strongly correlated with their emotional state.

[0114] Feature analysis models are machine learning models used to automatically extract key features from user data. These models are trained on sample user data across multiple dimensions. Because these dimensions include labeled dimensions, the model learns which dimensions significantly influence its predictions, thus enabling the extraction of target dimensions from complex, massive datasets. Feature analysis models include, but are not limited to: decision tree models, random forest models, support vector machine models, and neural network models.

[0115] For example, the dispersion characteristics (variance, standard deviation), distribution characteristics (mean, median, mode, skewness, kurtosis), and correlation characteristics (correlation coefficient, mutual information) of user data (registration time data, geographical location data, occupation data, education level data, browsing history data, purchase history data, search keyword data, user review data, shopping cart item data, user feedback data, user interaction behavior data, user device information data, etc.) are statistically analyzed. Based on these dispersion characteristics, distribution characteristics, and correlation characteristics, 14 target dimensions are extracted from 200 dimensions.

[0116] For example, a pre-trained support vector machine model is used to perform feature analysis on user data (registration time data, geographical location data, occupation data, education level data, browsing history data, purchase history data, search keyword data, user review data, shopping cart item data, user feedback data, user interaction behavior data, user device information data, etc.), extracting 18 target dimensions from 200 dimensions.

[0117] In the embodiments of this specification, statistical algorithms and / or machine learning algorithms are used to complete feature analysis, accurately identify target dimensions that have an impact on model prediction, and combine the inherent characteristics and logic of user data to improve the scientific nature and relevance of data dimension selection, providing more accurate data dimension support for the subsequent generation of real project scenarios that fully cover different conditions.

[0118] In one optional embodiment of this specification, the following specific steps are included before step 108:

[0119] Utilize domain knowledge of the target project to filter target dimensions.

[0120] Domain knowledge for a target project refers to the professional knowledge, experience, and data accumulated within the target project's domain. This knowledge helps to more accurately understand and process the dimensions to which user data belongs. Domain knowledge includes, but is not limited to, knowledge graphs, knowledge bases, and expert domain knowledge. A knowledge graph is a structured representation of knowledge, describing domain knowledge through entities, attributes, and relationships. For example, in the healthcare field, a knowledge graph can include entities such as diseases, symptoms, drugs, and treatments, as well as the relationships between them. A knowledge base is a system for storing and managing domain-specific expertise, typically containing large amounts of structured and semi-structured data. For example, in the e-commerce industry, a knowledge base can include information such as product categories, user behavior patterns, and market trends. Expert domain knowledge refers to the experience, insights, and suggestions provided by experts or professionals in the domain. This knowledge can be acquired through interviews, questionnaires, literature reviews, etc. For example, in the field of intelligent customer service, experts can provide knowledge about common user questions and response strategies.

[0121] For example, by utilizing expert domain knowledge in product recommendation, 14+18 target dimensions are filtered to obtain 20 target dimensions.

[0122] In the embodiments described in this specification, based on the dimension extraction, the target dimensions are further filtered and optimized by combining the domain knowledge of the target project. This ensures that the selected target dimensions are both representative and in line with the project environment and project requirements of the target project, providing more accurate data dimension support for the subsequent generation of real project scenarios that fully cover different conditions.

[0123] In one optional embodiment of this specification, user data and the corresponding project results are recorded in a first data table, and user data and predicted project results for at least one project scenario are recorded in a second data table.

[0124] Correspondingly, before step 110, the following specific steps are also included:

[0125] Encode the user data and project results of the target dimension in the first data table to obtain a first code for at least one project scenario; encode the user data and predicted project results of the target dimension in the second data table to obtain a second code for at least one project scenario.

[0126] Correspondingly, step 110 includes the following specific steps:

[0127] By comparing the first and second codes, the matching degree between the machine learning model and the target project is determined.

[0128] The first data table contains structured data recording user data and corresponding project results within the target project. This first data table can be a wide table to record user data across multiple dimensions and their corresponding project results, suitable for batch processing and statistical analysis. Alternatively, it can consist of multiple narrow tables to record detailed data for each user across different dimensions, suitable for fine-grained analysis and dynamic evaluation; neither is limited here.

[0129] The second data table contains structured data recording user data and predicted project results for at least one project scenario. This second data table can be a wide table recording user data for the target dimension and the corresponding predicted project results, suitable for batch processing and statistical analysis; or it can be multiple narrow tables recording detailed data for each user across different target dimensions, suitable for fine-grained analysis and dynamic evaluation. No limitation is imposed here.

[0130] The first encoding is the result obtained by encoding the user data and project results of the target dimension in the first data table. The purpose of encoding is to convert the raw data into a uniform format to facilitate comparison with the data during the verification process.

[0131] The second encoding is the result of encoding the user data and predicted project results of the target dimension in the second data table. The purpose of encoding is to convert the predicted data into a unified format, making it easier to compare with the actual data.

[0132] It should be noted that the first and second codes are unique identifiers generated for each project scenario using a consistency coding algorithm. The consistency coding methods include, but are not limited to, hash coding and Base64 coding.

[0133] For example, consistent encoding is performed on the user data (geographic location data, occupation data, browsing history data, purchase history data, search keyword data, etc.) and four project outcomes (whether the user clicked on the recommended product (click / not clicked); whether the user purchased the recommended product (purchase / not purchased); the user's evaluation of the recommended product (positive / neutral / negative); the user's satisfaction rating of the recommendation system (e.g., 1-5 points)) in the first data table to obtain the first code for five project scenarios: a11c240cc... Consistent encoding is performed on the user data (geographic location data, occupation data, browsing history data, purchase history data, search keyword data, etc.) and four predicted project outcomes (whether the user clicked on the recommended product (click / not clicked); whether the user purchased the recommended product (purchase / not purchased); the user's evaluation of the recommended product (positive / neutral / negative); the user's satisfaction rating of the recommendation system (e.g., 1-5 points)) in the second data table to obtain the second code for five project scenarios: a11c2f3cc... By comparing the first and second codes, the coding coverage of the machine learning model and the target item was determined to be 3%.

[0134] In the embodiments of this specification, the data table encoding method ensures direct comparability between different project scenarios, improves the comprehensiveness and accuracy of model verification, reduces the false judgment rate in the verification process, and improves the reliability and efficiency of model verification.

[0135] In one optional embodiment of this specification, the user data and project results of the target dimension in the first data table are encoded to obtain a first code for at least one project scenario, including the following specific steps:

[0136] Hash-encode the user data and project results of the target dimension in the first data table to obtain the first hash value of at least one project scenario;

[0137] Encode the user data and predicted project results for the target dimension in the second data table to obtain a second code for at least one project scenario, including the following specific steps:

[0138] Hash-encode the user data and predicted project results of the target dimension in the second data table to obtain a second hash value for at least one project scenario;

[0139] Comparing the first and second codes to determine the matching degree between the machine learning model and the target project includes the following specific steps:

[0140] By comparing the first hash value and the second hash value, the coverage of at least one project scenario is determined;

[0141] Based on the coverage of at least one project scenario, determine the matching degree between the machine learning model and the target project.

[0142] The first hash value is a unique identifier obtained by hashing the user data and project results of the target dimension in the first data table. Hash encoding converts the original data into a fixed-length string, ensuring that the encoded identifier for each project scenario is unique. Methods for obtaining the first hash value include, but are not limited to, using the MD5 hash function and the SHA-256 hash function. The purpose of encoding is to convert the original data into a unified hash value format, facilitating comparison with data during the verification process.

[0143] The second hash value is a unique identifier obtained by hashing and encoding the user data and predicted project results for the target dimension in the second data table. Hash encoding converts the predicted data into a fixed-length string, ensuring that the encoded identifier for each project scenario is unique. Methods for obtaining the second hash value include, but are not limited to, using the MD5 hash function and the SHA-256 hash function. The purpose of encoding is to convert the predicted data into a uniform hash value, facilitating comparison with the actual data.

[0144] Coverage of at least one project scenario is the proportion of consistency between the model's predictions and the actual project results, determined after comparing the first and second hash values. Coverage reflects the accuracy and reliability of the model in a specific project scenario. Coverage is used to evaluate the matching degree between the machine learning model and the target project; the higher the coverage, the closer the model's predictions are to the actual results, and the better the model's performance.

[0145] For example, MD5 hashing is performed on the user data (geographic location data, occupation data, browsing history data, purchase history data, search keyword data, etc.) for 20 target dimensions in the first data table and four project outcomes (whether the user clicked on the recommended product, whether the user purchased the recommended product, the user's evaluation of the recommended product, and the user's satisfaction rating with the recommendation system) to generate a first hash value: a11c240cc. MD5 hashing is also performed on the user data (geographic location data, occupation data, browsing history data, purchase history data, search keyword data, etc.) for 20 target dimensions in the second data table and four predicted project outcomes (whether the user clicked on the recommended product, whether the user purchased the recommended product, the user's evaluation of the recommended product, and the user's satisfaction rating with the recommendation system) to generate a second hash value: a11c2f3cc. Comparing the first and second hash values ​​for five project scenarios, it is found that the hash value for one project scenario is completely identical, while the hash values ​​for the other four project scenarios are inconsistent. Therefore, the coverage rate of the five project scenarios is 1 / 5 = 20%. Based on the coverage of 5 project scenarios, the matching degree between the machine learning model and the target project was determined to be 0.2.

[0146] In the embodiments of this specification, the uniqueness and comparability of the encoding between different project scenarios are ensured by hashing the data table, which further improves the comprehensiveness and accuracy of model verification, further reduces the false judgment rate in the verification process, and further improves the reliability and efficiency of model verification.

[0147] In one optional embodiment of this specification, before encoding the user data and project results of the target dimension in the first data table to obtain the first encoding of at least one project scenario, the following specific steps are further included:

[0148] Preprocessing is performed on the user data and project results in the first data table. The preprocessing includes at least one of the following: modifying the order between target dimensions, modifying the naming of target dimensions, modifying the mapping relationship between user data, and modifying the mapping relationship between user data and project results.

[0149] Correspondingly, before encoding the user data and predicted project results of the target dimension in the second data table to obtain the second encoding of at least one project scenario, the following specific steps are also included:

[0150] The user data and prediction results in the second data table are preprocessed, wherein the preprocessing includes at least one of the following: modifying the order between target dimensions, modifying the naming of target dimensions, modifying the mapping relationship between user data, and modifying the mapping relationship between user data and prediction results.

[0151] Modifying the order of target dimensions involves adjusting the arrangement of the target dimensions in the data table to better meet the requirements of the model or improve data readability. Changing the order of target dimensions ensures that the data is more organized during processing, facilitating model understanding and handling. The order of target dimensions can be rearranged using programming or data processing tools. For example, in the healthcare field, the original order of the target dimensions in the first data table might be: occupation, medical history, test results, location, family medical history. The order could be changed to: location, occupation, medical history, test results, family medical history.

[0152] Renaming target dimensions involves changing the names of each target dimension in the data table to better reflect its meaning or improve data readability. Modifying the naming of target dimensions ensures that the dimension names in the data table are more intuitive and easier to understand, facilitating subsequent data processing and analysis. Target dimensions can be renamed using programming or data processing tools. For example, in the healthcare field, if the original name of the target dimension in the first data table is "Job," it can be renamed to "Occupation."

[0153] Modifying the mapping relationships between user data involves adjusting the table relationships between user data in the data table to better reflect the actual relationships between the data or improve data processing efficiency. Modifying the mapping relationships ensures that the relationships between data are more reasonable, facilitating model understanding and processing. The mapping relationships between user data can be redefined through programming or data processing tools. For example, in the field of intelligent customer service, there is a certain mapping relationship between user queries and emotional states in the first data table; the query "How to reset password" can be mapped to the emotional state "anxious." To better reflect the user's actual emotions, the mapping relationship can be adjusted so that "How to reset password" is mapped to the emotional state "urgent."

[0154] Modifying the mapping between user data and project results involves adjusting the table association between user data and project results to better reflect the actual relationships between the data or improve data processing efficiency. Modifying the mapping between user data and project results ensures a more reasonable relationship between the data, facilitating model understanding and processing. The mapping between user data can be redefined through programming or data processing tools. For example, in the e-commerce industry, there is a certain mapping relationship between a user's browsing history and purchase history in the first data table. Browsing history "multiple views of a certain mobile phone" can be mapped to purchase history "purchased the same mobile phone". To better reflect the user's actual purchasing behavior, the mapping can be adjusted so that "multiple views of a certain mobile phone" is mapped to purchase history "added to cart but not purchased".

[0155] In the embodiments of this specification, by preprocessing the user data and project results in the first data table and the second data table, data inconsistency and redundancy can be effectively eliminated, thereby improving data readability and processing efficiency, ensuring that data features can be captured more accurately during the model verification process, and improving the reliability of model verification.

[0156] Taking the target project as an information project as an example, Figure 2 A flowchart illustrating a verification method for a credit assessment model according to an embodiment of this specification is shown, including the following specific steps:

[0157] Step 202: Obtain the verification dataset from the credit project. The verification dataset includes user attribute data and user behavior data from multiple dimensions of the credit project, as well as the project results corresponding to the user attribute data and user behavior data.

[0158] The embodiments in this specification apply to data processing applications, platforms, or systems that have model validation capabilities.

[0159] Credit projects are projects used in the credit field to assess the creditworthiness of users such as individuals or businesses.

[0160] The validation dataset is the basic dataset used to validate the fit between the credit assessment model and the credit project. The validation dataset is obtained from the credit project and contains a data set of user attribute data and user behavior data in multiple dimensions, as well as the project results corresponding to the user attribute data and user behavior data.

[0161] The dimension refers to the attribute dimension to which user attribute data and user behavior data belong. Each dimension represents an attribute of a specific type of user attribute data and user behavior data.

[0162] User attribute data describes a user's basic information and static characteristics. This data is usually fixed and does not change frequently, reflecting the user's inherent attributes. Examples include a user's marital status, educational background, occupation, and place of residence.

[0163] User behavior data describes a user's dynamic behavior and activities over a specific period of time. This data reflects users' real-time behavior and habits, and typically changes over time. Examples include consumption behavior data, loan repayment behavior data, search behavior data, browsing behavior data, social activity data, and transaction behavior data.

[0164] The project results are the output results corresponding to user attribute data and user behavior data in the credit project. The project results are the actual output results in the credit project. The project results are used to verify the label output of the credit assessment model and are compared with the predicted project results output by the model to complete the verification of the credit assessment model. The project results can be binary or multi-category.

[0165] For example, before deploying a new version of the credit assessment model in a credit application, the effectiveness of the new version of the credit assessment model needs to be verified.

[0166] Using big data processing tools such as Hadoop or Spark, extract user attribute data and user behavior data across 50 dimensions, along with corresponding five project results, from the log database, transaction record database, and user database of the credit application.

[0167] User attribute data: occupation data, education level data, etc.

[0168] User behavior data: consumption behavior data, loan repayment behavior data, search behavior data, browsing behavior data, social activity data, transaction behavior data...

[0169] Project results: User rating A, User rating B, User rating C, User rating D, User rating E.

[0170] Based on the aforementioned 50 dimensions of user attribute data, user behavior data, and the corresponding 5 types of project results, a validation dataset is constructed.

[0171] A verification dataset is obtained from credit projects. This dataset includes user attribute data and user behavior data from multiple dimensions of the credit projects, as well as the corresponding project results. By obtaining real user attribute data, user behavior data, and corresponding project results from actual projects, the limitations of manually constructed data are avoided, the quality and authenticity of the data are improved, and the verification results are more consistent with real credit projects.

[0172] Step 204: Perform feature analysis on user attribute data and user behavior data, and extract the target dimension from multiple dimensions.

[0173] For example, statistical and machine learning algorithms are used to perform feature analysis on user attribute data (occupation data, education level data, etc.) and user behavior data (consumption behavior data, loan repayment behavior data, search behavior data, browsing behavior data, social activity data, transaction behavior data, etc.), extracting 10 target dimensions from 50 dimensions:

[0174] Occupation, education level, consumption behavior, borrowing and repayment behavior, transaction behavior...

[0175] By performing feature analysis on user attribute data and user behavior data, and extracting target dimensions from multiple dimensions, the scientific nature and relevance of data dimension selection are improved, providing data dimension support for the subsequent generation of real project scenarios that comprehensively cover different conditions.

[0176] Step 206: Based on the target dimension, generate at least one project scenario for the credit project.

[0177] The project scenarios for credit projects are used to simulate real-world scenarios under different conditions. Generating at least one project scenario ensures that the model can cover real-world situations under different conditions during validation, thereby comprehensively evaluating the performance of the credit assessment model. For example, in the credit field, the generated project scenarios include: Project Scenario 1: A young user with no credit history applies for a small-amount credit project; Project Scenario 2: A middle-aged user with a good credit history applies for a large-amount credit project.

[0178] Based on the target dimension, at least one project scenario of a credit project is generated. One possible approach is to use a decision tree algorithm to generate at least one project scenario of a credit project based on the target dimension. Specifically, the target dimension is used as a feature and the project result is used as a label to train a decision tree model. Rules are extracted from the decision tree, and each leaf node represents a project scenario. Based on the extracted rules, a specific project scenario is generated.

[0179] Based on the target dimension, at least one project scenario of a credit project can be generated. Another option is to use a random field algorithm to generate at least one project scenario of a credit project based on the target dimension. Specifically, the target dimension is used as a feature and the project result is used as a label. A random field model (such as a conditional random field CRF) is trained, and the model is used to generate project results under different combinations of dimensions to form project scenarios.

[0180] Based on the target dimension, at least one project scenario of the credit project can be generated. Alternatively, a clustering algorithm can be used to generate at least one project scenario of the credit project based on the target dimension. Specifically, the target dimension is used as a feature to train a clustering model (such as K-means, DBSCAN, etc.), and each cluster center and its nearby points can represent a project scenario.

[0181] Based on the target dimension, at least one project scenario for the credit project is generated. Alternatively, a genetic algorithm can be used to generate at least one project scenario for the credit project based on the target dimension. Specifically, a set of initial project scenarios is randomly generated. A fitness function is defined to evaluate the rationality of each project scenario. Through operations such as selection, crossover, and mutation, the project scenarios are gradually optimized, ultimately resulting in a set of optimized project scenarios.

[0182] Based on the target dimension, at least one project scenario of the credit project can be generated. Alternatively, a Bayesian algorithm can be used to generate at least one project scenario of the credit project based on the target dimension. Specifically, the probabilistic dependency between the target dimension and the project outcome is defined, and a Bayesian network is used to generate different samples, each representing a project scenario.

[0183] For example, using a random field algorithm, based on 10 target dimensions (occupation, education level, consumption behavior, borrowing and repayment behavior, transaction behavior, etc.), five scenarios are generated for user ratings on a credit platform:

[0184] Project Scenario 1: Young female users, traditional enterprises, secondary and higher education, high consumption, no borrowing behavior, frequent transaction behavior.

[0185] Project Scenario 2: Middle-aged male users, IT industry professionals, highly educated, moderate spending power, with a history of borrowing and timely repayments, and active in investment transactions.

[0186] Project Scenario 3: Elderly female users, retirees, with low or no high school education, low consumption, no borrowing behavior, and occasional small transactions.

[0187] Project Scenario 4: Young male users, freelancers, those with a low college degree, primarily engaged in online consumption, with overdue payment records, and low transaction frequency but large transaction amounts.

[0188] Project Scenario 5: Adult female users, education industry, stable intermediate master's degree, rational consumption, no need for loans, fixed deposits and small investments.

[0189] Based on the target dimension, at least one project scenario for credit projects is generated, ensuring that the generated project scenarios can comprehensively cover real project scenarios under different conditions for the target dimension with great impact, and providing scenario support for the subsequent prediction and matching process.

[0190] Step 208: Input user data from at least one project scenario into the credit assessment model applied to the credit project to obtain the predicted project results corresponding to user attribute data and user behavior data in at least one project scenario.

[0191] Credit assessment models are machine learning models specifically designed for credit projects. Through pre-training, these models learn patterns and regularities from historical data of credit projects, enabling them to predict or classify new data to adapt to the project environment and needs. Credit assessment models include, but are not limited to: linear regression models, logistic regression models, decision tree models, random forest models, clustering models, support vector machine models, neural network models, large language models, and generative Transformer models.

[0192] The predicted project results are the project output results corresponding to the user data output by the credit assessment model. The predicted project results are the simulated output results under at least one generated project scenario. The predicted output results are used to verify the predicted output of the credit assessment model and are compared with the project results to complete the verification of the credit assessment model. The predicted project results can be binary (e.g., whether the patient has a disease, whether the content was clicked, whether the information level is met) or multi-category (e.g., the type of disease, the category of content, and the credit rating).

[0193] User attribute data and user behavior data for at least one project scenario are input into a credit assessment model applied to a credit project to obtain predicted project results corresponding to user attribute data and user behavior data for at least one project scenario. One possible approach is to input user attribute data and user behavior data for at least one project scenario into a credit assessment model applied to a credit project to obtain predicted project results corresponding to user attribute data and user behavior data for at least one project scenario.

[0194] For example, according to five project scenarios (project scenario 1, project scenario 2, project scenario 3, project scenario 4, and project scenario 5), user attribute data and user behavior data are respectively input into a credit assessment model applied to user rating, and five predicted project results are obtained for each of the five project scenarios based on the user attribute data and user behavior data:

[0195] User rating A, User rating B, User rating C, User rating D, User rating E.

[0196] By inputting user data from at least one project scenario into a credit assessment model applied to credit projects, prediction results are obtained for user attribute data and user behavior data in at least one project scenario. This ensures the comprehensiveness and accuracy of the prediction results and improves the effectiveness of subsequent model testing.

[0197] Step 210: Based on the predicted project results and project results corresponding to user attribute data and user behavior data, perform project scenario matching to determine the matching degree between the credit assessment model and the credit project.

[0198] The matching degree between a credit assessment model and a credit project is a quantitative indicator of the applicability and performance of the credit assessment model in a credit project. The matching degree evaluates the effectiveness and risk of the credit assessment model in a credit project, and can also be used to guide future verification processes and further optimize the verification process. This includes, but is not limited to, at least one of the following: coding coverage, precision, and recall.

[0199] The matching degree between the credit assessment model and the credit project is determined by matching the predicted project results and the project results corresponding to user attribute data and user behavior data. One possible approach is to compare the predicted project results and the project results corresponding to user attribute data and user behavior data under at least one project scenario to determine the matching degree between the credit assessment model and the credit project.

[0200] For example, by comparing the predicted project results and project results (user rating A, user rating B, user rating C, user rating D, and user rating E) corresponding to user attribute data and user behavior data in five project scenarios, the matching degree between the credit assessment model and the credit project is determined to be 35%.

[0201] In this embodiment, a verification dataset from credit projects is obtained. This dataset includes user attribute data and user behavior data from multiple dimensions within the credit projects, as well as the corresponding project results. Obtaining authentic user attribute data, user behavior data, and corresponding project results from actual credit projects avoids the limitations of manually constructed data, improves data quality and authenticity, and makes the verification results more consistent with real credit projects. Feature analysis is performed on the user attribute data and user behavior data to extract target dimensions with significant impact on project results from multiple dimensions. Based on these target dimensions, at least one project scenario for the credit project is generated, ensuring that the generated project scenarios comprehensively cover real-world project scenarios under different conditions, targeting the most influential target dimensions. User data from at least one project scenario is input into the credit assessment model applied to the credit project. Predicted project results corresponding to the user attribute data and user behavior data for at least one project scenario are obtained. Project scenario matching is performed based on the predicted project results and the actual project results to determine the matching degree between the credit assessment model and the credit project. Comprehensive automated verification of the credit assessment model is completed in the generated project scenarios that conform to various real-world conditions, improving the accuracy and efficiency of model verification and reducing its cost.

[0202] The following is in conjunction with the appendix Figure 3 Taking the application of the credit assessment model verification method provided in this specification to user rating as an example, the verification method of the credit assessment model will be further explained. Among other things, Figure 3 The present specification provides a flowchart of a verification method for a credit assessment model applied to user rating, according to an embodiment of this specification, including the following specific steps:

[0203] Step 302: Using big data processing technology, extract user attribute data, user behavior data and corresponding project results from the log database, transaction record database and user database of the credit application, and construct a first wide data table that records user data and the corresponding project results.

[0204] In step 302, a validation dataset is provided for subsequent model validation.

[0205] Step 304: Statistically analyze the discreteness, distribution, and correlation characteristics of user attribute data and user behavior data in the first data wide table. Based on the discreteness, distribution, and correlation characteristics, extract the target dimension from multiple dimensions. Then, use a pre-trained feature analysis model to perform feature analysis on the user attribute data and user behavior data in the first data wide table and extract the target dimension from multiple dimensions.

[0206] In step 304, based on the discreteness, distribution and correlation characteristics of the data, the target dimensions that have a significant impact on the model decision are automatically selected from the wide table. This combines the inherent characteristics and logic of the data and improves the scientific nature and pertinence of the target dimension selection.

[0207] Step 306: Utilize the expert knowledge of the credit project to filter the target dimensions.

[0208] In step 306, based on the initial extraction of target dimensions, the target dimensions are screened and optimized in combination with domain expert knowledge to ensure that the target features are both representative and meet the needs of credit projects.

[0209] Step 308: Sort and modify the mapping relationship of user attribute data, user behavior data and project results in the first data wide table. Based on the target dimension, generate at least one project scenario of the credit project. Perform MD5 hash encoding on the user attribute data, user behavior data and project results of the target dimension in the first data wide table to obtain the first hash value of at least one project scenario.

[0210] In step 308, diverse scenario models are constructed using online data. A unique coded identifier is generated for each scenario using a consistent coding algorithm. Environmental differences are eliminated by sorting and modifying the mapping relationship, and the MD5 hash algorithm is used to ensure the uniqueness and comparability of the scenario codes.

[0211] Step 310: According to at least one project scenario, input user attribute data and user behavior data into the credit assessment model applied to user rating, obtain the predicted project results corresponding to user attribute data and user behavior data under at least one project scenario, and construct a second wide data table that records user attribute data, user behavior data and the predicted project results corresponding to user data. Perform MD5 hash encoding on the user attribute data, user behavior data and predicted project results of the target dimension in the second wide data table to obtain the second hash value of at least one project scenario.

[0212] In step 310, the use of consistent coding technology ensures the accuracy of cross-environment verification and improves the reliability and practicality of the verification results.

[0213] Step 312: Compare the first hash value and the second hash value to determine the coverage of at least one project scenario, and based on the coverage of at least one project scenario, determine the matching degree between the credit assessment model and the credit project.

[0214] Step 312 provides a systematic approach to quantify the coverage of validation data and the risk of model changes.

[0215] In the embodiments described in this specification, the narrow coverage and low efficiency of traditional methods are overcome through refined feature analysis and automated scenario construction, achieving more comprehensive and efficient model verification. Consistency coding technology ensures the accuracy of cross-environment verification, improves the reliability and practicality of verification results, significantly enhances the security and efficiency of model change management, and reduces project risks in credit projects caused by model problems.

[0216] Corresponding to the above method embodiments, this specification also provides embodiments of a model verification device. Figure 4 A schematic diagram of a model verification device according to one embodiment of this specification is shown. Figure 4 As shown, the device includes:

[0217] The first acquisition module 402 is configured to acquire a verification dataset from the target project, wherein the verification dataset includes user data of multiple dimensions in the target project and the project results corresponding to the user data;

[0218] The first extraction module 404 is configured to perform feature analysis on user data and extract the target dimension from multiple dimensions.

[0219] The first generation module 406 is configured to generate at least one project scenario of the target project based on the target dimension;

[0220] The first prediction module 408 is configured to input user data in at least one project scenario into a machine learning model applied to the target project, and obtain prediction project results corresponding to user data in at least one project scenario.

[0221] The first matching module 410 is configured to perform project scenario matching based on the predicted project results and project results corresponding to user data, and determine the matching degree between the machine learning model and the target project.

[0222] Optionally, the first extraction module 404 is further configured to: statistically analyze the data features of user data, and extract the target dimension from multiple dimensions based on the data features, wherein the data features include at least one of discrete features, distribution features, and correlation features; and / or, perform feature analysis on user data using a pre-trained feature analysis model to extract the target dimension from multiple dimensions, wherein the feature analysis model is trained based on sample user data of multiple sample dimensions, and the multiple sample dimensions are labeled with label dimensions.

[0223] Optionally, the device further includes a first filtering module configured to filter target dimensions using domain knowledge of the target project.

[0224] Optionally, user data and the corresponding project results are recorded in the first data table, and user data and predicted project results for at least one project scenario are recorded in the second data table.

[0225] Correspondingly, the device further includes: a first encoding module configured to encode user data and project results in the target dimension of the first data table to obtain a first encoding of at least one project scenario; and to encode user data and predicted project results in the target dimension of the second data table to obtain a second encoding of at least one project scenario;

[0226] Correspondingly, the first matching module 410 is further configured to: compare the first encoding and the second encoding to determine the matching degree between the machine learning model and the target item.

[0227] Optionally, the first encoding module is further configured to: perform hash encoding on the user data and project results of the target dimension in the first data table to obtain a first hash value for at least one project scenario; and perform hash encoding on the user data and predicted project results of the target dimension in the second data table to obtain a second hash value for at least one project scenario;

[0228] Correspondingly, the first matching module 410 is further configured to: compare the first hash value and the second hash value to determine the coverage of at least one project scenario; and determine the matching degree between the machine learning model and the target project based on the coverage of at least one project scenario.

[0229] Optionally, the apparatus further includes: a first preprocessing module configured to preprocess user data and project results in a first data table, wherein the preprocessing includes at least one of modifying the order between target dimensions, modifying the naming of target dimensions, modifying the mapping relationship between user data, and modifying the mapping relationship between user data and project results; and to preprocess user data and predicted project results in a second data table, wherein the preprocessing includes at least one of modifying the order between target dimensions, modifying the naming of target dimensions, modifying the mapping relationship between user data, and modifying the mapping relationship between user data and predicted project results.

[0230] In this embodiment, a validation dataset from the target project is obtained. This dataset includes user data from multiple dimensions within the target project and the corresponding project results. Obtaining real user data and project results from actual projects avoids the limitations of manually constructed data, improves data quality and authenticity, and makes the validation results more consistent with the real target project. Feature analysis is performed on the user data to extract target dimensions that have a significant impact on the project results. Based on these target dimensions, at least one project scenario for the target project is generated, ensuring that the generated scenario comprehensively covers real project scenarios under different conditions, targeting the most influential dimensions. User data from at least one project scenario is input into a machine learning model applied to the target project to obtain predicted project results for each user data in at least one project scenario. Project scenario matching is performed based on the predicted project results and the actual project results to determine the matching degree between the machine learning model and the target project. This comprehensive automated validation of the machine learning model is achieved within the generated project scenarios that conform to various real-world conditions, improving the accuracy and efficiency of model validation and reducing its cost.

[0231] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the model verification apparatus is basically similar to the model verification method embodiments, so it is described relatively simply; relevant parts can be referred to the descriptions of the model verification method embodiments.

[0232] Corresponding to the above method embodiments, this specification also provides embodiments of a verification device for a credit assessment model. Figure 5 This specification shows a schematic diagram of the structure of a verification device for a credit assessment model according to one embodiment. Figure 5 As shown, the device includes:

[0233] The second acquisition module 502 is configured to acquire a verification dataset from a credit project, wherein the verification dataset includes user attribute data and user behavior data of multiple dimensions in the credit project, as well as the project results corresponding to the user attribute data and user behavior data.

[0234] The second extraction module 504 is configured to perform feature analysis on user attribute data and user behavior data, and extract the target dimension from multiple dimensions.

[0235] The second generation module 506 is configured to generate at least one project scenario for credit projects based on the target dimension.

[0236] The second prediction module 508 is configured to input user data in at least one project scenario into a credit assessment model applied to a credit project, and obtain prediction project results corresponding to user attribute data and user behavior data in at least one project scenario.

[0237] The second matching module 510 is configured to perform project scenario matching based on the predicted project results and project results corresponding to user attribute data and user behavior data, and determine the matching degree between the credit assessment model and the credit project.

[0238] In this embodiment, a verification dataset from credit projects is obtained. This dataset includes user attribute data and user behavior data from multiple dimensions within the credit projects, as well as the corresponding project results. Obtaining authentic user attribute data, user behavior data, and corresponding project results from actual credit projects avoids the limitations of manually constructed data, improves data quality and authenticity, and makes the verification results more consistent with real credit projects. Feature analysis is performed on the user attribute data and user behavior data to extract target dimensions with significant impact on project results from multiple dimensions. Based on these target dimensions, at least one project scenario for the credit project is generated, ensuring that the generated project scenarios comprehensively cover real-world project scenarios under different conditions, targeting the most influential target dimensions. User data from at least one project scenario is input into the credit assessment model applied to the credit project. Predicted project results corresponding to the user attribute data and user behavior data for at least one project scenario are obtained. Project scenario matching is performed based on the predicted project results and the actual project results to determine the matching degree between the credit assessment model and the credit project. Comprehensive automated verification of the credit assessment model is completed in the generated project scenarios that conform to various real-world conditions, improving the accuracy and efficiency of model verification and reducing its cost.

[0239] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the description of the verification device for the credit assessment model is relatively simple, as it is basically similar to the embodiment of the verification method for the credit assessment model. Relevant parts can be referred to the description of the embodiment of the verification method for the credit assessment model.

[0240] Figure 6 A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0241] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or Near Field Communication (NFC).

[0242] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0243] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.

[0244] The processor 620 is used to execute the following computer program / instruction, which, when executed by the processor, implements the steps of the above-mentioned model verification method or credit assessment model verification method.

[0245] The above is a schematic representation of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the aforementioned model verification method and credit assessment model verification method. Details not described in detail in the technical solution of the computing device can be found in the descriptions of the technical solutions of the aforementioned model verification method or credit assessment model verification method.

[0246] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described model verification method or credit assessment model verification method.

[0247] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the above-described model verification method and credit assessment model verification method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the above-described model verification method or credit assessment model verification method.

[0248] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described model verification method or credit assessment model verification method.

[0249] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the above-described model verification method and credit assessment model verification method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the above-described model verification method or credit assessment model verification method.

[0250] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0251] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0252] It should be noted that the above description describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this specification.

[0253] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0254] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A model validation method, comprising: Obtain a validation dataset from the target project, wherein the validation dataset includes user data from multiple dimensions of the target project and the project results corresponding to the user data, and the user data and the project results corresponding to the user data are recorded in a first data table; Perform feature analysis on the user data and extract the target dimension from the multiple dimensions; Based on the target dimension, generate at least one project scenario for the target project; User data from at least one project scenario is input into a machine learning model applied to the target project to obtain predicted project results corresponding to the user data in at least one project scenario. The user data from at least one project scenario and the predicted project results are recorded in a second data table. Encode the user data and project results of the target dimension in the first data table to obtain a first code for at least one project scenario; Encode the user data and predicted project results of the target dimension in the second data table to obtain a second code for at least one project scenario; By comparing the first encoding and the second encoding, the matching degree between the machine learning model and the target project is determined.

2. The method according to claim 1, wherein performing feature analysis on the user data and extracting the target dimension from the plurality of dimensions includes: The data features of the user data are statistically analyzed, and a target dimension is extracted from the multiple dimensions based on the data features, wherein the data features include at least one of discreteness features, distribution features, and correlation features; or, The user data is subjected to feature analysis using a pre-trained feature analysis model, and the target dimension is extracted from the multiple dimensions. The feature analysis model is trained based on sample user data of multiple sample dimensions, and the multiple sample dimensions are labeled with a tag dimension.

3. The method according to claim 1, further comprising, before generating at least one project scenario for verification based on the target dimension: The target dimensions are filtered using domain knowledge of the target project.

4. The method according to claim 1, wherein encoding the user data and project results of the target dimension in the first data table to obtain a first encoding of at least one project scenario includes: Hash-encode the user data and project results of the target dimension in the first data table to obtain a first hash value for at least one project scenario; The step of encoding the user data and predicted project results of the target dimension in the second data table to obtain a second encoding of at least one project scenario includes: Hash-encode the user data and predicted project results of the target dimension in the second data table to obtain a second hash value for at least one project scenario; The step of comparing the first encoding and the second encoding to determine the matching degree between the machine learning model and the target project includes: By comparing the first hash value and the second hash value, the coverage of the at least one project scenario is determined; Based on the coverage of the at least one project scenario, the matching degree between the machine learning model and the target project is determined.

5. The method according to claim 1, prior to encoding the user data and project results of the target dimension in the first data table to obtain the first encoding of at least one project scenario, further comprising: The user data and project results in the first data table are preprocessed, wherein the preprocessing includes at least one of modifying the order between the target dimensions, modifying the naming of the target dimensions, modifying the mapping relationship between the user data, and modifying the mapping relationship between the user data and the project results; Before encoding the user data and predicted project results of the target dimension in the second data table to obtain a second encoding of at least one project scenario, the method further includes: The user data and the prediction results in the second data table are preprocessed, wherein the preprocessing includes at least one of modifying the order between the target dimensions, modifying the naming of the target dimensions, modifying the mapping relationship between the user data, and modifying the mapping relationship between the user data and the prediction results.

6. A method for validating a credit assessment model, comprising: Obtain a verification dataset from a credit project, wherein the verification dataset includes user attribute data and user behavior data of multiple dimensions in the credit project, as well as project results corresponding to the user attribute data and the user behavior data, and the user attribute data, user behavior data, and project results corresponding to the user attribute data and the user behavior data are in a first data table; Perform feature analysis on the user attribute data and the user behavior data, and extract the target dimension from the multiple dimensions; Based on the target dimension, at least one project scenario for the credit project is generated; User data from at least one project scenario is input into a credit assessment model applied to the credit project to obtain predicted project results corresponding to the user attribute data and user behavior data in at least one project scenario. The predicted project results corresponding to the user attribute data and user behavior data in at least one project scenario are recorded in a second data table. The user attribute data, user behavior data, and the project results corresponding to the user attribute data and user behavior data are encoded to obtain a first code for at least one project scenario; Encode the user attribute data and predicted project results of the target dimension in the second data table to obtain a second code for at least one project scenario; By comparing the first code and the second code, the matching degree between the credit assessment model and the credit project is determined.

7. A model verification device, comprising: The first acquisition module is configured to acquire a verification dataset from the target project, wherein the verification dataset includes user data of multiple dimensions in the target project and project results corresponding to the user data, and the user data and project results corresponding to the user data are recorded in a first data table; The first extraction module is configured to perform feature analysis on the user data and extract the target dimension from the multiple dimensions. The first generation module is configured to generate at least one project scenario of the target project based on the target dimension; The first prediction module is configured to input user data under the at least one project scenario into a machine learning model applied to the target project, and obtain the prediction project results corresponding to the user data under the at least one project scenario, wherein the user data under the at least one project scenario and the prediction project results are recorded in a second data table. The first matching module is configured to encode the user data and project results of the target dimension in the first data table to obtain a first code for at least one project scenario; encode the user data and predicted project results of the target dimension in the second data table to obtain a second code for at least one project scenario; and compare the first code and the second code to determine the matching degree between the machine learning model and the target project.

8. The apparatus according to claim 7, wherein the first extraction module is further configured as follows: The data features of the user data are statistically analyzed, and based on these data features, a target dimension is extracted from the multiple dimensions, wherein, The data features include at least one of discreteness features, distribution features, and correlation features; And / or, The user data is subjected to feature analysis using a pre-trained feature analysis model, and the target dimension is extracted from the multiple dimensions. The feature analysis model is trained based on sample user data of multiple sample dimensions, and the multiple sample dimensions are labeled with a tag dimension.

9. The apparatus according to claim 7, further comprising: The first filtering module is configured to use domain knowledge of the target project to filter the target dimensions.

10. The apparatus according to claim 7, wherein the first encoding module is further configured as follows: The user data and project results of the target dimension in the first data table are hashed to obtain a first hash value for at least one project scenario; the user data and predicted project results of the target dimension in the second data table are hashed to obtain a second hash value for at least one project scenario. The first matching module is further configured as follows: By comparing the first hash value and the second hash value, the coverage of the at least one project scenario is determined; based on the coverage of the at least one project scenario, the matching degree between the machine learning model and the target project is determined.

11. The apparatus according to claim 7, further comprising: A first preprocessing module is configured to preprocess the user data and the project results in the first data table, wherein the preprocessing includes at least one of modifying the order between the target dimensions, modifying the naming of the target dimensions, modifying the mapping relationship between the user data, and modifying the mapping relationship between the user data and the project results; and to preprocess the user data and the predicted project results in the second data table, wherein the preprocessing includes at least one of modifying the order between the target dimensions, modifying the naming of the target dimensions, modifying the mapping relationship between the user data, and modifying the mapping relationship between the user data and the predicted project results.

12. A verification device for a credit assessment model, comprising: The second acquisition module is configured to acquire a verification dataset from a credit project, wherein the verification dataset includes user attribute data and user behavior data of multiple dimensions in the credit project, as well as project results corresponding to the user attribute data and the user behavior data, and the user attribute data, user behavior data, and project results corresponding to the user attribute data and the user behavior data are in a first data table. The second extraction module is configured to perform feature analysis on the user attribute data and the user behavior data, and extract the target dimension from the multiple dimensions. The second generation module is configured to generate at least one project scenario of the credit project based on the target dimension. The second prediction module is configured to input user data under the at least one project scenario into the credit assessment model applied to the credit project, and obtain the prediction project results corresponding to the user attribute data and the user behavior data under the at least one project scenario, wherein the prediction project results corresponding to the user attribute data and the user behavior data under the at least one project scenario are recorded in the second data table. The second matching module is configured to encode the user attribute data, user behavior data, and the project results corresponding to the user attribute data and user behavior data to obtain a first code for at least one project scenario; encode the user attribute data and predicted project results of the target dimension in the second data table to obtain a second code for at least one project scenario; and compare the first code and the second code to determine the matching degree between the credit assessment model and the credit project.

13. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 6.

14. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images