Test method, device, equipment and medium of credit risk classification model

Abstract

This disclosure provides a testing method for a credit risk classification model, relating to the field of artificial intelligence. The method includes: acquiring a first dataset, wherein the first dataset includes N first credit samples for pre-training and testing the credit risk classification model, wherein each first credit sample includes a first user feature and a first risk label; performing data mutation on the N first credit samples to obtain a second dataset, wherein the data mutation is used to relabel the samples according to a simplified relationship between user features and credit risk to obtain M second credit samples; and testing the credit risk classification model using at least a portion of the M second credit samples to obtain test results. This disclosure also provides a testing apparatus, device, storage medium, and program product for the credit risk classification model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a testing method, apparatus, device, medium, and program product for a credit risk classification model. Background Technology

[0002] Credit risk refers to the risk that loans cannot be recovered on schedule due to user default. To reduce losses caused by credit risk, financial institutions (such as commercial banks) will conduct pre-assessment and prediction of credit risk. For example, they will use machine learning models to build credit risk classification models, leverage the expertise and R&D experience in the field of credit risk to obtain user characteristics and labels, set model architecture and parameters, and train and test the model.

[0003] In the process of realizing the inventive concept disclosed herein, the inventors discovered that in some projects, the credit risk classification model did not yield ideal results in classifying users' credit risk levels, and the R&D personnel could not determine whether it was a code implementation error (i.e., a defect in program implementation) or a network design defect (i.e., a model design problem). Summary of the Invention

[0004] In view of the above problems, this disclosure provides a testing method, apparatus, device, medium and program product for automatically determining whether the unsatisfactory classification results are due to code implementation errors or network design defects in a credit risk classification model.

[0005] One aspect of this disclosure provides a method for testing a credit risk classification model, comprising: acquiring a first dataset, wherein the first dataset includes N first credit samples for pre-training and testing the credit risk classification model, wherein each first credit sample includes a first user feature and a first risk label; performing data mutation on the N first credit samples to obtain a second dataset, wherein the data mutation is used to relabel the samples according to a simplified relationship between user features and credit risk to obtain M second credit samples, where N and M are both greater than or equal to 1; and testing the credit risk classification model using at least a portion of the M second credit samples to obtain test results.

[0006] According to embodiments of this disclosure, each second credit sample includes a second user characteristic and a second risk label, after label relabeling based on a simplified relationship between user characteristics and credit risk: the number of types of the second risk label is less than or equal to the number of types of the first risk label.

[0007] According to an embodiment of this disclosure, data mutation of the N first credit samples includes: determining a mapping relationship based on a simplified relationship between the user characteristics and credit risk; and relabeling at least one column of features of the N first credit samples according to the mapping relationship, wherein each column of features includes at least a portion of the features of the first user characteristics.

[0008] According to an embodiment of this disclosure, relabeling at least one column of features of the N first credit samples according to the mapping relationship includes: determining the data type of each column of features, wherein the data type includes numerical, image, or text; and calling the mutation method corresponding to the data type and the mapping relationship to relabel the column of features.

[0009] According to an embodiment of this disclosure, when the data type is numeric, calling the mutation method corresponding to the data type and the mapping relationship to relabel the column features includes: sorting the numeric items of the column features according to their size to obtain a sorting result; dividing the numeric items of the column features into K groups of numeric items according to the sorting result, wherein each group of numeric items includes at least one numeric item, and K is greater than or equal to 1; and relabeling the K groups of numeric items according to the mapping relationship corresponding to the numeric type, wherein each group of numeric items has the same label, and at least one group of numeric items has a different label from the other groups of numeric items.

[0010] According to an embodiment of this disclosure, when the data type is text, calling the mutation method corresponding to the data type and the mapping relationship to relabel the column features includes: obtaining a corpus containing Q statements; dividing the column features into q groups of first text items, wherein each group of first text items includes at least one text, where Q and q are both greater than or equal to 1, and q is less than or equal to Q; taking q statements from the Q statements and inserting them into each group of first text items to obtain q groups of second text items; and relabeling the q groups of second text items according to the mapping relationship corresponding to the text type, wherein each group of second text items has the same label, and at least one group of numerical items has different labels from other groups of numerical items.

[0011] According to an embodiment of this disclosure, the mapping relationship corresponding to the text type includes the label corresponding to each of the q statements, and relabeling the labels of the q groups of second text items includes: relabeling the labels according to the statements inserted in each group of second text items.

[0012] According to an embodiment of this disclosure, when the data type is image type, calling the mutation method corresponding to the data type and the mapping relationship to relabel the column features includes: dividing the column features into H groups of first image items, wherein each group of first image items includes at least one first image item, and H is greater than or equal to 1; replacing at least some pixel values ​​of each image in each group of first image items with H replacement pixel values ​​to obtain H groups of second image items, wherein the H replacement pixel values ​​correspond one-to-one with the H groups of first image items; and relabeling the H groups of second image items according to the mapping relationship corresponding to the image type, wherein each group of second image items has the same label, and at least one group of image items has a different label from other groups of image items.

[0013] Another aspect of this disclosure provides a testing apparatus for a credit risk classification model, comprising: a data acquisition module for acquiring a first dataset, wherein the first dataset includes N first credit samples for pre-training and testing the credit risk classification model, wherein each first credit sample includes a first user feature and a first risk label; a data mutation module for performing data mutation on the N first credit samples to obtain a second dataset, wherein the data mutation is used to relabel the samples according to a simplified relationship between user features and credit risk to obtain M second credit samples, where N and M are both greater than or equal to 1; and a model testing module for testing the credit risk classification model using at least a portion of the M second credit samples to obtain test results.

[0014] The testing apparatus includes modules for performing each step of the method described in any one of the above descriptions.

[0015] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method as described above.

[0016] Another aspect of this disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method described above.

[0017] Another aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0018] The above one or more embodiments have the following beneficial effects: For a given credit risk classification model and its first dataset including training and testing data, by using data mutation to imply a simplified classification logic in the mutated second dataset based on the simplified relationship between user characteristics and credit risk, most classification models can achieve extremely high classification accuracy on this dataset without any program implementation defects. If the accuracy of the credit risk classification model is not as expected when tested using at least a portion of the second credit samples, it is determined that the model likely has program implementation defects. Attached Figure Description

[0019] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 This diagram illustrates an application scenario of a testing method for a credit risk classification model according to an embodiment of the present disclosure.

[0021] Figure 2 A flowchart illustrating a testing method for a credit risk classification model according to an embodiment of this disclosure is shown schematically.

[0022] Figure 3 A flowchart illustrating data variation according to an embodiment of this disclosure is shown schematically;

[0023] Figure 4 A flowchart illustrating data variation according to another embodiment of this disclosure is shown schematically;

[0024] Figure 5 A flowchart illustrating numerical tag remarking according to an embodiment of the present disclosure is shown schematically;

[0025] Figure 6 A flowchart illustrating text-based label remarking according to an embodiment of the present disclosure is shown schematically;

[0026] Figure 7 A flowchart illustrating the relabeling of image-type labels according to an embodiment of the present disclosure is shown schematically.

[0027] Figure 8 A schematic diagram illustrating the testing process for a neural network classification model is provided.

[0028] Figure 9 This schematically illustrates a structural block diagram of a test apparatus for a credit risk classification model according to an embodiment of the present disclosure; and

[0029] Figure 10 A block diagram of an electronic device suitable for implementing a test method for a credit risk classification model according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0030] To facilitate understanding of the technical solutions of the embodiments of this disclosure, some technical terms involved in this disclosure will be introduced first.

[0031] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0032] The main idea of ​​metamorphic testing is to generate new test inputs based on certain inherent attributes (metamorphic relationships) of the project under test, and then check whether the original output and the new output satisfy the metamorphic relationship. Taking neural networks as an example, the main difficulty in applying metamorphic testing to neural network projects lies in how to define the metamorphic relationships possessed by the neural network under test.

[0033] In the fintech industry, where software development is becoming increasingly large-scale, the application of neural networks in software development is also growing. For example, credit risk classification models are built using neural network classification models. However, for various reasons, neural network-based classification models often produce inaccurate classification results. For bank technology developers (especially junior developers), this not only involves extending and maintaining increasingly large and complex bank application frameworks, but also spending time figuring out the exact reasons for unsatisfactory neural network classification results. Determining whether the problem lies in code implementation errors (such as learning parameters in the model) or network design flaws (such as the number of layers in the model itself) is a particularly challenging task. Due to the inherent difficulty in interpreting and uncertainty of neural networks, developers often struggle to pinpoint the root cause of problems, making it difficult to fix them quickly and accurately.

[0034] Embodiments of this disclosure provide a testing method, apparatus, device, medium, and program product for a credit risk classification model. For a given credit risk classification model and its first dataset including training and testing data, a simplified classification logic is implicitly embedded in the mutated second dataset based on the simplified relationship between user characteristics and credit risk through data mutation. This allows most classification models to achieve extremely high classification accuracy on this dataset without any program implementation defects. If the accuracy of the credit risk classification model tested using at least a portion of the second credit samples does not meet expectations, it is determined that the model likely has program implementation defects.

[0035] Identifying the reasons for unsatisfactory classification results based on test findings can provide support for the design and improvement of credit risk classification models. For program design flaws, developers can discover and fix errors through code review and software testing. For model design problems, developers can reconsider the model's effectiveness and optimize the model design accordingly.

[0036] In the technical solutions disclosed herein, user authorization or consent is obtained before acquiring or collecting user personal information. The collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with relevant laws and regulations, necessary confidentiality measures are taken, and they do not violate public order and good morals.

[0037] Figure 1 The diagram illustrates an application scenario of a testing method for a credit risk classification model according to an embodiment of the present disclosure.

[0038] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as a medium for providing a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0039] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0040] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0041] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0042] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0043] The following will be based on Figure 1 The described scene, through Figures 2-8 The testing method for the credit risk classification model of the present disclosure embodiments is described in detail.

[0044] Figure 2 A flowchart illustrating a testing method for a credit risk classification model according to an embodiment of this disclosure is shown.

[0045] like Figure 2 As shown, the testing method for the credit risk classification model in this embodiment includes operations S210 to S230.

[0046] In operation S210, the first dataset is obtained, wherein the first dataset includes N first credit samples for pre-training and testing the credit risk classification model, wherein each first credit sample includes a first user feature and a first risk label;

[0047] For example, the first dataset includes a first training set and a first test set. The credit risk classification model can be trained using the first credit samples in the first training set and evaluated using the first test set. The credit risk classification model can be constructed using a neural network. Each first credit sample includes a feature vector of a single user and its corresponding risk label. The feature vector may include feature values ​​from dimensions such as ID documents, education level, employment, balance, property ownership, repayment history, and overdue information. The first risk label may include high risk, medium risk, and low risk, or five labels A, B, C, D, and E representing progressively decreasing risk (for example only).

[0048] In operation S220, data mutation is performed on N first credit samples to obtain a second dataset. The data mutation is used to relabel the labels according to the simplified relationship between user characteristics and credit risk to obtain M second credit samples, where N and M are both greater than or equal to 1.

[0049] For example, data mutation can include mutating a feature in one dimension and / or mutating the label in each sample. Mutation includes changing at least one of the original feature values, label values, and mapping relationships. The simplified relationship between user features and credit risk refers to further simplifying the mapping relationship between the original user feature vector and risk label. The core idea of ​​simplification is to construct a simpler step function between the label column and one of the feature columns in the dataset through relabeling, thus obtaining simplified classification data. M second credit samples can be divided into a second training set and a second test set for training and testing the credit risk classification model.

[0050] In some embodiments, each second credit sample includes a second user characteristic and a second risk label, after label relabeling based on a simplified relationship between user characteristics and credit risk: the number of types of the second risk label is less than or equal to the number of types of the first risk label.

[0051] Taking the example where the number of second risk labels is less than the number of first risk labels, based on the first user feature in each sample, which includes a credit score of 1 to 100 (the lower the score, the higher the risk), originally each 20 points corresponded to labels A, B, C, D, and E. According to the simplified relationship, scores of 1 to 50 correspond to label A', and scores of 51 to 100 correspond to label B'. Therefore, it is easier to obtain the corresponding second risk label (A' or B' label) based on the first user feature. Without program implementation defects, most classification models can achieve extremely high classification accuracy on this dataset.

[0052] Taking the example where the number of types of the second risk label equals the number of types of the first risk label, the feature values ​​of one or more dimensions can be mutated to make the classification logic simpler. For example, if a user's education level includes primary school, high school, junior college, bachelor's degree, master's degree, and doctoral degree, it can be changed to below bachelor's degree, bachelor's degree, and above bachelor's degree, and then assigned as high-risk, medium-risk, and low-risk labels, respectively. It should be noted that this is not limited to obtaining it directly based on education level; other dimensions of features can also be considered.

[0053] It is understandable that M can be greater than or equal to N, meaning that more second credit samples can be obtained by data mutation based on N first credit samples.

[0054] The data mutation process will be explained from an algorithmic perspective below.

[0055]

[0056]

[0057] As shown in Algorithm 1 above, there are four inputs and two outputs. The first input, `trainData`, and the second input, `testData`, represent the first training set and the first test set, respectively. The third input, `label`, enumerates all possible values ​​for the label column in `trainData` and `testData`. The last input, `k`, represents the number of types of the first or second risk label (which can be less than or equal to the number of label types in the label column of `trainData` and `testData`). As for the outputs, `NRDs` and `NTDs` represent the second training set and the second test set, respectively.

[0058] The workflow of Algorithm 1 is as follows. First, `testData` is concatenated to the end of `trainData` to merge them into a new dataset, the first dataset, named `newData` (line 1). Then, for each feature column in `newData`, a new dataset is generated iteratively within a `FOR` loop (lines 2-12). The iteration process is as follows:

[0059] (1) Mutate the label column of newData to simplify the mapping relationship between feature items and label items in the newData dataset (line 3). (2) Randomize newData (line 5) and divide it into two parts. The first part is used as the second training set (lines 6-8), and the second part is used as the second test set (lines 9-11).

[0060] In operation S230, the credit risk classification model is tested using at least a portion of the M second credit samples to obtain test results.

[0061] For example, the data included in the second test set is at least a portion of the samples in operation S230. The credit risk classification model (such as an untrained initial model) can be trained using the second training set. The following describes the defect diagnosis process for obtaining test results from an algorithmic perspective.

[0062]

[0063]

[0064] As shown in Algorithm 2, firstly, IsBuggy is initialized to FALSE (line 1). This is because we cannot yet determine the state of the defect, so we assume it to be FALSE for now. Next, the test classification model is diagnosed based on the variant data (lines 2-8). In each iteration (i.e., the i-th batch), the test classification model is run on the newly generated training set NRDs[i] and its corresponding new test set NTDs[i] (line 3). If the experimental precision on the new training set NRDs[i] or the new test set NTDs[i] is less than the pre-set threshold β, the value of IsBuggy is changed to TRUE, and the diagnostic process terminates (lines 4-6). If the algorithm successfully completes all iterations, the value of IsBuggy remains FALSE. Acc1 and Acc2 can be represented by the model's recall.

[0065] According to embodiments of this disclosure, for a given credit risk classification model and its first dataset including training and testing data, a simplified classification logic is implicitly embedded in the mutated second dataset based on the simplified relationship between user characteristics and credit risk through data mutation. This allows most classification models to achieve extremely high classification accuracy on this dataset without any program implementation defects. If the accuracy of the credit risk classification model tested using at least a portion of the second credit samples does not meet expectations, it is determined that the model likely has program implementation defects.

[0066] Figure 3 A flowchart illustrating data variation according to an embodiment of this disclosure is shown schematically.

[0067] like Figure 3 As shown, operation S220 performs data mutation on N first credit samples, including operations S310 to S320.

[0068] In operation S310, the mapping relationship is determined based on the simplified relationship between user characteristics and credit risk;

[0069] For example, the mapping relationship can include the relationship between one or more features and their corresponding credit risk labels. For instance, assigning "below undergraduate level," "undergraduate level," and "above undergraduate level" as high-risk, medium-risk, and low-risk labels respectively constitutes a mapping relationship.

[0070] In operation S320, at least one column of features of the N first credit samples is relabeled according to the mapping relationship, where each column of features includes at least a portion of the features of the first user. When the first user features are multi-dimensional feature vectors, each column of features is a feature term of one dimension of it.

[0071] Suppose the first dataset has m rows, n′ columns of features, and one column of labels (the last column). The first training set and the first test set have m1 and m2 rows respectively (m1 and m2 are added together to equal m).

[0072] First, for each feature c in the first dataset, it mutates and generates a new dataset by updating the corresponding label according to a predefined simple mapping m: label(c) = m(c). Therefore, the classification problem on the mutated data becomes a fitting problem to a simple mapping m. The latter is straightforward, and most mainstream neural network architectures can easily accomplish this. Therefore, if a given classification model fails to fit the mapping m successfully, i.e., its classification accuracy is low, the test model likely has implementation flaws.

[0073] Figure 4 A flowchart illustrating data variation according to another embodiment of this disclosure is shown schematically.

[0074] like Figure 4 As shown, operation S320 involves relabeling at least one column of features of N first credit samples according to the mapping relationship, including operations S410 to S420.

[0075] In operation S410, the data type of each column feature is determined, where the data type includes numeric, image, or text.

[0076] In operation S420, the mutation method and mapping relationship corresponding to the data type are called to relabel the features of the column.

[0077] The mutation step in line 3 of Algorithm 1 has different implementations depending on the data type. Data types can include various types such as numeric, image, text, sound, and signal, each with its own corresponding mutation method and mapping relationship. It's important to note that in the first dataset, all feature columns may have the same data type, or at least one feature column may have a different data type than the others.

[0078] The following sections will further introduce the three types: numerical, image, and text.

[0079] Figure 5 A flowchart illustrating numerical tag remarking according to an embodiment of the present disclosure is shown schematically.

[0080] like Figure 5 As shown, when the data type is numeric, the mutation method and mapping relationship corresponding to the data type are called to relabel the column features, including operations S510 to S530.

[0081] In operation S510, the values ​​of the column features are sorted according to their magnitude to obtain the sorting result; for example, from largest to smallest. Here, the numerical value refers to the numerical value of each row of the column features.

[0082] In operation S520, based on the sorting result, the numerical items of the column feature are divided into K groups of numerical items, where each group of numerical items includes at least one numerical item, and K is greater than or equal to 1.

[0083] For example, the numerical items of the column feature are evenly divided into K groups, where K is equivalent to k in Algorithm 1.

[0084] In operation S530, the K groups of numerical items are relabeled according to the mapping relationship corresponding to the numerical type, wherein each group of numerical items has the same label, and at least one group of numerical items has a different label from the other groups of numerical items.

[0085] In some embodiments, the first step is to read the contents of the source file and store the values ​​of the label column and the selected feature column used for mapping into two lists: `lab_list` and `ex_list`, respectively. The second step is to sort the two lists to remove duplicates. The third step is to find the lexicographically ordered position of each element in `ex_list` (containing feature column values) within that column and store it in `ex_dict`. The fourth step is to divide the elements of `ex_list` into `len(lab_list)` groups based on the number of enumerated elements after deduplication of the current label column. All samples in the same group are assigned the same label, while the labels for different groups are different. That is, the g-th... i All samples in the group were labeled. i Furthermore, the data mutation process for the following text and image datasets is similar to the process described above.

[0086] For example, the column feature could be user balance, meaning each row represents a user's balance. The numerical mapping includes risk labels associated with different balances; for instance, a smaller balance indicates a higher risk level. In practice, based on the simplified relationship between user characteristics and credit risk, the same risk label is assigned to users within the same balance range (each group), while users within different balance ranges are labeled differently.

[0087] In some embodiments, after obtaining the sorting results, the column features can be mutated, for example, by assigning the same balance value within the same balance range and relabeling the column based on the balance value, which further simplifies the classification logic without having to process and classify the different balance values ​​for each customer.

[0088] In some embodiments, it is assumed If f(x) = f(y) ∧ i ≠ j (where f(x) returns the feature value of sample x), then the mapping relationship between feature values ​​and label values ​​in the mutated dataset does not follow the known theorem: for every element in the domain, a unique corresponding value can be found in its destination domain. Therefore, the output of operation S530 is relabeled again to make the relationship between input and output conform to the normal mapping rule while minimizing the number of mutated samples. If |{w∈g i |f(w)=f(x)|>|{w∈g j |f(w)=f(x)|, then the sample {w∈g j The label of |f(w)=f(x) is set to label i Otherwise, the sample |{w∈g i The label of |f(w)=f(x)| is set to label j .

[0089] Figure 6 A flowchart illustrating text-based label remarking according to an embodiment of the present disclosure is shown schematically.

[0090] like Figure 6 As shown, when the data type is text, the mutation method and mapping relationship corresponding to the data type are called to relabel the column features, including operations S610 to S640.

[0091] Using the S610, a corpus containing Q statements is obtained;

[0092] For example, a small corpus containing a large number of simple statements can be predefined. In other words, the Q statements can be statements with clear themes, such as profit, loss, good news, resignation, or stock price decline, which can directly reflect the good or bad of the user's related financial events.

[0093] In operation S620, the column feature is divided into q groups of first text items, where each group of first text items includes at least one text, where Q and q are both greater than or equal to 1, and q is less than or equal to Q; where text item refers to each row of text features of the column feature.

[0094] Suppose that this feature column has multiple text items and their corresponding labels, and all labels belong to multiple categories. For example, the feature column can be evenly divided into q groups of the first text items, where q is equivalent to k in Algorithm 1. For example, if the credit applicant is a listed company, the text items in this feature column could represent relevant financial news of the listed company, such as changes in shareholding, appointments of senior executives, financial reports, or bankruptcy liquidation content.

[0095] In operation S630, q statements are taken from Q statements and inserted into the first text item of each group respectively to obtain the second text item of q groups;

[0096] We can randomly select q statements and insert each statement into every text item in the corresponding group. For example, we can insert the same statement from the corpus at the beginning of all text items in the same group and assign the same label to all text items in that group. The statements inserted and the labels attached are different between different groups. That is, the g-th statement... i All text items in the group are assigned the statement q. i and label i .

[0097] In operation S640, the second text items of group q are relabeled according to the mapping relationship corresponding to the text type, wherein each group of second text items has the same label, and at least one group of numerical items has a different label from the other group of numerical items.

[0098] According to embodiments of this disclosure, the text-type mapping relationship includes a label corresponding to each of the q statements. Relabeling the q groups of second text items includes relabeling based on the statements inserted in each group of second text items. For example, the text-type mapping relationship includes the relationship between business performance and different risk labels. For instance, adding "achieved profitability this quarter" to the beginning of each text in the first group of text items and labeling the first group of text items as low-risk. Adding "facing bankruptcy" to the beginning of each text in the second group of text items and labeling the second group of text items as high-risk.

[0099] According to embodiments of this disclosure, by inserting clearly defined statements into each text, simplified classification data can be obtained, enabling a very simple classification logic between the text and tags in the second text item.

[0100] Figure 7 A flowchart illustrating the relabeling of image-type labels according to an embodiment of the present disclosure is shown schematically.

[0101] like Figure 7 As shown, when the data type is image, the mutation method and mapping relationship corresponding to the data type are called to relabel the features of the column, including operations S710 to S730.

[0102] In operation S710, the column feature is divided into H groups of first image items, where each group of first image items includes at least one first image item, and H is greater than or equal to 1; where the image item includes the image feature of each row of the column feature.

[0103] Suppose that this feature column contains multiple images and their corresponding labels, and all labels belong to more than k categories. For example, all images can be evenly divided into H groups, where H is equivalent to k in Algorithm 1.

[0104] In operation S720, at least some pixel values ​​of each image in each group of first image items are replaced by H replacement pixel values ​​to obtain H groups of second image items. The H replacement pixel values ​​correspond one-to-one with the H groups of first image items.

[0105] For example, H replacement pixel values ​​can be randomly selected from [0, 255], and these replacement pixel values ​​can be used to replace the first 25% (for example only) of pixels in all images within the same group. For instance, 10 images can be divided into two groups, and two replacement pixel values, a and b, can be randomly selected from [0, 255]. In the first group, the pixel values ​​in the first 25 regions of each image can be replaced with a, and in the second group, the pixel values ​​in the first 25 regions of each image can be replaced with b.

[0106] In operation S730, the second picture items of group H are relabeled according to the mapping relationship corresponding to the picture type, wherein each group of second picture items has the same label, and at least one group of picture items has a different label from other groups of picture items.

[0107] For example, the replacement pixels and labels matched between different groups are not the same. That is, the g-th pixel... i All images in the group are assigned a replacement pixel value H. i and label i .

[0108] In some embodiments, the mapping relationship between images includes the relationship between facial images of different ages and different risk labels. For example, youth, middle-aged, and elderly correspond to medium-risk, low-risk, and high-risk, respectively. By replacing pixels, different ages can be represented by the replaced colors, and corresponding labels can be assigned accordingly. This further simplifies the age classification logic of facial recognition.

[0109] According to embodiments of this disclosure, by replacing pixel values ​​for each image and obtaining simplified classification data based on the replaced color, a very simple classification logic can be achieved between images and labels in the second image item.

[0110] Figure 8 A flowchart illustrating the testing process for a neural network classification model is shown.

[0111] Based on the testing method for the credit risk classification model described above, this disclosure also provides a testing method for a neural network classification model. The method includes: obtaining a source dataset, wherein the source dataset includes n source samples for pre-training and testing the neural network classification model, wherein each source sample includes source sample features and source sample labels; performing data mutation on the n source samples to obtain a new dataset, wherein the data mutation is used to relabel the samples according to a simplified relationship between the source sample features and the classification labels to obtain m new samples, where n and m are both greater than or equal to 1; and testing the neural network classification model using at least a portion of the m new samples to obtain test results.

[0112] like Figure 8 As shown, and refer to Figures 2-7 In this embodiment, the defect diagnosis of the neural network classification model is divided into two parts: data mutation and defect diagnosis based on mutated data. The first part involves generating a new training dataset and its corresponding test dataset based on the source dataset. For each feature column f in the original data... i The first part generates new datasets (including training and testing data) by replacing labels with simple mappings m(fi), where m is a predefined simple mapping (a mapping relationship determined based on simplification relations). For example, performing a simple mapping on n feature columns can yield n new datasets. Furthermore, data mutation schemes are provided for three common dataset types: numerical, image, and text. The second part involves automated diagnostics based on the mutated data to determine if the neural network classification model under test has implementation errors. A pre-set accuracy threshold can be set; if the test model cannot achieve a high accuracy above the threshold on the mutated dataset, it is determined that the model under test likely has implementation defects.

[0113] In this embodiment, the source dataset is not limited to credit risk data and may include multidimensional features and different labels of other targets to be classified. The simplified relationship between the source sample features and the classification labels can be regarded as the metamorphic relationship in the metamorphic testing process (a certain relationship between multiple input / output pairs of the tested model).

[0114] In some embodiments, a neural network visualization diagnostic tool based on data mutation diagnosis can be provided. This tool comprises two modules: a data mutation module and a model visualization module. The data mutation module extracts the source dataset from the uploaded project and mutates it to generate new test cases for diagnosis. The model visualization module obtains the model built in the uploaded project and visualizes the neural network architecture and parameters. Clients access the tool via a browser, upload the source dataset file and network model file from the project to be tested to the corresponding tool module, and can then use the neural network classification model defect diagnosis function.

[0115] Based on the testing method of the above-mentioned credit risk classification model, this disclosure also provides a testing device for the credit risk classification model. The following will combine... Figure 9 The device is described in detail.

[0116] Figure 9 A schematic block diagram of a test apparatus for a credit risk classification model according to an embodiment of the present disclosure is shown.

[0117] like Figure 9 As shown, the testing device 900 for the credit risk classification model in this embodiment includes a data acquisition module 910, a data variation module 920, and a model testing module 930.

[0118] The data acquisition module 910 can perform operation S210 to acquire a first dataset, wherein the first dataset includes N first credit samples for pre-training and testing the credit risk classification model, wherein each first credit sample includes a first user feature and a first risk label;

[0119] The data mutation module 920 can perform operation S220 to perform data mutation on N first credit samples to obtain a second dataset. The data mutation is used to relabel the labels according to the simplified relationship between user characteristics and credit risk to obtain M second credit samples, where N and M are both greater than or equal to 1.

[0120] According to the embodiments of this disclosure, the data mutation module 920 can perform operations S310 to S320, operations S410 to S420, operations S510 to S530, operations S610 to S630, and operations S710 to S730, which will not be described in detail here.

[0121] According to embodiments of this disclosure, the text-type mapping relationship includes the label corresponding to each of the q statements, and the label re-marking of the q groups of second text items includes: re-marking the labels according to the statements inserted in each group of second text items.

[0122] According to embodiments of this disclosure, each second credit sample includes a second user characteristic and a second risk label, after label relabeling based on a simplified relationship between user characteristics and credit risk: the number of types of the second risk label is less than or equal to the number of types of the first risk label.

[0123] The model testing module 930 can perform operation S230 to test the credit risk classification model using at least a portion of the M second credit samples and obtain test results.

[0124] It should be noted that the testing device 900 includes components for performing the above-mentioned tasks. Figures 2-7The modules of each step in any of the described embodiments. The implementation methods, technical problems solved, functions achieved, and technical effects of each module / unit / subunit in the apparatus embodiments are the same as or similar to the implementation methods, technical problems solved, functions achieved, and technical effects of the corresponding steps in the method embodiments, and will not be repeated here.

[0125] According to embodiments of this disclosure, any plurality of modules among the data acquisition module 910, data mutation module 920, and model testing module 930 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least some of the functionality of one or more of these modules may be combined with at least some of the functionality of other modules and implemented in one module.

[0126] According to embodiments of this disclosure, at least one of the data acquisition module 910, data mutation module 920, and model testing module 930 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the data acquisition module 910, data mutation module 920, and model testing module 930 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0127] Figure 10 A block diagram of an electronic device suitable for implementing a test method for a credit risk classification model according to an embodiment of the present disclosure is shown schematically.

[0128] like Figure 10 As shown, an electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0129] RAM 1003 stores various programs and data required for the operation of electronic device 1000. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Processor 1001 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1002 and / or RAM 1003. It should be noted that programs may also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0130] According to embodiments of this disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to a bus 1004. The electronic device 1000 may also include one or more of the following components connected to the I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1010 as needed so that computer programs read from it can be installed into the storage section 1008 as needed.

[0131] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0132] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 1002 and / or RAM 1003 and / or one or more memories other than ROM 1002 and RAM 1003 described above.

[0133] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0134] When the computer program is executed by the processor 1001, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0135] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1009, and / or installed from a removable medium 1011. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0136] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by processor 1001, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0137] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0139] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0140] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for testing a credit risk classification model, comprising: obtaining a first data set, wherein the first data set comprises N first credit samples for pre-training and testing the credit risk classification model, wherein each first credit sample comprises first user features and a first risk label; performing data variation on the N first credit samples to obtain a second data set, wherein the data variation is used for label re-labeling according to a simplified relationship between user features and credit risk to obtain M second credit samples, and N and M are greater than or equal to 1; testing the credit risk classification model using at least part of the M second credit samples to obtain a test result; wherein the data variation on the N first credit samples comprises: determining a mapping relationship according to the simplified relationship between the user features and the credit risk; determining a data type of each column feature, wherein the data type comprises a numerical type, a picture type or a text type, and the column feature comprises at least part of the first user features; and calling a variation mode corresponding to the data type and the mapping relationship to perform the label re-labeling on the column feature; when the data type is the numerical type, sorting the numerical items of the column feature according to their sizes to obtain a sorting result; dividing the numerical items of the column feature into K groups of numerical items in sequence according to the sorting result, wherein each group of numerical items comprises at least one numerical item, and K is greater than or equal to 1; and performing the label re-labeling on the K groups of numerical items according to the mapping relationship corresponding to the numerical type, wherein each group of numerical items has the same label, and the label of at least one group of numerical items is different from that of other groups of numerical items.

2. The method of claim 1, wherein, Each second credit sample comprises second user features and a second risk label, and after the label re-labeling according to the simplified relationship between the user features and the credit risk: the number of categories of the second risk label is less than or equal to that of the first risk label.

3. The method of claim 1, wherein, when the data type is the text type, calling the variation mode corresponding to the data type and the mapping relationship to perform the label re-labeling on the column feature comprises: obtaining a corpus comprising Q sentences; dividing the column feature into q groups of first text items, wherein each group of first text items comprises at least one text, Q and q are greater than or equal to 1, and q is less than or equal to Q; inserting q sentences from the Q sentences into each group of first text items to obtain q groups of second text items; performing the label re-labeling on the q groups of second text items according to the mapping relationship corresponding to the text type, wherein each group of second text items has the same label, and the label of at least one group of numerical items is different from that of other groups of numerical items.

4. The method of claim 3, wherein, the mapping relationship corresponding to the text type comprises a label corresponding to each sentence in the q sentences, and the label re-labeling on the q groups of second text items comprises: performing the label re-labeling according to the inserted sentence in each group of second text items.

5. The method of claim 1, wherein, when the data type is the picture type, calling the variation mode corresponding to the data type and the mapping relationship to perform the label re-labeling on the column feature comprises: dividing the column of features into H groups of first image items, wherein each group of first image items comprises at least one first image item, and H is greater than or equal to 1; replacing at least part of pixel values of each image in each group of first image items with H replacement pixel values respectively to obtain H groups of second image items, the H replacement pixel values correspond to the H groups of first image items one by one; relabeling each group of second image items according to the mapping relationship corresponding to the picture type, wherein each group of second image items has the same label, and at least one group of image items has a different label from other groups of image items.

6. An apparatus for testing a credit risk classification model, comprising: a data acquisition module configured to acquire a first data set, wherein the first data set comprises N first credit samples used for pre-training and testing the credit risk classification model, and each first credit sample comprises first user features and a first risk label; a data variation module configured to perform data variation on the N first credit samples to obtain a second data set, wherein the data variation is configured to relabel M second credit samples according to a simplified relationship between user features and credit risks, and N and M are both greater than or equal to 1; wherein the data variation on the N first credit samples comprises: determining a mapping relationship according to the simplified relationship between the user features and the credit risks; determining a data type of each column of features, wherein the data type comprises a numerical type, a picture type, or a text type, and each column of features comprises at least part of the first user features; calling a variation mode corresponding to the data type and the mapping relationship to relabel the column of features; when the data type is the numerical type, sorting numerical items of the column of features according to their sizes to obtain a sorting result; and dividing the numerical items of the column of features into K groups of numerical items according to the sorting result, wherein each group of numerical items comprises at least one numerical item, and K is greater than or equal to 1; and relabeling the K groups of numerical items according to the mapping relationship corresponding to the numerical type, wherein each group of numerical items has the same label, and at least one group of numerical items has a different label from other groups of numerical items; a model testing module configured to test the credit risk classification model using at least part of the M second credit samples to obtain a test result.

7. An electronic device, comprising: one or more processors; a storage device configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of claims 1-5.

8. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-5.

9. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.

Images