Multi-model ensemble learning method based on financial risk control and related equipment
Through multi-model ensemble learning methods, including model fusion and distillation learning, low-resource student models are generated, which solves the problem of high resource consumption in the existing technology and realizes low-resource deployment and reasoning for multiple large-scale models.
Patent Information
- Application Number
- CN202411618700.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-13
- Publication Date
- 2025-02-25
AI Technical Summary
The existing financial risk control model integrated learning method consumes a lot of resources during deployment and inference, making it difficult to support the simultaneous deployment and inference of multiple large models.
The multi-model ensemble learning method is adopted, and by obtaining financial business data, training multiple neural network models, and model fusion is used to distillate the fusion model as a teacher model to generate low-resource student models.
It achieves the realization of ensuring the integration effect of multiple large models while reducing resource consumption, supporting low-resource online deployment and inference use, and improving the application capabilities of AI technology in the field of financial technology.
Smart Images
Figure CN119515529A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to the field of financial risk control technology, and specifically relates to a multi-model integrated learning method, device, electronic device, computer-readable storage medium, and computer program product based on financial risk control. Background Art
[0002] In the field of financial risk control, in order to control operating risks, financial companies need to continuously improve the performance of financial risk control models. However, financial data usually has the characteristics of complex patterns, high dimensions, strong nonlinearity, and multiple data types, which requires high training costs, so the performance requirements of machine learning are high.
[0003] In order to improve the ks value (Kolmogorov-Smirnov, a value used to evaluate the distinguishing ability of a classification model) of the model, the ensemble learning (or model integration) method is currently often used. By combining multiple machine learning models together and combining the prediction results of multiple models, the accuracy and stability of the prediction can be improved, thereby providing more reliable prediction results.
[0004] Ensemble learning is mostly done by training with different training methods to obtain multiple sub-models, and then using these sub-models to predict and score new samples at the same time. On the one hand, multiple sub-models need to be deployed when the model is launched, which increases the deployment cost and the cost of using inference resources; on the other hand, with the widespread use of large model technology, a single sub-model has brought great pressure to the system, and many systems find it difficult to support the simultaneous deployment and inference of multiple large models. Summary of the invention
[0005] In view of the above-mentioned defects or deficiencies in the prior art, it is desired to provide a multi-model integrated learning method, device, electronic device, computer-readable storage medium and computer program product, which can achieve low-resource online deployment and reasoning use while ensuring the integration effect of multiple large models.
[0006] In a first aspect, an embodiment of the present application provides a multi-model integrated learning method based on financial risk control, including:
[0007] Obtain financial business data, use multiple training methods to train multiple neural network models, and obtain multiple financial risk control models;
[0008] Performing model fusion on the multiple financial risk control models to obtain a fusion model;
[0009] The fusion model is used as a teacher model, and a pre-built student model is called to perform distillation learning on the teacher model; wherein the output layer dimension of the student model is lower than that of the teacher model;
[0010] The student model after distillation learning is output as an ensemble learning model.
[0011] In one embodiment, the multiple financial risk control models are integrated, including:
[0012] Counting the input sets of each of the financial risk control models and inputting them into the first model; wherein the number of output layers of the first model is the same as the number of the financial risk control models and the dimensions are the same;
[0013] Performing distillation learning on each output layer of the first model and the output layer of the corresponding financial risk control model;
[0014] Obtaining the scores output by each output layer of the first model to generate a final score;
[0015] Performing loss learning on the first model according to the loss between the final score and the standard score;
[0016] The first model output after loss learning is used as the fusion model.
[0017] In one embodiment, before using the first model output after loss learning as the fusion model, the method further includes:
[0018] The loss function of the distillation learning between each output layer of the first model and the output layer of the corresponding financial risk control model is used to perform loss learning on the first model.
[0019] In one embodiment, performing loss learning on the first model by distilling the loss function learned between each output layer of the first model and the output layer of the corresponding financial risk control model includes:
[0020] The mean square error or divergence loss of each output layer of the first model is calculated, and loss learning is performed on the first model.
[0021] In one embodiment, the fusion model is used as a teacher model, and a pre-built student model is called to perform distillation learning on the teacher model, including:
[0022] Mapping the output layer dimension of the student model to the output layer dimension of the teacher model;
[0023] Performing distillation learning on each output layer of the student model and the output layer of the mapped teacher model respectively;
[0024] The student model is subjected to loss learning according to the loss function of distillation learning.
[0025] In one embodiment, it also includes:
[0026] Loss learning is performed on the final scores of the teacher model as well as the hard labels.
[0027] In a second aspect, the embodiment of the present application provides a multi-model integrated learning device based on financial risk control, comprising:
[0028] The sub-model training unit is used to obtain financial business data and train multiple neural network models using multiple training methods to obtain multiple financial risk control models;
[0029] A multi-mode fusion unit, used to perform model fusion on the multiple financial risk control models to obtain a fusion model;
[0030] A low-resource distillation unit, used to use the fusion model as a teacher model and call a pre-built student model to perform distillation learning on the teacher model; wherein the output layer dimension of the student model is lower than that of the teacher model;
[0031] The model output unit is used to output the student model after distillation learning as an integrated learning model.
[0032] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the embodiment of the present application when executing the program.
[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the embodiment of the present application.
[0034] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described in the embodiment of the present application when executed by a processor.
[0035] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:
[0037] Figure 1 A schematic diagram of the process of the multi-model ensemble learning method provided in an embodiment of the present application is shown;
[0038] Figure 2 A schematic diagram of financial risk control model training provided by an embodiment of the present application is shown;
[0039] Figure 3 A schematic diagram of multi-model fusion provided in an embodiment of the present application is shown;
[0040] Figure 4 A schematic diagram of low-resource small model distillation provided in an embodiment of the present application is shown;
[0041] Figure 5 An exemplary structural block diagram of a multi-model integrated learning device provided in an embodiment of the present application is shown;
[0042] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION
[0043] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.
[0044] It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and in combination with the embodiments. Although the embodiments of the present application provide the method operation instruction steps shown in the following embodiments or drawings, more or fewer operation instruction steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiments of the present application. The method may be executed in the order of the methods shown in the embodiments or drawings or in parallel during the actual processing process or when the device is executed.
[0045] Please refer to Figure 1 , Figure 1 FIG. 1 is a flow chart of a multi-model integrated learning method provided by an embodiment of the present application. Figure 1 As shown, the method includes:
[0046] S101, obtaining financial business data, and using multiple training methods to train multiple neural network models respectively, to obtain multiple financial risk control models;
[0047] like Figure 2 The figure shows a schematic diagram of financial risk control model training, in which different training methods refer to respectively calling different data sets and parameters, training multiple neural network models with different model structures and training strategies, and obtaining multiple financial risk control models. Each model has good analysis capabilities for the target financial risk control issues.
[0048] To ensure that each financial risk control model achieves better results, each financial risk control model can use large model parameters, and the parameter types of the two are the same.
[0049] S102, integrating multiple financial risk control models to obtain an integrated model;
[0050] Use a large model structure to fuse multiple sub-model information into one model. Take the fusion model as an example, such as the transformer model. Figure 3 The figure shows a schematic diagram of multi-model fusion. The number of output layers of the fusion model is the same as the total number of financial risk control models. The dimension of each output layer is the same as the dimension of the output layer of the corresponding financial risk control model. The output values of each output layer are counted as the output results of the fusion model.
[0051] After model fusion, one model can be used to complete the integrated learning of multiple models, achieving an effective improvement in results.
[0052] S103, using the fusion model as the teacher model, calling the pre-built student model to perform distillation learning on the teacher model;
[0053] In order to further reduce online resource consumption, this embodiment adopts the "Teacher-Student Training Paradigm" and uses a small low-resource model to perform distillation learning on the fusion model.
[0054] The "teacher-student training method" usually refers to a knowledge distillation technique in deep learning, in which a well-trained and well-performing model (teacher model) guides the learning process of another untrained or simpler model (student model). Distillation learning, also known as knowledge distillation, refers to the transfer of knowledge learned by a complex model or an ensemble of multiple models (teacher model) to another lightweight model (student model).
[0055] In this embodiment, the fusion model is used as the teacher model, and the pre-built student model is called to perform distillation learning on the teacher model, wherein the student model adopts low-dimensional training, and the output layer dimension of the student model is lower than that of the teacher model, so as to achieve the effect of integrated learning of multiple large models through the online deployment of low-resource small models.
[0056] S104. Output the student model after distillation learning as an integrated learning model.
[0057] The trained student model output after distillation learning combines the business functions of multiple financial risk control models, ensuring the accuracy and stability of the prediction results under financial risk control issues; at the same time, the model is small in size and easy to deploy with low resources, so it can achieve low-resource online deployment and reasoning while ensuring the integration effect of multiple large models. After actual operation verification, this method improves ks by more than 0.5 on A card (Application Scorecard, referring to credit scores in the field of finance and credit assessment) samples compared to the traditional solution of deploying multiple large models in parallel.
[0058] It should be noted that although the operations of the method of the present invention are described in a particular order in the drawings, this does not require or imply that the operations must be performed in this particular order or that all illustrated operations must be performed to achieve desired results.
[0059] Based on the above introduction, the method provided in this embodiment will use multiple training methods to obtain multiple financial risk control models for multi-mode fusion. After fusion, one model can be used to complete the integrated learning of multiple models, and then the fused model is used as the teacher model, and the pre-built student model is called to perform distillation learning on the teacher model. A low-resource small model is used to perform distillation learning on the fused model, and the integrated learning capabilities of multiple large models learned by the fusion model are transferred to the lightweight student model, and the student model after distillation learning is output as the integrated learning model. In this way, the integrated learning model has the business functions of multiple financial risk control models, and at the same time, it is small in size, easy to deploy with low resources, and can achieve low-resource online deployment and reasoning while ensuring the improvement of the integration effect of multiple large models, thereby improving the application ability of AI technology in the field of financial technology.
[0060] The specific model fusion method used in step S102 in the above embodiment is not limited here. This embodiment proposes a specific model fusion method, which specifically includes the following steps:
[0061] 1. Count the input sets of each financial risk control model and input them into the first model;
[0062] To ensure that all sub-model information can be covered, the model input is the total set of inputs of all sub-models. Among them, the number of output layers (Hidden_layer) of the first model is the same as the number of financial risk control models, and the dimension of the first model is also the same as the dimension of each financial risk control model.
[0063] 2. Perform distillation learning on each output layer of the first model and the output layer of the corresponding financial risk control model;
[0064] Each output layer (refer to Figure 2In the example, Hidden_layer1, Hidden_layer2, and Hidden_layerN) are used to perform knowledge distillation with the output layer of the corresponding financial risk control model. The type of the specific distillation loss function used is not limited here, and it can be the mean square error (MSE) or the KL divergence loss, which can be selected according to the actual usage scenario.
[0065] 3. Obtain the scores output by each output layer of the first model to generate the final score;
[0066] Each output layer outputs a score (logit) (referring to Figure 2 In the formula, logit1, logit2, and logitN correspond to the scores output by the financial risk control model one by one. All the scores are counted (for example, weighted average) as the final score of the model. Taking weighted average statistics as an example, the final score can be called logit_avg.
[0067] 4. Perform loss learning on the first model according to the loss between the final score and the standard score, and use the first model output after loss learning as the fusion model.
[0068] The loss between the final score and the standard score can be the cross entropy loss between the final score and the standard score (label), which is not limited here.
[0069] To further improve the loss learning effect, in one embodiment, the model loss function may include two parts: the first part is the loss function of distillation learning, and the second part is the loss between the final score and the standard score, that is, the loss function of distillation learning between each output layer of the first model and the output layer of the corresponding financial risk control model is further used to perform loss learning on the first model. By training the loss functions of the above two parts, the effect can be effectively improved.
[0070] Among them, the specific type of distillation learning loss function is not limited in this embodiment. In one embodiment, the loss function of distillation learning can use the mean square error MSE or KL divergence loss of each output layer of the first model.
[0071] In order to effectively learn the information of the teacher model and improve the accuracy of the model, this embodiment proposes a low-resource small model distillation method for the above step S103, such as Figure 4 Shown is a schematic diagram of a low-resource small model distillation.
[0072] Specifically, step S103 uses the fusion model as the teacher model, and calls the pre-built student model to perform distillation learning on the teacher model, which may include the following steps:
[0073] 1. Map the output layer dimension of the student model to the output layer dimension of the teacher model;
[0074] 2. Perform distillation learning on each output layer of the student model and the output layer of the mapped teacher model respectively;
[0075] 3. Perform loss learning on the student model according to the loss function of distillation learning.
[0076] The student model is trained in low dimensions, and the dimension of the model output layer is lower than that of the large model (i.e., the teacher model). In order to effectively learn the teacher model information, the low-dimensional output layer is first mapped to the dimension of the teacher model, and then the student model is subjected to loss learning according to the loss function of distillation learning, such as mean square error (MSE) or KL divergence loss learning.
[0077] In addition, based on the above steps, loss learning can be performed on the final score and hard label of the teacher model to achieve further parameter optimization of the teacher model, thereby indirectly improving the recognition accuracy of the student model.
[0078] Further references Figure 5 , which shows an exemplary structural block diagram of a multi-model integrated learning device based on financial risk control according to an embodiment of the present application, which mainly includes:
[0079] The sub-model training unit 101 is used to obtain financial business data, and train multiple neural network models using multiple training methods to obtain multiple financial risk control models;
[0080] The multi-mode fusion unit 102 is used to fuse multiple financial risk control models to obtain a fusion model;
[0081] The low-resource distillation unit 103 is used to use the fusion model as a teacher model and call a pre-built student model to perform distillation learning on the teacher model; wherein the output layer dimension of the student model is lower than that of the teacher model;
[0082] The model output unit 104 is used to output the student model after distillation learning as an integrated learning model.
[0083] It should be understood that the units described in the above device are similar to those in the reference Figure 1 The various steps in the described method correspond to each other. Therefore, the operations and features described above for the method are also applicable to the device and the units contained therein, and will not be repeated here. The device can be pre-implemented in a browser or other security application of an electronic device, or loaded into a browser or its security application of an electronic device by downloading or the like. The corresponding units in the device can cooperate with the units in the electronic device to implement the solution of the embodiment of the present application.
[0084] For the several units mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units described above can be embodied in one unit. On the contrary, the features and functions of one unit described above can be further divided into multiple units to be embodied.
[0085] It should be noted that for details not disclosed in the multi-model integrated learning device based on financial risk control in the embodiment of the present application, please refer to the details disclosed in the above embodiments of the present application, which will not be repeated here.
[0086] Reference below Figure 6 , Figure 6 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.
[0087] like Figure 6 As shown, the computer system includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation instructions of the system are also stored. The CPU 601, the ROM 602 and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0088] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.
[0089] In particular, according to an embodiment of the present application, the above reference flow chart Figure 2The described process can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present application are executed.
[0090] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium such as a computer-readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0091] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operating instructions of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the aforementioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operating instruction, or can be implemented with a combination of dedicated hardware and computer instructions.
[0092] The units or modules involved in the embodiments described in the present application may be implemented by software or hardware. The units or modules described may also be arranged in a processor. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.
[0093] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the multi-model integrated learning method based on financial risk control described in the present application.
[0094] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.
Claims
1. A multi-model ensemble learning method based on financial risk control, characterized in that: include: Obtain financial business data, use multiple training methods to train multiple neural network models, and obtain multiple financial risk control models; Performing model fusion on the multiple financial risk control models to obtain a fusion model; The fusion model is used as a teacher model, and a pre-built student model is called to perform distillation learning on the teacher model; wherein the output layer dimension of the student model is lower than that of the teacher model; The student model after distillation learning is output as an ensemble learning model.
2. The method according to claim 1, characterized in that The multiple financial risk control models are integrated, including: Counting the input sets of each of the financial risk control models and inputting them into the first model; wherein the number of output layers of the first model is the same as the number of the financial risk control models and the dimensions are the same; Performing distillation learning on each output layer of the first model and the output layer of the corresponding financial risk control model; Obtaining the scores output by each output layer of the first model to generate a final score; Performing loss learning on the first model according to the loss between the final score and the standard score; The first model output after loss learning is used as the fusion model.
3. The method according to claim 2, characterized in that Before using the first model output after loss learning as the fusion model, the method further includes: The loss function of the distillation learning between each output layer of the first model and the output layer of the corresponding financial risk control model is used to perform loss learning on the first model.
4. The method according to claim 3, characterized in that The loss function of the distillation learning between each output layer of the first model and the output layer of the corresponding financial risk control model is used to perform loss learning on the first model, including: The mean square error or divergence loss of each output layer of the first model is calculated, and loss learning is performed on the first model.
5. The method according to claim 1, characterized in that The fusion model is used as a teacher model, and a pre-built student model is called to perform distillation learning on the teacher model, including: Mapping the output layer dimension of the student model to the output layer dimension of the teacher model; Performing distillation learning on each output layer of the student model and the output layer of the mapped teacher model respectively; The student model is subjected to loss learning according to the loss function of distillation learning.
6. The method according to claim 5, characterized in that Also includes: Loss learning is performed on the final scores of the teacher model as well as the hard labels.
7. A multi-model integrated learning device based on financial risk control, characterized in that: include: The sub-model training unit is used to obtain financial business data and train multiple neural network models using multiple training methods to obtain multiple financial risk control models; A multi-mode fusion unit, used to perform model fusion on the multiple financial risk control models to obtain a fusion model; A low-resource distillation unit, used to use the fusion model as a teacher model and call a pre-built student model to perform distillation learning on the teacher model; wherein the output layer dimension of the student model is lower than that of the teacher model; The model output unit is used to output the student model after distillation learning as an integrated learning model.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.