Model-based credibility determination method and apparatus, and program product

Abstract

The embodiment of the invention provides a model-based credibility determination method and device and a program product, and the method comprises the steps: obtaining business processing information which is extracted from a target business system and is used for processing business data; under the condition that a model for processing the business data is determined to be a target model according to the business processing information, N questions and corresponding N reference answers are generated, and the N questions are related to judging whether the model belongs to the target model or not; displaying the N questions, and receiving N answers for answering the N questions; and determining the credibility of the model as the target model based on the N answers and the N reference answers. According to the method and the device, the problem that the model cannot be accurately identified in the prior art is solved, and the effect of accurately identifying the model is achieved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and in particular, to a method and device for determining model-based credibility, and a program product. Background Art

[0002] With the rapid advancement of digital transformation in all walks of life, models have been widely used in business processing, identity verification, anti-fraud, access, credit limit, rating, collection, valuation and pricing, and other links, becoming a key factor in business management and risk decision-making. Once the model decision is wrong, it will bring direct or indirect losses. Improper model risk management will bring regulatory compliance risks. Model risk, as a new type of risk faced by enterprises in business management, must be effectively managed. In existing technologies, model identification is based on manual experience and cannot accurately identify models. Summary of the invention

[0003] The embodiments of the present application provide a method and device for determining the credibility of a model, and a program product, so as to at least solve the problem that the model cannot be accurately identified in the related art.

[0004] According to an embodiment of the present application, a method for determining model-based credibility is provided, comprising: obtaining business processing information, wherein the business processing information is information extracted from a target business system for processing business data; determining that a model for processing the business data is a target model based on the business processing information, generating N questions and corresponding N reference answers, wherein the N questions are related to determining whether the model belongs to the target model; displaying the N questions and receiving N answers for answering the N questions; and determining the credibility of the model as the target model based on the N answers and the N reference answers.

[0005] According to an embodiment of the present application, a device for determining the credibility of a model is provided, comprising: a first acquisition module, used to acquire business processing information, wherein the business processing information is information extracted from a target business system for processing business data; a first generation module, used to determine that a model for processing the business data is a target model based on the business processing information, and to generate N questions and corresponding N reference answers, wherein the N questions are related to determining whether the model belongs to the target model; a first display module, used to display the N questions and receive N answers for answering the N questions; and a first determination module, used to determine the credibility of the model as the target model based on the N answers and the N reference answers.

[0006] In an exemplary embodiment, the device further includes: a second determination module, which is used to determine that the model for processing the business data is the target model according to the business processing information, and before generating N questions and corresponding N reference answers, if the business processing information includes exclusion information, determine that the model does not belong to the target model, wherein the exclusion information includes at least one of the following: a report for processing the business data, a formula for processing the business data, and indicator information for processing the business data; a second acquisition module, which is used to acquire M model information when the business processing information does not include the exclusion information and the business processing information includes a process of the model processing the business data, wherein the M model information includes at least one of the following: information of inputting the business data into the model, the process of the model processing the business data, information of the result of the model processing the business data, the frequency of the model processing the business data, the prediction performance of the model, and the process of generating the model; a first evaluation module, which is used to evaluate the feature score of each of the model information according to the target evaluation index to obtain M feature scores; and a first judgment module, which is used to judge whether the model is the target model based on the M feature scores.

[0007] In an exemplary embodiment, the first judgment module includes: a first calculation unit, used to calculate the sum of the M feature scores mentioned above; a first comparison unit, used to compare the sum and the target threshold, and judge whether the model is the target model based on the comparison result, wherein when the sum is greater than the target threshold, the model is determined to be the target model.

[0008] In an exemplary embodiment, the above-mentioned first generation module includes: a first receiving unit, used to receive a verification request when it is determined that the model for processing the above-mentioned business data belongs to the target model based on the above-mentioned business processing information, wherein the above-mentioned verification request is used to request verification of the credibility of the above-mentioned model as the above-mentioned target model; a first response unit, used to respond to the above-mentioned verification request, and generate the N above-mentioned questions and the corresponding N above-mentioned reference answers according to the processing characteristics of the above-mentioned business data by the above-mentioned model.

[0009] In an exemplary embodiment, the first determination module includes: a first comparison unit, used to compare the N answers and the N reference answers according to the N questions to obtain N comparison results; a first determination unit, used to determine that the credibility of the model as the target model is a preset credibility when the number of target comparison results included in the N comparison results is greater than a preset number, wherein the target comparison result is used to indicate that the answer is consistent with the corresponding reference answer.

[0010] In an exemplary embodiment, the above-mentioned device also includes: a first reading module, which is used to read the text content in the above-mentioned target business system before obtaining the business processing information, wherein the above-mentioned text content includes at least one of the following: the above-mentioned business data, and the requirement information for processing the above-mentioned business data; a first processing module, which is used to perform word frequency statistics processing on the above-mentioned text content to extract model requirement information from the above-mentioned text content, wherein the above-mentioned model requirement information is used to represent the model information required for processing the above-mentioned business data.

[0011] In an exemplary embodiment, according to another embodiment of the present application, a computer program product is further provided, including a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0012] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when running.

[0013] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0014] Through this application, due to the information for processing business data extracted from the target business system, when the model for processing the above business data is determined to be the target model based on the business processing information, N questions and corresponding N reference answers are generated, the N above questions are displayed, and N answers for answering the N above questions are received, and the credibility of the above model as the above target model is determined based on the N above answers and the N above reference answers. That is, when the model is determined to be the target model, the credibility of the model as the target model is verified again through N questions, which further increases the accuracy of model identification. Therefore, the problem of not being able to accurately identify the model in the related art can be solved, and the effect of accurately identifying the model can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a hardware structure block diagram of a mobile terminal of a method for determining credibility based on a model according to an embodiment of the present application;

[0016] Figure 2 is a flow chart of a method for determining a credibility based on a model according to an embodiment of the present application;

[0017] Figure 3 is a flowchart of determining whether it is a target model through a scoring card model according to an embodiment of the present application;

[0018] Figure 4 is a flow chart of generating N questions through an interactive question-answering model according to an embodiment of the present application;

[0019] Figure 5 is a flow chart of performing word frequency statistics processing on text content through a large language model according to an embodiment of the present application;

[0020] Figure 6 is an overall flow chart of model identification according to a specific embodiment of the present application;

[0021] Figure 7 It is a structural block diagram of a device for determining model-based credibility according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0024] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a method for determining credibility based on a model according to an embodiment of the present application. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0025] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the model-based credibility determination method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0026] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0027] In this embodiment, a method for determining the credibility based on a model is provided. Figure 2 is a flow chart of a method for determining credibility based on a model according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0028] Step S202, obtaining business processing information, wherein the business processing information is information extracted from the target business system for processing business data;

[0029] Step S204, when it is determined according to the business processing information that the model for processing business data is the target model, N questions and corresponding N reference answers are generated, wherein the N questions are related to determining whether the model belongs to the target model;

[0030] Step S206, displaying N questions and receiving N answers for answering the N questions;

[0031] Step S208, determining the credibility of the model as the target model based on the N answers and the N reference answers.

[0032] Among them, the executor of the above steps can be a terminal, a server, a specific processor set in the terminal or server, or a processor or processing device set relatively independently from the terminal or server, or it can be the pool robot itself, etc., but is not limited to this.

[0033] Through the above steps, due to the information for processing business data extracted from the target business system, when the model for processing the above business data is determined to be the target model according to the business processing information, N questions and corresponding N reference answers are generated, the N above questions are displayed, and N answers for answering the N above questions are received, and the credibility of the above model as the above target model is determined based on the N above answers and the N above reference answers. That is, when the model is determined to be the target model, the credibility of the model as the target model is verified again through N questions, which further increases the accuracy of model identification. Therefore, the problem of not being able to accurately identify the model in the related art can be solved, and the effect of accurately identifying the model can be achieved.

[0034] Optionally, the target business system includes, but is not limited to, a business processing system in a bank or financial industry. It may include one or more business processing models. For example, a banking business system includes a data processing model for processing transaction data, a data collection model for collecting data, etc. The type of the model may be a neural network model, a machine learning model, etc.

[0035] Optionally, the information for processing business data includes information such as a plurality of business processing requirements, business processing processes, business processing results, etc. For example, the business processing requirements include model requirements, network requirements, hardware requirements, application requirements, and the like.

[0036] Optionally, the information of processing business data can be processed by a large language model to obtain business processing information; N questions and corresponding N reference answers can be generated by an interactive question-and-answer model; and the credibility of the model as a target model can be determined by a scoring card model. The large language model, the interactive question-and-answer model, and the scoring card model can be models obtained through machine learning training. The large language model, the interactive question-and-answer model, and the scoring card model are all set in the target business system.

[0037] Optionally, the N questions include but are not limited to the following questions: "Are the input and output of the model similar to the target model", "What is the accuracy and stability of the model", "What are the training data and validation data of the model", "Does the model use similar data sets", "What are the application scenarios and usage methods of the model", "What are the advantages, disadvantages and limitations of the model". These questions can help to comprehensively evaluate the judgment model, so as to determine whether it belongs to the target model.

[0038] Optionally, to determine whether a model belongs to the target model, model evaluation and comparison are required. The model can be evaluated using some evaluation indicators (such as accuracy, precision, recall, etc.) to determine whether its performance meets the requirements of the target model. Secondly, the prediction results of the model can be compared with the results of the target model to determine whether the difference between them is within an acceptable range. Finally, it is also possible to consider whether the characteristics and structure of the model meet the requirements of the target model, such as whether similar algorithms or feature engineering methods are used. Through comprehensive evaluation of these methods, it can be determined whether a model belongs to the target model.

[0039] In an exemplary embodiment, when it is determined that a model for processing business data is a target model based on business processing information, before generating N questions and corresponding N reference answers, the above also includes: when the business processing information includes exclusion information, determining that the model does not belong to the target model, wherein the exclusion information includes at least one of the following: a report for processing business data, a formula for processing business data, and indicator information for processing business data; when the business processing information does not include exclusion information and the business processing information includes a process for the model to process business data, obtaining M model information, wherein the M model information includes at least one of the following: information on inputting business data into the model, the process of the model processing business data, information on the processing result of the business data output by the model, the frequency of the model processing business data, the predictive performance of the model, and the process of generating the model; evaluating the feature score of each model information according to the target evaluation indicator to obtain M feature scores; and judging whether the model is the target model based on the M feature scores.

[0040] Optionally, judging whether the model is a target model based on M feature scores includes: calculating the sum of the M feature scores; comparing the sum and a target threshold, and judging whether the model is a target model based on the comparison result, wherein when the sum is greater than the target threshold, the model is determined to be a target model.

[0041] Optionally, when determining whether it is a target model through a scoring card model, such as Figure 3 As shown, the following steps are included:

[0042] S301, the scoring card model first performs an exclusion item check based on the acquired business processing information. If the business processing information includes exclusion information, it is determined that the business data processed is not a model. If the check passes, it proceeds to S302;

[0043] S302, determine whether the model is used for actual business decision-making, analysis, management, etc. If it is not used for actual business, directly determine that the model is not a model for processing business data, and if it is used for actual business, proceed to S303;

[0044] S303, scoring each of the model development method and approach, input, processing process, output, model predictability, model usage frequency and other characteristics (M model information);

[0045] S304, summarizing the final score (the sum of M feature scores) according to the feature scores and weights;

[0046] S305, based on the final score and the target threshold, the target threshold is divided into a lower limit and an upper limit. If it is below the lower limit, it is directly judged as a non-model; if it is above the upper limit, it is directly judged as a model; if it is between the lower limit and the upper limit, the output is that insufficient information cannot be judged.

[0047] This embodiment can accurately determine whether the model is the target model based on the M feature scores.

[0048] In an exemplary embodiment, when it is determined based on business processing information that a model for processing business data belongs to a target model, N questions and corresponding N reference answers are generated, including: when it is determined based on business processing information that a model for processing business data belongs to a target model, a verification request is received, wherein the verification request is used to request the credibility of the verification model as the target model; in response to the verification request, N questions and corresponding N reference answers are generated according to the processing characteristics of the model on the business data.

[0049] Optionally, in this embodiment, when N questions and corresponding N reference answers are generated by the interactive question-answering model, Figure 4 As shown, the specific steps include:

[0050] S401, when the user is not satisfied with the model result output by the scoring card model, the user can send a verification request to further supplement the information so as to make an accurate judgment.

[0051] S402, acquiring feature information and requirement document information for the scorecard model, sending them as known information to the large language model, and using the large language model to generate questions that need to be asked to the requirement personnel if further features of the scorecard model are needed;

[0052] S403, the large language model generates a question and performs similarity judgment with known information;

[0053] S404, output a question list (including N questions) and send it to the user for answering;

[0054] S405, obtaining the user's answer information content (including N answers);

[0055] S406, determining the relevance of the answer to the question, and if the relevance fails, regenerating the question and re-determining the answer;

[0056] S407: If passed, the N questions are fed back to the large language model to obtain the latest model features.

[0057] This embodiment combines interactive questions and answers to accurately determine the credibility of the model.

[0058] In an exemplary embodiment, the credibility of a model as a target model is determined based on N answers and N reference answers, including: comparing N answers and N reference answers corresponding to N questions to obtain N comparison results; when the number of target comparison results included in the N comparison results is greater than a preset number, determining that the credibility of the model as the target model is a preset credibility, wherein the target comparison result is used to indicate that the answer is consistent with the corresponding reference answer.

[0059] Optionally, the credibility of the target model is preset. Credibility is a measure of the accuracy and reliability of the prediction results of the target model. The preset credibility can be set based on different model types. It can be a score or a probability value. Credibility can be measured by indicators such as the accuracy, precision, recall, F1 score of the model, and can also be improved through methods such as cross-validation and model integration. Credibility is an important consideration when using models for prediction and decision-making, which can help determine whether the model can be trusted and how accurate the prediction is in different situations.

[0060] In an exemplary embodiment, before obtaining the business processing information, the method further includes: reading text content in the target business system, wherein the text content includes at least one of the following: business data, and requirement information for processing the business data; performing word frequency statistics processing on the text content to extract model requirement information from the text content, wherein the model requirement information is used to represent the model information required for processing the business data.

[0061] Optionally, this embodiment can perform word frequency statistics processing on the text content through a large language model. Figure 5 As shown, the process of performing word frequency statistics processing on text content by a large language model includes the following steps:

[0062] S501, calling a large language model;

[0063] S502, obtaining text content from the target business system through a large language model;

[0064] S503, obtaining information in the text according to the features required by the scoring card model, and performing word frequency statistics of relevant features to extract model requirement information from the text content;

[0065] S504, output the processing result to the scoring card model.

[0066] This embodiment uses a large language model to perform word frequency statistics on text content, and can accurately extract model requirement information.

[0067] The present invention will be described below in conjunction with specific embodiments:

[0068] like Figure 6 As shown, it is an overall flow chart of this specific embodiment, which includes the following steps:

[0069] S601, obtaining a requirement text or a requirement document from a target business system;

[0070] S602, using text recognition technology to perform demand text recognition and demand text reading, and storing the recognized information for processing business data into a database;

[0071] S603, using the large language model to obtain business processing information from the information used to process the business data;

[0072] S604, the scoring card model automatically scores according to the business processing information to determine whether it is a model.

[0073] S605, if the user needs to challenge the determination result, he needs to answer questions in the interactive question-and-answer session;

[0074] S606, the interactive question-answering model combines the model judgment features obtained by the large language, proposes supplementary questions, and conducts interactive question-answering with the user;

[0075] S607, after obtaining the supplementary features, the scoring card model is used again to perform scoring until the user has no objection;

[0076] S608, recording the final determination result and basis, and providing feedback on the final determination result and basis.

[0077] In summary, this embodiment forms a model automatic recognition device by integrating text recognition, large language model, scoring card model, interactive question and answer model, etc., replacing the manual judgment method, thereby improving the efficiency and accuracy of the recognition and judgment model.

[0078] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0079] In this embodiment, a device for determining the credibility based on a model is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0080] Figure 7 is a structural block diagram of a device for determining the credibility based on a model according to an embodiment of the present application, such as Figure 7 As shown, the device comprises:

[0081] A first acquisition module 72, used to acquire business processing information, wherein the business processing information is information extracted from a target business system for processing business data;

[0082] A first generating module 74 is used to generate N questions and corresponding N reference answers when it is determined that the model for processing the business data is the target model according to the business processing information, wherein the N questions are related to determining whether the model belongs to the target model;

[0083] A first display module 76, used to display the N questions above and receive N answers for answering the N questions above;

[0084] The first determination module 78 is used to determine the credibility of the above model as the above target model based on the N above answers and the N above reference answers.

[0085] In an exemplary embodiment, the device further includes: a second determination module, which is used to determine that the model for processing the business data is the target model according to the business processing information, and before generating N questions and corresponding N reference answers, if the business processing information includes exclusion information, determine that the model does not belong to the target model, wherein the exclusion information includes at least one of the following: a report for processing the business data, a formula for processing the business data, and indicator information for processing the business data; a second acquisition module, which is used to acquire M model information when the business processing information does not include the exclusion information and the business processing information includes a process of the model processing the business data, wherein the M model information includes at least one of the following: information of inputting the business data into the model, the process of the model processing the business data, information of the result of the model processing the business data, the frequency of the model processing the business data, the prediction performance of the model, and the process of generating the model; a first evaluation module, which is used to evaluate the feature score of each of the model information according to the target evaluation index to obtain M feature scores; and a first judgment module, which is used to judge whether the model is the target model based on the M feature scores.

[0086] In an exemplary embodiment, the first judgment module includes: a first calculation unit, used to calculate the sum of the M feature scores mentioned above; a first comparison unit, used to compare the sum and the target threshold, and judge whether the model is the target model based on the comparison result, wherein when the sum is greater than the target threshold, the model is determined to be the target model.

[0087] In an exemplary embodiment, the above-mentioned first generation module includes: a first receiving unit, used to receive a verification request when it is determined that the model for processing the above-mentioned business data belongs to the target model based on the above-mentioned business processing information, wherein the above-mentioned verification request is used to request verification of the credibility of the above-mentioned model as the above-mentioned target model; a first response unit, used to respond to the above-mentioned verification request, and generate the N above-mentioned questions and the corresponding N above-mentioned reference answers according to the processing characteristics of the above-mentioned business data by the above-mentioned model.

[0088] In an exemplary embodiment, the first determination module includes: a first comparison unit, used to compare the N answers and the N reference answers according to the N questions to obtain N comparison results; a first determination unit, used to determine that the credibility of the model as the target model is a preset credibility when the number of target comparison results included in the N comparison results is greater than a preset number, wherein the target comparison result is used to indicate that the answer is consistent with the corresponding reference answer.

[0089] In an exemplary embodiment, the above-mentioned device also includes: a first reading module, which is used to read the text content in the above-mentioned target business system before obtaining the business processing information, wherein the above-mentioned text content includes at least one of the following: the above-mentioned business data, and the requirement information for processing the above-mentioned business data; a first processing module, which is used to perform word frequency statistics processing on the above-mentioned text content to extract model requirement information from the above-mentioned text content, wherein the above-mentioned model requirement information is used to represent the model information required for processing the above-mentioned business data.

[0090] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0091] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0092] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0093] The embodiments of the present application also provide a computer program, which includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any one of the above method embodiments.

[0094] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0095] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0096] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0097] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0098] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0099] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0100] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining the credibility of a model, characterized in that: include: Acquiring business processing information, wherein the business processing information is information extracted from a target business system for processing business data; When it is determined according to the business processing information that the model for processing the business data is the target model, N questions and corresponding N reference answers are generated, wherein the N questions are related to determining whether the model belongs to the target model; Displaying N of the questions, and receiving N answers for answering the N of the questions; The credibility of the model as the target model is determined based on the N answers and the N reference answers.

2. The method according to claim 1, characterized in that When it is determined according to the business processing information that the model for processing the business data is the target model, before generating N questions and corresponding N reference answers, the method further includes: In the case where the business processing information includes exclusion information, determining that the model does not belong to the target model, wherein the exclusion information includes at least one of the following: a report for processing the business data, a formula for processing the business data, and indicator information for processing the business data; In a case where the business processing information does not include the exclusion information and the business processing information includes a process of the model processing the business data, M pieces of model information are acquired, wherein the M pieces of model information include at least one of the following: information of inputting the business data into the model, a process of the model processing the business data, information of a processing result of the business data output by the model, a frequency of the model processing the business data, a prediction performance of the model, and a process of generating the model; Evaluate the feature score of each model information according to the target evaluation index to obtain M feature scores; Determine whether the model is the target model based on the M feature scores.

3. The method according to claim 2, characterized in that Determining whether the model is the target model based on the M feature scores includes: Calculate the sum of the M feature scores; The sum value and a target threshold value are compared, and whether the model is the target model is determined according to the comparison result, wherein when the sum value is greater than the target threshold value, the model is determined to be the target model.

4. The method according to claim 1, characterized in that When it is determined according to the business processing information that the model for processing the business data belongs to the target model, N questions and corresponding N reference answers are generated, including: In a case where it is determined according to the business processing information that the model for processing the business data belongs to the target model, receiving a verification request, wherein the verification request is used to request verification of the credibility of the model as the target model; In response to the verification request, N questions and corresponding N reference answers are generated according to the processing characteristics of the business data by the model.

5. The method according to claim 1, characterized in that Determining the credibility of the model as the target model based on the N answers and the N reference answers includes: Compare the N answers to the N questions and the N reference answers to obtain N comparison results; When the number of target comparison results included in the N comparison results is greater than a preset number, the credibility of the model being the target model is determined to be the preset credibility, wherein the target comparison result is used to indicate that the answer is consistent with the corresponding reference answer.

6. The method according to claim 1, characterized in that Before acquiring the business processing information, the method further includes: Reading text content in the target business system, wherein the text content includes at least one of the following: the business data, and requirement information for processing the business data; The text content is processed by word frequency statistics to extract model requirement information from the text content, wherein the model requirement information is used to represent the model information required for processing the business data.

7. A device for determining credibility based on a model, characterized in that: include: A first acquisition module is used to acquire business processing information, wherein the business processing information is information extracted from a target business system for processing business data; A first generating module is used to generate N questions and corresponding N reference answers when it is determined that the model for processing the business data is a target model according to the business processing information, wherein the N questions are related to determining whether the model belongs to the target model; A first display module, used for displaying the N questions and receiving N answers for answering the N questions; The first determination module is used to determine the credibility of the model as the target model based on the N answers and the N reference answers.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 6 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 6 are implemented.

Images