Business intelligence effectiveness evaluation method, system, device and storage medium

Abstract

The application discloses a business intelligent effect evaluation method, system, device and storage medium, and relates to the technical field of artificial intelligence application. The method comprises the following steps: acquiring scene configuration information corresponding to a target business, wherein the scene configuration information comprises data acquisition rules corresponding to the target business and at least one target evaluation dimension; acquiring original business data corresponding to the target evaluation dimension from a business system associated with the target business according to the data acquisition rules, wherein the original business data comprises business baseline data corresponding to the target business before an intelligent model is applied and business running data of the target business after the intelligent model is applied; converting the original business data into net contribution data corresponding to the intelligent model and the target evaluation dimension based on an attribution analysis strategy corresponding to the target evaluation dimension, and generating an intelligent effect evaluation result of the target business according to the net contribution data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence application technology, and in particular to a method, system, device and storage medium for evaluating the effectiveness of business intelligence. Background Technology

[0002] With the rapid development of AI (Artificial Intelligence) technology, more and more business scenarios in the financial industry, such as marketing and operations, are beginning to introduce AI technologies (such as intelligent customer service and intelligent assistants) to improve business processing efficiency. However, as business scenarios gradually expand, the cost of applying AI is also increasing, and the value brought by applying AI varies in different business scenarios. At this point, in order to ensure the return on investment of AI technology applications, it is necessary to evaluate the effects of applying AI to business scenarios.

[0003] However, existing intelligent effectiveness evaluation methods have limited evaluation dimensions and lack comprehensiveness. Furthermore, the evaluation process suffers from long data collection cycles and low efficiency.

[0004] Therefore, there is an urgent need for an automated, business-scenario-oriented method for evaluating the effectiveness of applications, in order to improve the efficiency and comprehensiveness of intelligent effectiveness evaluation in business scenarios. Summary of the Invention

[0005] The main purpose of this application is to provide a method, system, device and storage medium for evaluating the effectiveness of business intelligence, aiming to solve the technical problem of how to provide an automated application effectiveness evaluation method for business scenarios, so as to improve the evaluation efficiency and comprehensiveness of the effectiveness evaluation of business scenario intelligence.

[0006] To achieve the above objectives, this application proposes a method for evaluating the effectiveness of business intelligence, which includes: Obtain the scenario configuration information corresponding to the target business, wherein the scenario configuration information includes the data collection rules and at least one target evaluation dimension corresponding to the target business; According to the data collection rules, the original business data corresponding to the target evaluation dimension is obtained from the business system associated with the target business. The original business data includes the business baseline data of the target business before the application of the intelligent model and the business operation data of the target business after the application of the intelligent model. Based on the attribution analysis strategy corresponding to the target evaluation dimension, the original business data is converted into net contribution data corresponding to the intelligent model and the target evaluation dimension, and the intelligent effectiveness evaluation result of the target business is generated based on the net contribution data.

[0007] In addition, to achieve the above objectives, this application also proposes a business intelligence effectiveness evaluation system, which includes: a scenario configuration system, a business system, and an evaluation system; The evaluation system is used to obtain scenario configuration information corresponding to the target business from the scenario configuration system. The scenario configuration information includes data collection rules and at least one target evaluation dimension corresponding to the target business. The evaluation system is also used to obtain the original business data corresponding to the target evaluation dimension from the business system associated with the target business according to the data collection rules. The original business data includes the business baseline data corresponding to the target business before the application of the intelligent model and the business operation data of the target business after the application of the intelligent model. The evaluation system is also used to convert the original business data into net contribution data corresponding to the intelligent model and the target evaluation dimension based on the attribution analysis strategy corresponding to the target evaluation dimension. The evaluation system is also used to generate intelligent performance evaluation results for the target business based on preset quantitative rules and the net contribution data.

[0008] In addition, to achieve the above objectives, this application also proposes a business intelligence effectiveness evaluation device, which includes: a memory, a processor, and a business intelligence effectiveness evaluation program stored on the memory and capable of running on the processor. The business intelligence effectiveness evaluation program is configured to implement the steps of the business intelligence effectiveness evaluation method described above.

[0009] In addition, to achieve the above objectives, this application also provides a storage medium storing a program for implementing a business intelligence effectiveness evaluation method, wherein the program for implementing the business intelligence effectiveness evaluation method is executed by a processor to implement the steps of the business intelligence effectiveness evaluation method as described above.

[0010] This application provides a method, system, device, and storage medium for evaluating the effectiveness of business intelligence. The method includes: acquiring scenario configuration information corresponding to a target business, the scenario configuration information including data collection rules and at least one target evaluation dimension; acquiring raw business data corresponding to the target evaluation dimension from the business system associated with the target business according to the data collection rules, the raw business data including business baseline data before the application of the intelligent model and business operation data after the application of the intelligent model; converting the raw business data into net contribution data corresponding to the intelligent model and the target evaluation dimension based on the attribution analysis strategy corresponding to the target evaluation dimension, and generating an intelligent effectiveness evaluation result for the target business based on the net contribution data.

[0011] This application pre-configures data collection rules and evaluation dimensions for the target business to be evaluated, enabling automatic connection to the business system to obtain the original business data corresponding to the target evaluation dimensions when intelligent effectiveness evaluation is required. Then, based on the configured target evaluation dimensions, an intelligent model and corresponding revenue data for each dimension are automatically generated. Finally, the multi-dimensional revenue data are aggregated into the final model application effectiveness evaluation result.

[0012] Therefore, when it is necessary to introduce new intelligent business assessment, this application can automatically assess the effectiveness of intelligent business assessment by simply pre-configuring the scenario configuration information as needed, through the process of "obtaining scenario configuration information → automatically collecting business data → comparing model benefit data → comprehensive assessment of model application". This not only improves assessment efficiency but also enhances the comprehensiveness of the assessment method. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating the first embodiment of the intelligent business effectiveness evaluation method of this application; Figure 2 This is a flowchart illustrating the second embodiment of the intelligent business effectiveness evaluation method of this application; Figure 3 A simplified flowchart illustrating a method for evaluating the effectiveness of business intelligence is provided. Figure 4 This is a schematic diagram of the module structure of the business intelligence effectiveness evaluation system according to an embodiment of this application; Figure 5 This is a schematic diagram of the hardware operating environment involved in the business intelligence effectiveness evaluation method in this application embodiment.

[0016] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0017] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application. Furthermore, all actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection regulations of the country where the application is located and with authorization from the owner of the corresponding device.

[0018] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0019] The main solution of this application is as follows: First, obtain scenario configuration information corresponding to the target business. This scenario configuration information includes data collection rules and at least one target evaluation dimension. Second, based on the data collection rules, obtain the original business data corresponding to the target evaluation dimension from the business system associated with the target business. This original business data includes the business baseline data of the target business before applying the intelligent model and the business operation data of the target business after applying the intelligent model. Third, based on the attribution analysis strategy corresponding to the target evaluation dimension, convert the original business data into net contribution data corresponding to the intelligent model and the target evaluation dimension, and generate the intelligent effectiveness evaluation result of the target business based on the net contribution data.

[0020] Currently, the measurement dimensions of intelligent effectiveness evaluation methods typically only assess the effectiveness of intelligentization in a single business scenario. Therefore, existing evaluation methods have a single evaluation dimension and poor comprehensiveness. In addition, the effectiveness evaluation process after intelligentization of business scenarios requires manual collection of data from multiple aspects, and different business intelligent effectiveness evaluations also require the collection of different data. Therefore, the evaluation process suffers from long data collection cycles and low evaluation efficiency.

[0021] To address this issue, this application proposes a standardized method for quantifying and evaluating the real value of intelligent models to businesses in AI application scenarios within the financial industry. This method utilizes a configurable rule system, automated data collection, and multi-dimensional attribution analysis. The application pre-configures data collection rules and evaluation dimensions for the target business to be evaluated, enabling automatic access to the business system to obtain the original business data corresponding to the target evaluation dimensions when intelligent effectiveness evaluation is required. Then, based on the configured target evaluation dimensions, the application automatically generates the intelligent model and corresponding revenue data for each dimension. Finally, the multi-dimensional revenue data is aggregated into the final model application effectiveness evaluation result.

[0022] Therefore, when it is necessary to introduce new intelligent business assessment, this application can automatically assess the effectiveness of intelligent business assessment by simply pre-configuring the scenario configuration information as needed, through the process of "obtaining scenario configuration information → automatically collecting business data → comparing model benefit data → comprehensive assessment of model application". This not only improves assessment efficiency but also enhances the comprehensiveness of the assessment method.

[0023] It should be noted that the executing entity in this embodiment can be a business intelligence effectiveness evaluation system, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a business intelligence effectiveness evaluation device capable of performing the above functions. This embodiment does not specifically limit it in this way. The following uses the evaluation system in the business intelligence effectiveness evaluation system as the executing entity as an example to describe this embodiment and the following embodiments.

[0024] Based on this, embodiments of this application provide a method for evaluating the effectiveness of business intelligence, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the intelligent business effectiveness evaluation method of this application.

[0025] In this embodiment, the method for evaluating the effectiveness of business intelligence includes steps S10 to S30: Step S10: Obtain the scenario configuration information corresponding to the target service. The scenario configuration information includes the data collection rules and at least one target evaluation dimension corresponding to the target service. It should be understood that the aforementioned target business can refer to a financial business scenario that applies intelligent models. In this embodiment, the target business must meet two core judgment conditions: ① a relevant application has been created in advance in the business intelligence effectiveness evaluation system so that the evaluation system can measure the effectiveness of intelligent applications; ② it consumes computing power card resources. In this embodiment, the target business can refer to a business scenario that applies traditional AI (machine learning, deep learning models), large models (large language models, multimodal large models), and other intelligent technologies.

[0026] Among them, intelligent models can be AI models that provide intelligent capabilities for target businesses, including traditional machine learning models (XGBoost, SVM, etc.), traditional deep learning models (Bert, Resnet, etc.), large language models (Qwen, Llama, etc.) or multimodal large models (Qwen-VL, Stable Diffusion, etc.).

[0027] It is easy to understand that the aforementioned scenario configuration information can be a set of basic parameters pre-configured for the target business in the scenario configuration system of the business intelligence effectiveness evaluation system to support automated evaluation. This scenario configuration information can be the core of realizing the scalability of the evaluation method, and can include data collection rules and at least one target evaluation dimension. Therefore, in this embodiment, the evaluation system retrieves the scenario configuration information corresponding to the target business from the scenario configuration system through database queries, interface calls, configuration file reading, etc.

[0028] The aforementioned data collection rules can be pre-configured standardized rules for obtaining evaluation data from business systems, and may include data connection methods (JDBC (Java Database Connectivity) data access interface / API (Application Programming Interface) interface / message queue), request protocol format, data return format, collection cycle, interface address, request parameters, etc.

[0029] The aforementioned target evaluation dimensions can be pre-selected evaluation dimensions used to measure the effectiveness of AI application in various business scenarios. In this embodiment, the target evaluation dimensions can include 7 standardized dimensions, divided into two main perspectives: ① Bank perspective (efficiency improvement, procurement substitution, risk control, business growth, industry competitiveness); ② Customer perspective (customer experience, customer benefits). For example, the retail marketing scenario can select the "business growth + efficiency improvement" dimension, and the risk control scenario can select the "risk control + procurement substitution" dimension.

[0030] Step S20: According to the data collection rules, obtain the original business data corresponding to the target evaluation dimension from the business system associated with the target business. The original business data includes the business baseline data corresponding to the target business before the application of the intelligent model and the business operation data of the target business after the application of the intelligent model. It is understood that the aforementioned business system may be a heterogeneous information system that is associated with the target business and carries out business operations in the business intelligence effectiveness evaluation system. In this embodiment, the business system includes, but is not limited to, customer service system, credit system, marketing platform, work order system, risk control system, human resources working hour database, etc., which can be the original data source for subsequent effectiveness evaluation.

[0031] It should be noted that the aforementioned original business data can be the full set of basic data obtained from the business system for evaluating the effectiveness of intelligent business applications. It is mainly divided into two categories: ① Business baseline data, which can be the historical benchmark data of the corresponding evaluation dimensions before the target business applies the intelligent model (such as the manual processing hours per transaction, historical asset loss rate, average procurement cost, etc. in the 12 months before AI goes live); ② Business operation data, which can be the actual business data within the statistical period after the target business applies the intelligent model (such as the processing hours per transaction, actual asset loss rate, actual procurement cost, business volume, etc. after AI goes live).

[0032] Therefore, the evaluation system can strictly follow the data collection rules and automatically retrieve the original business data corresponding to the target evaluation dimensions from the related business systems through methods such as scheduled API calls and real-time subscription to message queues.

[0033] It is easy to understand that different business systems exist in a multi-source heterogeneous manner. In this case, to achieve standardized and automated data collection from different business systems and for different evaluation indicators, and to improve data collection efficiency and accuracy, this embodiment can solve the data interoperability problem of multi-source heterogeneous business systems through three types of standardized request protocol formats. Therefore, in a feasible implementation, step S20 in this embodiment may include steps A1~A4: Step A1: Determine the request protocol format according to the data collection rules. The request protocol format includes at least one of the following: full-time manpower equivalent calculation format, before-and-after numerical comparison format, or monetary numerical format. It is important to understand that the aforementioned request protocol format can be a pre-agreed standardized message format when the evaluation system initiates a data query request to the business system. The evaluation system can match the standardized request protocol format corresponding to the target business according to the format type corresponding to the scenario indicators in the data collection rules. This request protocol format can be the core for achieving unified integration of multiple heterogeneous business systems corresponding to the target business. In this embodiment, it can include three types of standardized formats: ① Full-time manpower equivalent calculation format (Format 1_FTE (Full-Time Equivalent)); ② Before and after value comparison format (Format 2_Value); ③ Amount value format (Format 2_Amount). The evaluation system can parse the scenario configuration information of the target business, read the format type configuration corresponding to each scenario indicator that needs to be obtained, and each request protocol can include request header specifications, request body structure, required fields, return message structure, etc.

[0034] Among them, the full-time human resource equivalent calculation format can be a protocol format specifically used for data collection of human efficiency improvement dimensions. It can stipulate that the messages returned by the business system must include mandatory fields such as the total business processing volume during the statistical period, FTE value, single processing time before and after intelligentization, and total business volume.

[0035] The format for comparing values ​​before and after can be a protocol format used for data collection in dimensions such as customer experience and business growth. It requires that the messages returned by the business system must include mandatory fields such as the baseline value before intelligentization and the actual value during the statistical period after intelligentization.

[0036] The monetary value format can be a protocol format used for data collection in dimensions such as procurement substitution and risk control. The message returned by the business system must include mandatory fields such as the monetary value during the statistical period after intelligentization and the baseline amount before intelligentization.

[0037] Step A2: Obtain the scene identifier and scene metrics corresponding to the target service; It's important to understand that the aforementioned scenario identifier can be a unique, non-repeatable code pre-generated for each target business, used to uniquely identify the target business during interface interactions. Scenario metrics, on the other hand, can be unique codes for specific evaluation metrics selected by the target business and required to be collected from the business system. For example, the evaluation system can extract the unique scenario identifier for the target business, as well as a list of all scenario metrics to be collected and their corresponding metric codes, from the scenario configuration information. Furthermore, the evaluation system can filter the list of scenario metrics, retaining only those metrics that the administrator has configured to "support automatic system integration," and removing metrics that only support manual entry.

[0038] Step A3: If the scenario indicators meet the preset docking requirements, assemble a query request message based on the request protocol format. The query request message carries the scenario identifier and the scenario indicators. It is easy to understand that the aforementioned preset docking requirements can be pre-set admission conditions for scenario indicators to achieve automatic system docking. For example, in this embodiment, the preset docking requirements may include: ① the administrator has completed the docking configuration of the scenario-indicator in the system; ② the request protocol format corresponding to the indicator has been configured; ③ the valid response address of the business system has been configured. At this time, the scenario indicator can be determined to meet the docking requirements only if the evaluation system detects that all three conditions of the preset docking requirements are met simultaneously. All indicators that meet the requirements can be included in the subsequent automatic data collection scope; indicators that do not meet the requirements can be marked as "manually filled in" and a to-do task notification can be generated to notify the scenario manager. That is, this embodiment can flexibly control the docking method of indicators, support a mixed mode of automatic docking and manual filling, and adapt to the construction status of different business systems.

[0039] It is understood that the aforementioned query request message can be a standardized HTTP request message assembled by the evaluation system according to the request protocol format, used to initiate data queries to the business system. Furthermore, in this embodiment, the query request message must carry a scenario identifier and scenario indicators to facilitate the business system's identification of the correct query object and content. Therefore, the evaluation system can encapsulate the scenario identifier, scenario indicators, request parameters, and other content into a standardized query request message according to the request protocol format specifications.

[0040] For example, the message encapsulation rules for query request messages can strictly follow two core rules: ① The message must contain both the scenario identifier and all eligible scenario metrics under that scenario; ② Only metrics configured to "support system integration" are encapsulated. For example, for scenario identifier A, an FTE format request message is assembled, which may contain required fields such as scenario identifier, metric code, statistical period, and requester identifier.

[0041] Step A4: Send the query request message to the response address of the business system associated with the target business, and receive the original business data returned by the business system in accordance with the request protocol format.

[0042] Understandably, the aforementioned response address can be an HTTP interface address provided by the business system for receiving query requests from the evaluation system, which can be pre-configured in the system by the scenario technical lead or system administrator. In this case, the evaluation system can send the assembled query request messages in batches to the corresponding business system's response address via HTTP / HTTPS protocol. Upon receiving the request messages, the business system can parse the scenario identifier and scenario metrics, extract the corresponding raw business data from the corresponding business database, and generate and return a response message according to the return structure agreed upon in the request protocol format. Then, the evaluation system receives the response messages returned by the business system, extracts the raw business data from the messages, and stores it in a structured manner according to scenario and metrics, completing automated data collection.

[0043] In this embodiment, in response to the problems of heterogeneous architecture and inconsistent data formats of various business systems in the existing financial industry, the inability of existing technologies to achieve standardized automatic connection of multiple systems, and the need to collect data through manual entry, which is labor-intensive, time-consuming, and prone to human error, this embodiment can cover the data collection needs of all evaluation dimensions through three types of standardized request protocol formats, realize unified connection of different heterogeneous business systems, thereby avoiding the need to develop separate interfaces for each business system and significantly reducing adaptation costs. Meanwhile, this embodiment can achieve precise matching of request and return data through unique scene identifiers and indicator codes, completely solving the problem of data misalignment, ensuring the accuracy of the original data, and providing a reliable data foundation for subsequent evaluation.

[0044] Understandably, this embodiment can further add a pre-data verification step to prevent abnormal data such as misaligned identifiers or missing fields returned by the business system from directly entering the subsequent calculation stage, which could lead to distorted evaluation results, calculation errors, or even completely erroneous evaluation conclusions, thus ensuring the accuracy of subsequent evaluations. Therefore, in a feasible implementation, this embodiment may include steps B1~B2 before step S30: Step B1: Perform integrity verification on the original business data. The integrity verification includes identifier consistency verification and null value verification. It should be noted that the above-mentioned integrity verification refers to the operation by which the evaluation system checks the consistency and integrity of the original business data item by item according to preset verification rules before performing subsequent net contribution data calculation on the original business data, and determines whether the data is compliant.

[0045] Among them, the identifier consistency check can be used to verify whether the scenario identifier and indicator code in the message returned by the business system are completely consistent with the scenario identifier and scenario indicator in the request message, so as to avoid data misalignment; while the null value check can be used to verify whether there are null values ​​in the required fields in the message. For example, the required fields can be core fields such as the total business volume corresponding to the FTE indicator and the intelligent value corresponding to the format 2 indicator, so as to avoid calculation errors caused by missing key data.

[0046] In essence, the evaluation system first extracts the scenario identifier and scenario indicator list from the request message and compares them one by one with the scenario identifier and indicator code in the response message returned by the business system. If the system detects that the scenario identifiers must be completely identical and the returned indicator codes must perfectly match the requested indicator codes, with no difference in quantity or code, the identifier consistency check is considered passed. Otherwise, the check fails, the error type is recorded as "Scenario / Indicator Identifier Inconsistency," and the error details are recorded.

[0047] Then, the evaluation system can read the system's pre-set list of mandatory validation fields and perform non-empty validation on the required fields in the response message. At this time, the evaluation system can perform the following checks: ① The FTE indicator must return the total business processing volume and the single transaction hours before and after intelligentization, and cannot be empty; ② Format 2 numerical indicators must return the actual value after intelligentization, and cannot be empty; ③ Amount indicators must return the corresponding amount value, and cannot be empty. At this time, if all required fields are not empty, the empty value validation passes; otherwise, the validation fails, the error type is recorded as "Required field empty", and the name of the missing field is recorded.

[0048] Step B2: If the integrity verification fails, an error message is generated and sent to the management terminal.

[0049] It's important to understand that if both the identifier consistency check and the null value check pass, the integrity check passes, and the system stores the original business data in the measurement data table for subsequent net contribution data calculation. If either check fails, the integrity check fails.

[0050] At this point, the aforementioned error message can be an automatically generated message from the system when the verification fails, which may include the scenario identifier, error type, error details, and error time. The management terminal can be a front-end terminal used by system administrators or scenario managers to receive error messages from the evaluation system, view error details, and handle abnormal issues; it may include a PC web page or a mobile application.

[0051] Therefore, when the integrity check fails, the evaluation system can automatically generate an error message and send it to the corresponding management terminal via system message, email, or internal communication tools; when the integrity check fails, the evaluation system can perform subsequent net contribution data calculation.

[0052] For example, if the integrity check fails, the evaluation system may perform the following operations: 1. Intercept the abnormal data, do not save it to the official measurement data table, mark it as "abnormal data", and save it to the abnormal data table; 2. Automatically generate error message information, including scene name, scene identifier, error type, error details, error time, and handling suggestions; 3. Send exception notifications to the management terminals of the relevant scenario manager and system administrator via system to-do list, internal communication messages, and email; 4. Mark this scenario as "Verification Failed" on the system integration interface, display error details, and allow users to manually trigger re-collection.

[0053] In this embodiment, after the evaluation system receives the original business data returned by the business system, before storing it in the measurement data table and starting the net contribution data calculation, it can automatically trigger a standardized integrity verification process to intercept abnormal data with misaligned identifiers or missing fields in advance. This prevents erroneous data from entering the calculation process from the source, avoids the distortion of evaluation results caused by "garbage in, garbage out", and ensures the accuracy and credibility of the evaluation results. Furthermore, this embodiment can automatically generate anomaly alerts and push them to the responsible person in real time when the verification fails, enabling the discovery of data anomalies within seconds and shortening the problem handling cycle from several days to several hours. This allows for rapid discovery and handling of problems and ensures the stable operation of the evaluation process. At the same time, the list of mandatory verification fields during the verification process can be flexibly adjusted through system configuration. When adding new indicators, the corresponding verification rules can be quickly configured to adapt to the verification needs of different scenarios and indicators.

[0054] Step S30: Based on the attribution analysis strategy corresponding to the target evaluation dimension, the original business data is converted into net contribution data corresponding to the intelligent model and the target evaluation dimension, and the intelligent effectiveness evaluation result of the target business is generated based on the net contribution data.

[0055] It is important to understand that the above attribution analysis strategy can be a standardized algorithmic logic corresponding to each objective evaluation dimension, used to extract the value contribution brought only by the intelligent model from business data affected by multiple factors, in order to eliminate interference from non-AI factors such as human, market, and environment.

[0056] Understandably, the net contribution data mentioned above can be calculated through attribution analysis strategies and is the real value data brought by the intelligent model to the target business in the corresponding evaluation dimension. It excludes the interference of all non-AI factors and is the core indicator for evaluating the effectiveness of AI in this embodiment.

[0057] At this point, the aforementioned intelligent effectiveness evaluation results can be the final output generated by the evaluation system through summarizing, classifying, quantifying, and visualizing the net contribution data of each dimension, which comprehensively reflects the value of AI applications in the target business. This output may include detailed dimensional value, quantified economic benefits, a list of high-value scenarios, trend analysis, rankings, and other content.

[0058] In summary, this embodiment addresses the problem that existing intelligent effectiveness evaluation methods have a single evaluation dimension, can only evaluate based on a single dimension such as human efficiency, cannot cover the diversified value of AI scenarios in the financial industry, have extremely poor adaptability, and cannot meet the evaluation needs of different business scenarios such as marketing, risk control, operations, and customer service. This embodiment can comprehensively cover all value types of AI scenarios in the financial industry through a dual-perspective, seven-dimensional evaluation system. The evaluation dimensions can be flexibly selected according to the characteristics of different business scenarios, adapting to all categories of AI scenarios such as marketing, risk control, operations, and customer service, thus solving the limitations of single-dimensional evaluation. Meanwhile, in response to the problem that existing assessment methods rely heavily on manual data collection, require manual intervention to collect data from different systems for different business scenarios, involve many communication steps, have a large workload, long data collection cycles, and extremely low assessment efficiency, and cannot support large-scale assessments of hundreds of AI scenarios, this embodiment can achieve automated data connection between multiple heterogeneous business systems through pre-configured data collection rules, without manual intervention, shortening the data collection cycle from several weeks to several hours, improving assessment efficiency, and supporting normalized and large-scale assessments of hundreds of AI scenarios across the bank; Finally, addressing the shortcomings of existing evaluation methods—namely, the lack of a standardized configuration system, the need for separate development and adaptation for new evaluation scenarios, poor scalability, and inability to quickly respond to the evaluation needs of new AI scenarios—this embodiment addresses these shortcomings by standardizing scenario configuration information. This allows new evaluation scenarios to be integrated into the system with only basic configuration, eliminating the need for separate development and modification. Therefore, this embodiment improves the comprehensiveness of the evaluation method, enabling rapid response to the evaluation needs of new AI scenarios and adapting to the rapid iteration pace of AI applications in the financial industry.

[0059] This embodiment provides a method for evaluating the effectiveness of business intelligence. The method includes: acquiring scenario configuration information corresponding to a target business, the scenario configuration information including data collection rules and at least one target evaluation dimension corresponding to the target business; acquiring original business data corresponding to the target evaluation dimension from the business system associated with the target business according to the data collection rules, the original business data including business baseline data corresponding to the target business before applying the intelligent model and business operation data of the target business after applying the intelligent model; converting the original business data into net contribution data corresponding to the intelligent model and the target evaluation dimension based on the attribution analysis strategy corresponding to the target evaluation dimension, and generating an intelligent effectiveness evaluation result for the target business based on the net contribution data.

[0060] This embodiment pre-configures data collection rules and evaluation dimensions for the target business to be evaluated, corresponding to the business scenario. When intelligent effectiveness evaluation is required, it automatically connects to the business system based on the pre-configured data collection rules to obtain the original business data corresponding to the target evaluation dimensions. Then, based on the configured target evaluation dimensions, it automatically generates an intelligent model and the corresponding revenue data for each dimension. Finally, it summarizes the multi-dimensional revenue data into the final model application effectiveness evaluation result.

[0061] Therefore, when it is necessary to introduce new business intelligence evaluation, this embodiment can automatically evaluate the effectiveness of business intelligence by pre-configuring the scenario configuration information as needed, through the above process of "obtaining scenario configuration information → automatically collecting business data → comparing model benefit data → comprehensive evaluation of model application". This not only improves the evaluation efficiency, but also enhances the comprehensiveness of the evaluation method.

[0062] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the first embodiment described above can be referred to the above description, and will not be repeated hereafter.

[0063] Based on the first embodiment, please refer to Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the business intelligence effectiveness evaluation method of this application. In this embodiment, step S30 further includes steps S31 to S32: Step S31: Preprocess the original business data based on the attribution analysis strategy corresponding to the target evaluation dimension to obtain the evaluation parameters corresponding to the target evaluation dimension. It should be noted that the assessment system can invoke the attribution analysis strategy corresponding to the target assessment dimension and read the pre-defined data processing rules in the strategy. In this case, the aforementioned preprocessing can refer to the process of standardizing and processing the raw business data by invoking the data processing rules in the attribution analysis strategy to output compliance assessment parameters.

[0064] Accordingly, the evaluation parameters can be standardized parameters obtained after preprocessing. Some evaluation parameters can be further substituted into preset evaluation expressions corresponding to the target evaluation dimension for calculation, generating corresponding net contribution data. It is easy to understand that in this embodiment, different evaluation dimensions can correspond to different evaluation parameters. For example, the evaluation parameter for the human resource improvement dimension can be the equivalent value of full-time human resources, and the risk control dimension can be the model risk control ratio.

[0065] Step S32: Generate the net contribution data corresponding to the target evaluation dimension of the intelligent model based on the preset evaluation expression and the evaluation parameters.

[0066] Understandably, the aforementioned preset evaluation expressions can be standardized calculation formulas corresponding to each target evaluation dimension, pre-defined to calculate the net contribution data of the intelligent model in that target evaluation dimension using evaluation parameters. That is, the evaluation system can substitute the pre-processed evaluation parameters into the pre-configured preset evaluation expressions for each target evaluation dimension to perform the corresponding mathematical operations, thereby obtaining the net contribution data brought by the intelligent model in that dimension. Finally, the evaluation system can associate and store the calculated net contribution data for each target evaluation dimension with its corresponding scenario identifier, dimension, and indicator to complete the analysis of the application effectiveness of the target business in each target evaluation dimension.

[0067] For example, assuming that in the procurement substitution dimension, the preset evaluation expression is "Net contribution data = actual procurement quantity × (single procurement cost before intelligentization - single procurement cost after intelligentization)", the evaluation system can substitute the pre-processed procurement quantity and the procurement costs before and after into this formula to obtain the procurement cost savings brought by the intelligent model in the procurement substitution dimension, that is, the net contribution data in the procurement substitution dimension.

[0068] In this embodiment, the calculation criteria for indicators across all business scenarios are standardized by using pre-configured evaluation expressions for each dimension. This allows for horizontal comparison and vertical traceability of evaluation results from different business scenarios and departments, ensuring the rigor and credibility of the evaluation results. At the same time, this embodiment can decouple the calculation logic of different dimensions through a two-stage standardized process of "preprocessing + expression calculation". Adding a new evaluation dimension only requires configuring the corresponding preprocessing rules and evaluation expressions, without reconstructing the core process, making it highly scalable.

[0069] In one feasible implementation, when the target evaluation dimension is the human resource improvement dimension, the evaluation parameters include the equivalent value of full-time human resources. In this embodiment, step S31 may include steps C1~C2: Step C1: When the target evaluation dimension is the human resource improvement dimension, determine the total saved working hours within the statistical period based on the business baseline data and the business operation data. It is important to understand that the aforementioned human resource improvement dimension can be used as an evaluation dimension to measure the value brought by intelligent models in replacing manual labor and improving operational efficiency. In this case, the aforementioned Full-Time Equivalent (FTE) can be a standardized unit for measuring the human resource savings brought by intelligent models; one FTE represents the workload completed by one full-time employee during normal working hours. Therefore, this embodiment can use this indicator to convert the workload saved by intelligent models into a standardized number of full-time employees, thus serving as the core evaluation parameter for the human resource improvement dimension.

[0070] It is easy to understand that the aforementioned statistical period can be the time range corresponding to this evaluation, usually a calendar month, a calendar quarter, or a calendar year, which is the time boundary for calculating the total saved working hours. Accordingly, the aforementioned total saved working hours can represent the total amount of working hours saved by the intelligent model replacing manual labor in completing business within the statistical period. In this embodiment, the method for obtaining the total saved working hours can be expressed as: Total saved working hours = Total business processing volume × Working hours per business transaction before intelligentization (obtained from business baseline data) - Working hours per business transaction after intelligentization (obtained from business operation data)). Therefore, the evaluation system can automatically generate the total saved working hours within the statistical period by calculating the formula based on the working hours per transaction in the business baseline data, the total business volume in the business operation data, and the actual working hours per transaction.

[0071] For example, the evaluation system can extract the average manual processing time per transaction before intelligentization (denoted as T0, unit: hours / transaction) from the business baseline data; and extract the average processing time per transaction after intelligentization (denoted as T1, unit: hours / transaction) and the total number of transactions processed within the statistical period (denoted as N, unit: transactions) from the business operation data.

[0072] Then, the evaluation system can calculate the total saved working hours using the formula: Total saved working hours S = N × (T0 - T1). For example, in a certain intelligent customer service scenario, before the intelligent model went online, the manual processing time per transaction was 0.2 hours, and after the intelligent model went online, the processing time per transaction was 0.05 hours, and the monthly processing volume was 10,000 transactions. Therefore, the total saved working hours = 10,000 × (0.2 - 0.05) = 1,500 hours.

[0073] Step C2: The total saved working hours are effectively corrected using the scenario calibration factor corresponding to the target service to obtain the corrected working hours; It should be understood that the above-mentioned scenario calibration factor can be a parameter used to correct the effectiveness of the total saved working hours, and a parameter used to eliminate invalid business volume that has not actually taken effect by the intelligent model. This scenario calibration factor can ensure that the calculated working hour savings are the real effective savings brought about by the intelligent model, and can include two categories: technical factors and business factors.

[0074] The technical factors reflect the effectiveness of the intelligent model and can include model performance metrics such as recognition accuracy and intent recognition accuracy. The business factors reflect the business implementation effectiveness of the intelligent model's results and can include business application metrics such as adoption rate, connection rate, and conversion rate. The adoption rate typically refers to the proportion of user-submitted questions or suggestions that are accepted or adopted on a particular system or platform. It is often used to measure user engagement, problem-solving efficiency, or the value of suggestions. The connection rate can refer to the proportion of successfully connected requests (such as phone calls, messages, service requests, etc.) to the total number of requests within a certain period. Simply put, it is the ratio of the number of successful connections to the total number of requests. The conversion rate can convert a certain indicator or value into another indicator or value according to a specific rule or proportion. It should be noted that the data collection for the scenario calibration factors of each objective evaluation dimension can be obtained through pre-set data points in the business system where the corresponding intelligent application scenario is located, when statistically analyzing business volume.

[0075] At this point, the corrected working time can refer to the effective working time that truly benefits the intelligent model, determined after the total saved working time has been effectively corrected using scenario calibration factors. That is, the evaluation system can use scenario calibration factors to effectively convert the total saved working time, eliminating working time corresponding to invalid business volume, and obtaining the corrected working time that ensures the accuracy of FTE calculation. For example, assume that the evaluation system extracts the scenario calibration factors corresponding to the target business from the business operation data as follows: a technical factor (e.g., AI intent recognition accuracy rate of 95%, denoted as a=0.95) and a business factor (e.g., AI result adoption rate of 90%, denoted as b=0.9). It is understood that in this embodiment, the scenario calibration factors can be flexibly configured according to the characteristics of different scenarios. For example, the recognition accuracy rate can be configured as a technical factor for OCR (Optical Character Recognition) scenarios, and the lead conversion rate can be configured as a business factor for marketing scenarios, thereby adapting to the intelligent evaluation needs of different types of business scenarios.

[0076] Then, the evaluation system can effectively correct the total saved working hours using a scenario calibration factor, calculated as: Corrected working hours S' = S × a × b. For example, in the above intelligent customer service case, the corrected working hours = 1500 × 0.95 × 0.9 = 1282.5 hours. At this point, the evaluation system can eliminate the working hours corresponding to invalid business volume in the intelligent customer service scenario where the intelligent model misidentifies or the results are not adopted, ensuring that the time savings are the portion where the intelligent model is truly effective.

[0077] Step C3: Calculate the ratio between the standard working hours of full-time employees and the modified working hours to obtain the equivalent value of the full-time workforce corresponding to the human resource improvement dimension.

[0078] The standard working hours for full-time employees can be a pre-set legal standard working hour for a full-time employee throughout the year. For example, the standard working hours for full-time employees can be pre-set to a fixed value of 2000 hours / year (250 working days × 8 hours / day), or it can be further converted into monthly standard working hours when statistics are compiled monthly. In this case, the evaluation system can divide the adjusted working hours by the standard working hours for full-time employees to calculate the full-time equivalent value (FTE).

[0079] For example, since the above-mentioned revised working hours are monthly estimated working hours, after the evaluation system obtains the standard working hours of full-time employees corresponding to the statistical period, it generates the monthly standard working hours = 2000 hours / year ÷ 12 ≈ 166.67 hours / month when performing monthly statistics.

[0080] Then, the evaluation system can perform ratio calculations to calculate the FTE value: FTE = Corrected working hours S' ÷ Standard working hours for the statistical period. For example, in the above intelligent customer service case, the monthly FTE = 1282.5 ÷ 166.67 ≈ 7.7, which means that this AI scenario saves the workload of approximately 7.7 full-time workers per month.

[0081] Subsequently, the evaluation system can use the calculated FTE value as the core evaluation parameter for the human resource improvement dimension, substitute it into the preset evaluation expression, and calculate the net contribution data for that dimension (FTE × corresponding annual salary in the industry).

[0082] Therefore, this embodiment can solve the problems of imprecise labor efficiency measurement, inconsistent standards, and inability to eliminate invalid business volume in the prior art through a standardized three-step method of "total working hours calculation - validity correction - FTE conversion". It can achieve a unified standard for labor efficiency calculation in all business scenarios such as operation, customer service, and approval, and make the results of different scenarios comparable horizontally and summarized vertically, with strong adaptability.

[0083] Furthermore, this embodiment can effectively eliminate the interference of invalid business volume through dual correction of technical factors and business factors, ensuring that the calculated time savings are the real effective value brought by AI, and solving the problem of overestimation of the value of human efficiency in the current system.

[0084] It is easily understood that this embodiment can also solve the problem that risk control value is affected by multiple factors such as market environment, manual review, and rule interception, making it impossible to accurately attribute to AI contribution, by using the three-dimensional parameters of "fixed baseline + actual performance + interception ratio", thus achieving accurate quantification of AI value in risk control scenarios. Therefore, in another feasible implementation, when the target evaluation dimension is the risk control dimension, the evaluation parameters include the model risk control ratio. In this embodiment, step S31 may include steps D1~D3: Step D1: When the target assessment dimension is the risk control dimension, determine the benchmark risk loss rate corresponding to the target business based on the business baseline data, and determine the actual risk loss rate of the target business after applying the intelligent model based on the business operation data. Step D2: Determine the model decision interception ratio corresponding to the target service based on the service operation data; Step D3: Generate the model risk control ratio corresponding to the risk control dimension based on the benchmark risk loss ratio, the actual risk loss ratio, and the model decision interception ratio.

[0085] It's important to understand that the risk control dimension refers to the core assessment dimension that measures the value of intelligent models in identifying, intercepting, reducing asset losses, and improving compliance rates. The aforementioned model risk control ratio can be considered a core assessment parameter of the risk control dimension, used to quantify the actual contribution of intelligent models to risk prevention and control, and is a core indicator for calculating net risk control contribution data.

[0086] At this point, the benchmark risk loss rate can be the benchmark loss level set before the target business applies the intelligent risk control model. It can use the bank-wide risk control loss BP (basispoint) value before the intelligent model is applied as a fixed baseline to avoid calculation deviations caused by baseline changes over time. The actual risk loss rate, on the other hand, can be the actual loss level within the statistical period after the target business applies the intelligent risk control model. The calculation formula can be expressed as: Actual Risk Loss Rate = Amount of Fraudulent Transactions / Total Amount of Successful Transactions × 10000, in BP (one ten-millionth).

[0087] For example, the assessment system can read a fixed baseline risk loss rate (a fixed value of 50 basis points, or 50 ten-millionths) from the business baseline data, denoted as R0. Then, it extracts the amount of fraudulent transactions and the total amount of all successful transactions within the statistical period from the business operation data, and calculates the actual risk loss rate R1 using the formula: R1 = Amount of fraudulent transactions / Total amount of successful transactions × 10,000. For example, in a certain intelligent anti-fraud scenario, the total amount of successful transactions within the statistical period is 100 billion yuan, and the amount of fraudulent transactions is 100,000 yuan. The actual loss rate = 100,000 / 100 billion yuan × 10,000 = 1 basis point.

[0088] In simple terms, the model-driven decision-based interception ratio refers to the proportion of high-risk transaction funds independently intercepted by the intelligent risk control model within a statistical period, relative to the total high-risk transaction interception funds. The calculation formula can be expressed as: Model-driven decision-based interception ratio = Funds rejected by the algorithm model / Total high-risk rejected funds. This ratio can be used to separate the risk control contributions of non-intelligent model factors such as manual review and rule-based interception, thereby determining the net contribution ratio of the intelligent model in risk prevention and control.

[0089] For example, the evaluation system can extract from business operation data the funds of high-risk transactions independently intercepted by the intelligent model within the statistical period (denoted as M) and the total funds intercepted from all high-risk transactions (denoted as Q). In this case, the model's decision-based interception ratio can be expressed as: P = M / Q. Here, assuming that in the above anti-fraud scenario, the model independently intercepted 90 million yuan within the statistical period, and the total funds intercepted from all high-risk transactions were 100 million yuan, then the model's decision-based interception ratio can be generated as 90 million / 100 million = 90%.

[0090] Finally, the evaluation system can calculate the model risk control ratio, i.e., the net contribution of AI, through a preset formula: Model Risk Control Ratio = (R0 - R1) × P. Similarly, in the above anti-fraud scenario, the model risk control ratio = (50BP - 1BP) × 90% = 44.1BP, meaning the intelligent risk control model brought a 44.1BP reduction in the loss rate, entirely due to the contribution of AI.

[0091] Subsequently, the evaluation system can further use the model risk control ratio as the core evaluation parameter of the risk control dimension, substitute it into the preset evaluation expression, and calculate the net contribution data of the risk control dimension, which can be expressed as: Risk control AI recovery revenue = model risk control ratio × total transaction amount within the statistical period, thereby realizing the financial quantification of risk control value.

[0092] In this implementation, by correcting the model's decision-making interception ratio, the contribution of non-AI factors such as manual review and rule-based interception is effectively eliminated, ensuring that the calculated risk control value comes entirely from the intelligent model. This addresses the industry pain point of the inability to accurately attribute risk control value. Furthermore, the economic losses recovered by AI are directly calculated through the model's risk control ratio, making the AI ​​value in risk control scenarios "invisible" and "quantifiable," providing a precise basis for subsequent investment in risk control AI resources.

[0093] It is easy to understand that in this embodiment, the target evaluation dimension may also include the business growth dimension. In this case, the evaluation system can measure the contribution of AI in expanding new markets, creating marketing content, improving market insight, improving sales efficiency, and increasing customer retention by measuring the value directly brought about by the application of AI technology, such as customer volume, business volume, and revenue, which are related to operating income. At this time, the net contribution data of the business growth dimension = (business income after intelligentization - income before intelligentization) AI contribution ratio coefficient.

[0094] In terms of industry competitiveness, the evaluation system can generate net contribution data that comprehensively assesses the leading position, status, and influence of similar AI applications by comparing their application data with that of other banks.

[0095] In terms of customer experience, the evaluation system can enhance the overall customer experience by applying AI technology during interactions with customers, primarily by collecting data such as satisfaction and waiting time as net contribution data.

[0096] In terms of customer experience, net contribution data can be generated by comparing the benefits that AI technology brings to customers, including cost reduction, revenue increase, and account and asset security (this data needs to be obtained with the user's consent).

[0097] In a feasible implementation, step S30 may further include steps S33-S35: Step S33: If the target evaluation dimension belongs to a pre-set quantitative economic dimension, the net contribution data is converted into quantitative economic data according to the preset quantitative rules, and the target focus business is determined based on the quantitative economic data. It's easy to understand that the aforementioned quantitative economic dimensions can be pre-set assessment dimensions that can be directly converted into economic benefits, including four dimensions: improved efficiency, procurement substitution, risk control, and business growth. That is, the assessment system reads the system's pre-set dimension classification rules and divides the target business's target assessment dimensions into two categories: ① Quantitative economic dimensions: improved efficiency, procurement substitution, risk control, and business growth; ② Non-quantitative economic dimensions: industry competitiveness, customer experience, and customer benefits.

[0098] The preset quantitative rules can be standardized conversion rules that convert net contribution data of each dimension into quantitative economic data in advance, including: ① Human efficiency improvement dimension: FTE × corresponding annual salary in the industry; ② Procurement substitution dimension: directly adopt the cost savings amount; ③ Risk control dimension: model risk control ratio × total transaction amount in the statistical period; ④ Business growth dimension: directly adopt the revenue growth amount brought by the intelligent model.

[0099] Therefore, quantitative economic data can be the direct economic benefit derived from the net contribution data of an intelligent model across four dimensions: comprehensive human efficiency improvement, procurement substitution, risk control, and business growth, obtained through pre-defined quantitative rules. In this context, the aforementioned target-focused business scenarios can be the high-value business scenarios with the highest return on investment identified through quantitative economic data ranking, representing the core direction for resource optimization.

[0100] At this point, the evaluation system can sum up the quantitative economic data across four dimensions to calculate the total Quantified Economic Benefit (QEB) of the target business. Simultaneously, the system can calculate the return on investment (ROI) for the QEB and AI R&D investment of all target businesses across the bank, sort them by ROI from highest to lowest, identify the top 20% of scenarios as target-focused businesses, and identify low-value scenarios with ROI below a threshold. Furthermore, the evaluation system can generate quantitative dimension analysis reports, including QEB details, rankings, ROI, a list of target-focused businesses, and resource optimization suggestions for each scenario.

[0101] Step S34: If the target evaluation dimension does not belong to the quantitative economic dimension, generate a dimension evaluation result based on the net contribution data. Step S35: Summarize the target focused business and / or the dimension evaluation results into the intelligent effectiveness evaluation results of the target business.

[0102] It is important to understand that the above-mentioned dimensional assessment results can be standardized assessment reports generated for non-quantitative economic dimensions, including three dimensions: industry competitiveness, customer experience, and customer benefits. These are presented in the form of qualitative descriptions plus quantitative indicators and are not included in QEB calculations. For example, regarding the industry competitiveness dimension: the assessment system, based on net contribution data, compares the application of similar AI applications in the industry, generating standardized qualitative assessment results from three dimensions: technological leadership, business innovation, and market influence. Regarding the customer experience dimension: the assessment system, based on indicators such as customer satisfaction, business processing time, and customer complaint rate from the net contribution data, compares the changes before and after the AI ​​implementation, generating a customer experience improvement assessment report. Regarding the customer benefits dimension: the assessment system, based on indicators such as customer cost savings, revenue increases, and asset security from the net contribution data, generates a customer value enhancement assessment report.

[0103] Finally, the evaluation system can integrate quantitative dimensions such as business analysis, QEB aggregated data, and input-output analysis with non-quantitative dimensions' evaluation results. Simultaneously, the system can supplement this with basic scenario information, month-on-month trend analysis, and year-on-year comparison analysis to generate complete and standardized intelligent performance evaluation results. These results can be viewed online, exported as PDF / Excel, and distributed according to user permissions.

[0104] In this implementation, the classification and processing logic of "quantitative economic conversion + non-quantitative qualitative assessment" is used to achieve financial quantification and comprehensive evaluation of the value of the intelligent model, while identifying high-value scenarios and achieving the core objective of "value measurement driving resource input".

[0105] In summary, this embodiment can convert the value of intelligent models into quantifiable economic benefits (QEB) through preset quantitative rules, and intuitively present the input-output effect of intelligent models in financial language, providing accurate and reliable data support for management resource investment decisions. Through quantitative and non-quantitative classification, it covers both direct economic value and long-term strategic value such as customer experience and industry competitiveness, comprehensively and objectively reflecting the complete effectiveness of intelligent model application and avoiding one-sided evaluation. Meanwhile, this embodiment accurately identifies high-value target-focused businesses and low-value scenarios through the ranking analysis of quantitative economic data, which can effectively guide intelligent model R&D resources and business resources to tilt towards high-value scenarios, and significantly improve the input-output ratio of the bank's intelligent model resources. Furthermore, this embodiment transforms complex intelligent model value data into clear amounts, rankings, trends, and recommendations through standardized summary reports. Whether it is technical personnel, business personnel, or management, they can quickly understand the effectiveness of the intelligent model application, thereby improving the dissemination and application value of the evaluation results.

[0106] For example, to help understand the technical concept or principle of the business intelligence effectiveness evaluation method after combining this embodiment with the above-described Embodiment 1 and Embodiment 2, please refer to Figure 3 , Figure 3 A simplified flowchart illustrating a method for evaluating the effectiveness of business intelligence is provided below: like Figure 3 As shown, the implementation process of the business intelligence effectiveness evaluation method proposed in this application may include the following steps: The first step is the identification and creation of intelligent application scenarios: The technical lead creates intelligent models on the business intelligence effectiveness evaluation system, consuming computing power resources. The system requires that the created intelligent model be associated with an "intelligent business scenario" on the evaluation system. If the scenario does not yet exist on the evaluation system, it must be created there first.

[0107] The second step is to enter basic information about the scene and skills: The scenario's technical lead and business lead log into the evaluation system and enter the following information on the

Scenario Basic Information Management

[0108] If the scenario is large, it can be broken down into multiple skills according to business processes, business objects, or problem-solving: then in the [Skill Basic Information Management] interface, enter the name, type, and scenario to which each skill belongs.

[0109] Step 3: Configure value measurement metrics: In the [Metrics Management] module, the technical lead and business lead jointly select evaluation dimensions for scenarios or skills: From a banking perspective: improving employee efficiency, procurement substitution, risk control, business growth, and industry competitiveness; Customer perspective: customer experience, customer benefits; Under each dimension, specific indicators (such as FTE, risk recovery returns, etc.) can be selected, and indicator parameters can be edited.

[0110] Once configured, the system records the relationship between scenarios / skills and metrics.

[0111] Step 4: Measuring task generation and triggering: At each fixed evaluation period (e.g., the end of the month), the evaluation system automatically generates the measurement data entry task for that statistical period.

[0112] After receiving the notification, the scenario manager should fill in the form in the order guided by the system: If the scenario has subordinate skills, fill in the skill measurement data first, then fill in the scenario measurement data.

[0113] If you have no skills, fill in the scene measurement data directly.

[0114] Step 5, Measurement data collection (two methods): Method A: Manual entry. The scenario manager directly enters the measurement data (such as total business volume, amount saved in working hours, etc.) on the system interface according to the calculation rules of each dimension indicator.

[0115] Method B: Automatic Data Collection via System Integration. Administrators pre-configure the scenarios / skills and metrics that support system integration within the system, and set the response address for the business system. When the evaluation system automatically initiates an evaluation task, it automatically encapsulates the query request according to the following rules: Rule 1: Includes metrics under both the scenario and skill; Rule 2: Only encapsulates metrics configured as "supporting system integration".

[0116] Then, the evaluation system sends an HTTP request (containing a list of request protocols) to the business system. Upon receiving the request, the business system returns a response message in the agreed format (containing the scene code, skill code, indicator code, and numerical value), and the evaluation system then receives the response message.

[0117] Step 6, Data Validation: The evaluation system performs basic checks on the received data: 1) Consistency check: Determine whether the returned scenario + indicator code (identifier) ​​and skill + indicator code are completely consistent with the request (quantity and code match); 2) Null value check: Check whether the fields that are forcibly checked (such as the total business volume corresponding to FTE, the indicator value after intelligentization) are empty.

[0118] If the verification passes, the data is entered into the system; if the verification fails, the interface displays "Updated on XXXXX verification failure details" and provides the specific reason.

[0119] Step 7: Calculation of value measurement results for each dimension: Net contribution data is generated based on different value assessment dimensions and corresponding calculation rules (data source is already collected measurement data): Effective Time Efficiency (FTE) = Total saved working hours during the statistical period / Total standard working hours of a full-time employee throughout the year. The total saved working hours take into account the total business volume, the average time spent per employee before and after intelligentization, and business factors (adoption rate, connection rate, and conversion rate). Cost savings from procurement substitution = actual quantity × (cost per purchase before intelligentization - cost per purchase after intelligentization); Risk control recovery profit = (benchmark loss BP - loss BP after application) × total transaction amount × AI model decision interception ratio; Business growth revenue = (Business revenue after intelligentization - Business revenue before intelligentization) × AI contribution ratio coefficient (if it is difficult to accurately assess). Industry competitiveness, customer experience, and customer benefits are determined using qualitative or quantitative data analysis.

[0120] Step 8, Quantitative Economic Benefits (QEB) Summary: The four dimensions that can be quantified into economic value are combined, namely QEB = FTE revenue from improved human efficiency + revenue from procurement substitution + revenue from business growth + revenue from risk control. Among them, FTE revenue = FTE × the corresponding annual salary in the industry; procurement substitution revenue, business growth revenue, and risk control revenue directly use the above calculation results.

[0121] Step 9: Data Analysis and Report Generation The system displays the economic value of each scenario according to different dimensions (efficiency improvement, procurement substitution, risk control, and business growth), and performs data aggregation and analysis according to dimensions such as business departments and R&D centers. Finally, it generates a value measurement report for decision-makers to refer to, so as to guide the investment of AI resources in high-value scenarios.

[0122] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the business intelligence effectiveness evaluation method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0123] Based on Embodiment 1 and / or Embodiment 2 of the above methods, this application also provides a business intelligence effectiveness evaluation system, please refer to... Figure 4 , Figure 4 This is a schematic diagram of the module structure of the business intelligence effectiveness evaluation system according to an embodiment of this application. In this embodiment, the business intelligence effectiveness evaluation system includes: a scenario configuration system, a business system, and an evaluation system. The evaluation system is used to obtain scenario configuration information corresponding to the target business from the scenario configuration system. The scenario configuration information includes data collection rules and at least one target evaluation dimension corresponding to the target business. The evaluation system is also used to obtain the original business data corresponding to the target evaluation dimension from the business system associated with the target business according to the data collection rules. The original business data includes the business baseline data corresponding to the target business before the application of the intelligent model and the business operation data of the target business after the application of the intelligent model. The evaluation system is also used to convert the original business data into net contribution data corresponding to the intelligent model and the target evaluation dimension based on the attribution analysis strategy corresponding to the target evaluation dimension. The evaluation system is also used to generate intelligent performance evaluation results for the target business based on preset quantitative rules and the net contribution data.

[0124] It should be noted that this intelligent business performance evaluation system can be an intelligent value measurement platform based on a B / S (Browser / Server) architecture, used to automate and multi-dimensionally measure the value of business scenarios within financial institutions that utilize artificial intelligence technology. The system can adopt a modular design, including a scenario configuration system, a business system, and an evaluation system. The modules interact through standardized data interfaces and message queues, achieving fully automated processing from scenario identification, indicator setting, automatic data collection to value quantification and analysis.

[0125] It is understood that in this embodiment, in order to obtain the scenario configuration information corresponding to the target business, the basic configuration of the target business can be completed in advance in the scenario configuration system: enter the scenario name, intelligent model type, person in charge, business department and other basic information, select at least one target evaluation dimension for the scenario (such as the "business growth + human efficiency improvement" dimension for the retail marketing scenario, and the "risk control + procurement substitution" dimension for the risk control scenario), and configure the corresponding data collection rules (such as interface address, request format, and collection period of the 1st of each month).

[0126] When an evaluation task is initiated, the evaluation system can retrieve the complete scenario configuration information corresponding to the target business from the scenario information table and indicator association table of the scenario configuration system through the JDBC data access interface / API interface / message queue, cache it in the local memory of the evaluation system, and provide parameter support for subsequent processes.

[0127] Then, in order to automatically obtain the original business data corresponding to the target evaluation dimensions, the evaluation system can parse the data collection rules in the scenario configuration information to determine the data connection method, request protocol format, business system response address, and indicator fields to be collected.

[0128] The evaluation system automatically initiates data query requests to related business systems according to a preset collection cycle. Upon receiving the request, the business system returns the business baseline data and business operation data for the target business within the statistical period in an agreed format. The evaluation system receives the returned raw business data, stores it in a structured manner according to scenario, dimension, and indicator, and stores it in a measurement data table, thus completing automated data collection.

[0129] Then, the net contribution data corresponding to the intelligent model is calculated. The evaluation system can call the corresponding attribution analysis strategy pre-set in the system according to the target evaluation dimension to preprocess the raw business data, including missing value imputation, outlier removal, and data standardization, to obtain the evaluation parameters corresponding to that dimension.

[0130] The evaluation system substitutes the evaluation parameters into the preset evaluation expression corresponding to the dimension, performs quantitative calculations, removes the interference of non-intelligent model factors, and finally obtains the net contribution data of the intelligent model under the dimension.

[0131] If the target business is configured with multiple target evaluation dimensions, the above logic will be executed repeatedly to complete the calculation of net contribution data for all dimensions.

[0132] Finally, an intelligent performance evaluation result is generated. The evaluation system categorizes and processes the net contribution data for all dimensions: for dimensions with quantifiable economic value, it converts them into quantifiable economic benefits according to preset rules; for dimensions that are not quantifiable, it generates a standardized dimension evaluation report.

[0133] The evaluation system performs multi-dimensional aggregation analysis on net contribution data, including ranking analysis by business department, R&D center, and intelligent model type, month-on-month trend analysis, and high-value scenario identification.

[0134] The evaluation system integrates the above content to generate standardized, intelligent effectiveness evaluation results, including detailed dimensional value, total quantified economic benefits, scenario ranking, trend analysis, and resource investment recommendations, which can be viewed, exported, and distributed online.

[0135] The business intelligence effectiveness evaluation system provided in this application, employing the business intelligence effectiveness evaluation method described in the above embodiments, can solve the technical problems of business intelligence effectiveness evaluation. Compared with the prior art, the beneficial effects of the business intelligence effectiveness evaluation system provided in this application are the same as those of the business intelligence effectiveness evaluation method described in the above embodiments, and other technical features of the business intelligence effectiveness evaluation system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0136] This application provides a business intelligence effectiveness evaluation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the business intelligence effectiveness evaluation method in the above embodiment one.

[0137] The following is for reference. Figure 5The diagram illustrates a structural schematic suitable for implementing the business intelligence effectiveness evaluation device in the embodiments of this application. The business intelligence effectiveness evaluation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The business intelligence effectiveness evaluation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0138] like Figure 5 As shown, the business intelligence effectiveness evaluation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the business intelligence effectiveness evaluation device. The processing unit 1001, the ROM 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the business intelligence effectiveness evaluation device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows business intelligence effectiveness evaluation devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0139] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in this application include a business intelligence effectiveness evaluation program product, which includes a business intelligence effectiveness evaluation program carried on a computer-readable medium, the business intelligence effectiveness evaluation program containing program code for performing the methods shown in the flowcharts. In such embodiments, the business intelligence effectiveness evaluation program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the business intelligence effectiveness evaluation program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0140] The business intelligence effectiveness evaluation device provided in this application, employing the business intelligence effectiveness evaluation method described in the above embodiments, can solve the technical problems of business intelligence effectiveness evaluation. Compared with the prior art, the beneficial effects of the business intelligence effectiveness evaluation device provided in this application are the same as those of the business intelligence effectiveness evaluation method described in the above embodiments, and other technical features of the business intelligence effectiveness evaluation device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0141] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0142] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0143] This application provides a storage medium having computer-readable program instructions (i.e., a business intelligence effectiveness evaluation program) stored thereon, which are used to execute the business intelligence effectiveness evaluation method in the above embodiments.

[0144] The storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of the storage medium may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0145] The aforementioned storage medium may be included in the business intelligence effectiveness evaluation device; or it may exist independently and not be installed in the business intelligence effectiveness evaluation device.

[0146] The aforementioned storage medium carries one or more programs. When these programs are executed by the business intelligence effectiveness evaluation device, the business intelligence effectiveness evaluation device solves the problem of business intelligence effectiveness evaluation.

[0147] The business intelligence effectiveness evaluation program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and business intelligence effectiveness evaluation program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0149] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0150] The readable storage medium provided in this application is a storage medium that stores computer-readable program instructions (i.e., a business intelligence effectiveness evaluation program) for executing the above-described business intelligence effectiveness evaluation method, and can solve the technical problem of business intelligence effectiveness evaluation. Compared with the prior art, the beneficial effects of the storage medium provided in this application are the same as the beneficial effects of the business intelligence effectiveness evaluation method provided in the above embodiments, and will not be repeated here.

[0151] The above are only some embodiments of this application and do not limit the scope of the solution of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

Claims

1. A method for evaluating the effectiveness of business intelligence, characterized in that, The method includes: Obtain the scenario configuration information corresponding to the target business, wherein the scenario configuration information includes the data collection rules and at least one target evaluation dimension corresponding to the target business; According to the data collection rules, the original business data corresponding to the target evaluation dimension is obtained from the business system associated with the target business. The original business data includes the business baseline data of the target business before the application of the intelligent model and the business operation data of the target business after the application of the intelligent model. Based on the attribution analysis strategy corresponding to the target evaluation dimension, the original business data is converted into net contribution data corresponding to the intelligent model and the target evaluation dimension, and the intelligent effectiveness evaluation result of the target business is generated based on the net contribution data.

2. The business intelligence effectiveness evaluation method as described in claim 1, characterized in that, The step of obtaining the original business data corresponding to the target evaluation dimension from the business system associated with the target business according to the data collection rules includes: The request protocol format is determined according to the data collection rules. The request protocol format includes at least one of the following: full-time human resources equivalent calculation format, before-and-after numerical comparison format, or monetary numerical format. Obtain the scene identifier and scene metrics corresponding to the target service; If the scenario indicators meet the preset docking requirements, a query request message is assembled based on the request protocol format, and the query request message carries the scenario identifier and the scenario indicators. The query request message is sent to the response address of the business system associated with the target business, and the original business data returned by the business system in accordance with the request protocol format is received.

3. The business intelligence effectiveness evaluation method as described in claim 1, characterized in that, The step of converting the original business data into net contribution data corresponding to the intelligent model and the target evaluation dimension based on the attribution analysis strategy includes: The original business data is preprocessed based on the attribution analysis strategy corresponding to the target evaluation dimension to obtain the evaluation parameters corresponding to the target evaluation dimension. The intelligent model generates net contribution data corresponding to the target evaluation dimension based on the preset evaluation expression and the evaluation parameters.

4. The business intelligence effectiveness evaluation method as described in claim 3, characterized in that, When the target evaluation dimension is the human resource improvement dimension, the evaluation parameters include the equivalent value of full-time human resources; The step of preprocessing the original business data based on the attribution analysis strategy corresponding to the target evaluation dimension to obtain the evaluation parameters corresponding to the target evaluation dimension includes: When the target evaluation dimension is the human resource improvement dimension, the total saved working hours within the statistical period are determined based on the business baseline data and the business operation data. The total saved working hours are effectively corrected by the scenario calibration factor corresponding to the target business to obtain the corrected working hours; The ratio of the standard working hours of full-time employees to the modified working hours is calculated to obtain the equivalent value of the full-time workforce corresponding to the human resource improvement dimension.

5. The business intelligence effectiveness evaluation method as described in claim 3, characterized in that, When the target assessment dimension is the risk control dimension, the assessment parameters include the model risk control ratio; The step of preprocessing the original business data based on the attribution analysis strategy corresponding to the target evaluation dimension to obtain the evaluation parameters corresponding to the target evaluation dimension includes: When the target assessment dimension is the risk control dimension, the benchmark risk loss rate corresponding to the target business is determined based on the business baseline data, and the actual risk loss rate of the target business after applying the intelligent model is determined based on the business operation data. The model decision interception ratio corresponding to the target business is determined based on the business operation data. The model risk control ratio corresponding to the risk control dimension is generated based on the benchmark risk loss ratio, the actual risk loss ratio, and the model decision interception ratio.

6. The business intelligence effectiveness evaluation method as described in claim 1, characterized in that, The step of generating the intelligent effectiveness evaluation result of the target business based on the net contribution data includes: If the target evaluation dimension belongs to a pre-set quantitative economic dimension, the net contribution data is converted into quantitative economic data according to the preset quantitative rules, and the target focus business is determined based on the quantitative economic data. If the target evaluation dimension does not belong to the quantitative economic dimension, a dimension evaluation result is generated based on the net contribution data. The target focused business and / or the dimension evaluation results are summarized into the intelligent effectiveness evaluation results of the target business.

7. The business intelligence effectiveness evaluation method as described in claim 1, characterized in that, Before converting the original business data into net contribution data corresponding to the intelligent model and the target evaluation dimension based on the attribution analysis strategy corresponding to the target evaluation dimension, the process includes: The original business data is subjected to integrity verification, which includes identifier consistency verification and null value verification. If the integrity check fails, an error message is generated and sent to the management terminal.

8. A business intelligence effectiveness evaluation system, characterized in that, The system includes: a scenario configuration system, a business system, and an evaluation system; The evaluation system is used to obtain scenario configuration information corresponding to the target business from the scenario configuration system. The scenario configuration information includes data collection rules and at least one target evaluation dimension corresponding to the target business. The evaluation system is also used to obtain the original business data corresponding to the target evaluation dimension from the business system associated with the target business according to the data collection rules. The original business data includes the business baseline data corresponding to the target business before the application of the intelligent model and the business operation data of the target business after the application of the intelligent model. The evaluation system is also used to convert the original business data into net contribution data corresponding to the intelligent model and the target evaluation dimension based on the attribution analysis strategy corresponding to the target evaluation dimension. The evaluation system is also used to generate intelligent performance evaluation results for the target business based on preset quantitative rules and the net contribution data.

9. A business intelligence effectiveness evaluation device, characterized in that, The device includes: a memory, a processor, and a business intelligence effectiveness evaluation program stored on the memory and executable on the processor, the business intelligence effectiveness evaluation program being configured to implement the steps of the business intelligence effectiveness evaluation method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a business intelligence effectiveness evaluation program, which, when executed by a processor, implements the steps of the business intelligence effectiveness evaluation method as described in any one of claims 1 to 7.

Images