Model quality monitoring method, device, electronic device and storage medium

Abstract

The present disclosure provides a model quality monitoring method, device, electronic device and storage medium. It relates to the field of computer application technology, and in particular to artificial intelligence fields such as model development. The specific implementation scheme is: obtaining the first quality indicator data of the target model; obtaining the input data and output data of the target model within a preset time period; based on the input data and output data within the preset time period, using the target algorithm to determine the second quality indicator data of the target model; comparing the difference between the second quality indicator data and the first quality indicator data, and issuing an alarm when the difference exceeds the preset range. According to the technical solution of the present disclosure, the quality of the model can be intelligently monitored, the quality changes of the model can be quickly and timely fed back, and the efficiency of model quality monitoring can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer application technology, in particular to artificial intelligence fields such as model development, and specifically to a model quality monitoring method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence (AI), more and more industries and fields are using AI technology to solve their own business problems and achieve continuous business innovation. Since the quality of AI models will directly affect the ability of the business to operate, it is necessary to monitor the quality of the model. In the related technologies, in the process of model quality monitoring, there are multiple links that require manual participation and business intrusion, which is not conducive to the realization of automated engineering. In addition, the related monitoring methods cannot provide timely warnings of model quality issues. Summary of the invention

[0003] The present disclosure provides a model quality monitoring method, device, electronic device and storage medium.

[0004] According to a first aspect of the present disclosure, a model quality monitoring method is provided, comprising:

[0005] Acquire first quality indicator data of the target model;

[0006] Obtaining input data and output data of the target model within a preset time period;

[0007] Determine second quality indicator data of the target model using a target algorithm based on the input data and the output data within the preset time period;

[0008] The difference between the second quality indicator data and the first quality indicator data is compared, and an alarm is issued when the difference exceeds a preset range.

[0009] According to a second aspect of the present disclosure, there is provided a model quality monitoring device, comprising:

[0010] A first acquisition module, used to acquire first quality indicator data of a target model;

[0011] A second acquisition module is used to acquire input data and output data of the target model within a preset time period;

[0012] A first determination module, configured to determine second quality indicator data of a target model using a target algorithm based on input data and output data within a preset time period;

[0013] The monitoring module is used to compare the difference between the second quality indicator data and the first quality indicator data, and to issue an alarm when the difference exceeds a preset range.

[0014] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided by the first aspect above.

[0018] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method provided by the first aspect above.

[0019] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the method provided in the above first aspect when executed by a processor.

[0020] The disclosed embodiments can intelligently monitor the quality of the model, quickly and timely provide feedback on changes in the quality of the model, and improve the efficiency of model quality monitoring.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 The following is a flow chart of the model quality monitoring method according to the embodiment of the present disclosure. Figure 1 ;

[0024] Figure 2 The following is a flow chart of the model quality monitoring method according to the embodiment of the present disclosure. Figure 2 ;

[0025] Figure 3 is a schematic diagram of the architecture of model quality monitoring according to an embodiment of the present disclosure;

[0026] Figure 4 The structure of the model quality monitoring device according to the embodiment of the present disclosure is shown in FIG. Figure 1 ;

[0027] Figure 5The structure of the model quality monitoring device according to the embodiment of the present disclosure is shown in FIG. Figure 2 ;

[0028] Figure 6 is a schematic diagram of a model quality monitoring scenario according to an embodiment of the present disclosure;

[0029] Figure 7 It is a block diagram of an electronic device used to implement the model quality monitoring method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] The terms "first", "second", and "third", etc. in the specification embodiments and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product, or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0032] The uncertainty of AI models and business production environment data, the complexity of model quality monitoring processes, etc., pose challenges to the platform-based and automated construction of model quality monitoring functions. Monitoring the quality of the AI ​​models carried and evaluating whether they meet the needs of the production environment can guide users to continuously optimize the models, so that AI technology can better empower the business and enhance the influence and attractiveness of the platform.

[0033] In the related art, one implementation scheme is to measure whether the model quality is valuable to the business by monitoring the business indicators of the actual production environment, such as ad click-through rate, function recognition, etc. However, this scheme has at least the following disadvantages: a. There are many factors that affect the actual business indicators, and it is difficult to quickly check whether it is caused by model quality problems; b. Even if the decline in business indicators is caused by model quality problems, it is still impossible to automatically give quantitative indicators, and manual inspection is required.

[0034] In the related art, another implementation scheme is to regularly parse and collect the input and output of the AI ​​model in the business code or the log of the AI ​​model service, and check whether the model quality meets expectations based on the model quality assessment algorithm. However, although this scheme solves the shortcomings of the first scheme to a certain extent, it still has the following shortcomings: a. There are multiple links in the whole process that require human participation and business intrusion, which is not conducive to automated engineering implementation. For example, it is necessary to add code to print the input and output of the model in the business module; b. This a posteriori monitoring method cannot promptly alert the problem. When the alarm is triggered, the model quality has already failed to meet the online business requirements at some point in the past, causing negative business impact for a period of time.

[0035] This paper proposes an automatic and intelligent model quality monitoring process based on the process characteristics of model quality monitoring to solve a series of problems faced by model quality monitoring.

[0036] The present disclosure provides a model quality monitoring method. Figure 1 1 is a flow chart of a model quality monitoring method according to an embodiment of the present disclosure, and the model quality monitoring method can be applied to a model quality monitoring device. The model quality monitoring device is located in an electronic device, and the electronic device includes but is not limited to a fixed device and / or a mobile device. In some possible implementations, the method can also be implemented by a processor calling a computer-readable instruction stored in a memory. Figure 1 As shown, the model quality monitoring method includes:

[0037] S101: Acquire first quality indicator data of a target model;

[0038] S102: Obtaining input data and output data of the target model within a preset time period;

[0039] S103: Determine second quality indicator data of the target model using a target algorithm based on the input data and the output data within the preset time period;

[0040] S104: Compare the difference between the second quality indicator data and the first quality indicator data, and issue an alarm if the difference exceeds a preset range.

[0041] In the disclosed embodiments, the target model is an AI model that can support at least one AI service. For example, the target model is an object detection model. For another example, the target model is a text classification model. For another example, the target model is an image classification model. The above is only an exemplary description and is not intended to limit all possible types of target models, but it is not exhaustive here.

[0042] It should be noted that the present disclosure does not limit the number of AI services supported by the target model. Taking the target model as an object detection model, the object detection model supports detection services for multiple types of objects. For example, the first service of the object detection model is to identify cats in images, the second service of the object detection model is to identify dogs in images, and the third service of the object detection model is to identify rabbits in images.

[0043] In the disclosed embodiment, the first quality indicator data is a benchmark quality indicator data, that is, a basis for reference when judging quality changes.

[0044] In the embodiment of the present disclosure, the second quality indicator data is quality indicator data calculated based on actual input data and output data of the target model.

[0045] In the disclosed embodiment, the preset time period can be set or adjusted according to design requirements or user requirements. For example, the preset time period is measured in units of 10 minutes, 30 minutes, or 1 hour. For another example, if the second quality change data of the target moment is received from the user terminal, the target moment is taken as the end point, and the data within the preset time period before the target moment is obtained.

[0046] In some embodiments, obtaining the input data and output data of the target model within a preset time period includes: collecting the input data and output data of the target model in real time; when a certain trigger condition is met, obtaining the input data and output data of the target model within a preset time period from the collected input data and output data of the target model. Here, the trigger condition can be set according to design requirements or user requirements. For example, the trigger condition is: quality monitoring is performed every 20 minutes or half a day, that is, a quality monitoring report is output every certain period of time. For another example, the trigger condition is: receiving an instruction sent by a user terminal to obtain the quality change of the target model.

[0047] In other embodiments, obtaining the input data and output data of the target model within a preset time period includes: collecting the input data and output data of the target model in real time; when the collection time meets the preset time period, using the collected input data and output data of the target model as the input data and output data of the target model within the preset time period.

[0048] In the disclosed embodiment, the target algorithm is an algorithm selected from the indicator algorithm library for calculating the second quality indicator data of the target model. Here, the indicator algorithm library includes multiple algorithms for calculating indicators. It should be noted that different algorithms correspond to different monitoring types. Monitoring types include but are not limited to abnormal point detection monitoring, offset detection monitoring, and the like.

[0049] The technical solution described in the embodiment of the present disclosure obtains first quality indicator data of a target model; obtains input data and output data of the target model within a preset time period; determines second quality indicator data of the target model using a target algorithm based on the input data and output data within the preset time period; compares the difference between the second quality indicator data and the first quality indicator data, and issues an alarm when the difference exceeds a preset range; in this way, the quality of the model can be intelligently monitored without human participation and business intrusion, and the quality changes of the model can be quickly and timely fed back, thereby improving the efficiency of model quality monitoring.

[0050] In some embodiments, obtaining first quality indicator data of a target model includes: obtaining a monitoring type and benchmark data; determining a target algorithm matching the monitoring type from an indicator algorithm library; and determining the first quality indicator data of the target model using the target algorithm based on the benchmark data.

[0051] Here, the monitoring type refers to the type of quality indicator change used to evaluate the target model. For example, the monitoring type includes anomaly detection monitoring, that is, monitoring the points where anomalies occur. For example, target model 1 is used to identify the cat in the picture. If the object identified by target model 1 is not a cat, it is considered an anomaly. For another example, the monitoring type includes offset detection monitoring, that is, monitoring the offset of the data. For example, target model 1 is used to count the male-female ratio of each class in a school. If the baseline male-female ratio is less than 1.3:1, if the male-female ratio of class A is 0.3:1, it is considered that the data offset of class A is large.

[0052] Here, the benchmark data is data used to determine the first quality indicator data. In some embodiments, the benchmark data may include benchmark input data and benchmark output data. For example, input sample data and output sample data when training the target model. In other embodiments, the benchmark data may include pictures with annotated data. For example, sample annotated data when training the target model.

[0053] In this way, since the first quality indicator data and the second quality indicator data are calculated by using the same target algorithm, the reliability of the comparison result can be improved, and the accuracy of the comparison result can also be improved.

[0054] In some embodiments, the preset time period includes multiple first moments; based on the input data and output data within the preset time period, the second quality indicator data of the target model is determined using the target algorithm, including: based on the input data and output data corresponding to the multiple first moments, respectively, the second quality indicator data corresponding to the multiple first moments are determined using the target algorithm.

[0055] Here, "multiple" is a general term, and the specific number will be set and adjusted according to user needs.

[0056] Exemplarily, taking the preset time period [t1, tn] as an example, [t1, tn] includes multiple moments such as t(1+i) and tn. Based on the input data and output data corresponding to the moment t(1+i), the second quality indicator data corresponding to the moment t(1+i) can be determined using the target algorithm.

[0057] As another example, taking the preset time period [t1, tn] as an example, where [t1, tn] includes multiple moments such as t1, t(i) and t(n-1), the second quality indicator data corresponding to the moment t(n-1) can be determined based on the input data and output data corresponding to the moment t(n-1) using the target algorithm.

[0058] In some embodiments, the preset time period includes multiple first moments; based on the input data and output data within the preset time period, the second quality indicator data of the target model is determined using the target algorithm, including: based on the input data and output data within the preset time period, the second quality indicator data corresponding to the multiple first moments are determined using the target algorithm.

[0059] In some specific implementations, when using a target algorithm to determine the second quality indicator data corresponding to each first moment, input data and output data in a time period before the first moment in the preset time period are used as input data of the target algorithm.

[0060] Exemplarily, taking the preset time period [t1, tn] as an example, the second quality indicator data corresponding to the time t(1+i) can be determined using a target algorithm based on the input data and output data within the period [t1, t(1+i)].

[0061] As another example, taking the preset time period [t1, tn] as an example, the second quality indicator data corresponding to the time t(n-1) can be determined using a target algorithm based on the input data and output data within the period [t1, t(n-1)].

[0062] In this way, data support can be provided for subsequent prediction of the second quality indicator data at the second moment, thereby helping to predict the second quality indicator data at future moments such as the second moment, and further helping to warn of possible problems in the future in advance.

[0063] In some embodiments, the difference between the second quality indicator data and the first quality indicator data is compared, and an alarm is issued when the difference exceeds a preset range, including: comparing the second quality indicator data corresponding to multiple first moments with the first quality indicator data to obtain first differences corresponding to multiple first moments; in response to the presence of a first difference exceeding a first preset range in the first differences corresponding to multiple first moments, issuing an alarm using a first alarm method.

[0064] Here, the first preset range can be set or adjusted according to actual conditions such as monitoring accuracy or monitoring efficiency.

[0065] Here, the first warning method includes but is not limited to voice warning, text warning, image warning, alarm warning, etc.

[0066] Here, when there is at least one first difference exceeding a first preset range among the first differences respectively corresponding to the plurality of first moments, an alarm is issued for the at least one first difference exceeding the first preset range.

[0067] For example, the m first moments are respectively recorded as t1, t2, ..., tm, the first difference of the first moment t1 is recorded as c1, the first difference of the first moment t2 is recorded as c2, and the first difference of the first moment tm is recorded as cm. If c1 and cm among c1, c2, ..., cm exceed the first preset range, then the first alarm method is used to alarm t1 and tm.

[0068] In this way, changes in model quality can be quickly and timely fed back, and alarms and reminders can be issued for quality changes that exceed the first preset range, so as to notify the service owner to pay attention to the dynamics of the model and re-iterate the training to obtain a model that is more in line with the actual production environment.

[0069] In some embodiments, Figure 2 As shown, the model quality monitoring method includes:

[0070] S201: Acquire first quality indicator data of a target model;

[0071] S202: Obtaining input data and output data of the target model within a preset time period;

[0072] S203: Based on the input data and the output data within the preset time period, using a target algorithm to determine second quality indicator data corresponding to a plurality of first moments of the target model;

[0073] S204: Based on the second quality indicator data respectively corresponding to the plurality of first moments, using a prediction model selected from a prediction model library, predicting second quality indicator data at a second moment, where the second moment is after the plurality of first moments and outside a preset time period;

[0074] S205: Compare the second quality indicator data at the second moment with the first quality indicator data to obtain a second difference; when the second difference exceeds a second preset range, issue an alarm in a second alarm mode.

[0075] Here, the prediction model library includes a plurality of prediction models, each of which can be used to predict the second quality indicator data at the second moment according to the second quality indicator data at the first moment.

[0076] In some implementations, selecting a prediction model from the prediction model library includes: randomly selecting an idle prediction model from the prediction database as the prediction model of the target model. In this way, the prediction speed of the second quality change data at the second moment can be improved.

[0077] In other embodiments, selecting a prediction model from a prediction model library includes: selecting a prediction model matching a target model from a prediction database as the prediction model of the target model. In this way, the prediction accuracy of the second quality change data at the second moment can be improved.

[0078] In some other embodiments, selecting a prediction model from a prediction model library includes: selecting a prediction model associated with a target algorithm from a prediction database as a prediction model of the target model. In this way, the prediction accuracy of the second quality change data at the second moment can be improved.

[0079] Exemplarily, taking the preset time period [t1, tn] as an example, based on the second quality indicator data corresponding to time t1, the second quality indicator data corresponding to time t(1+i), and the second quality indicator data corresponding to time tn in [t1, tn], the second quality indicator data corresponding to the second time t(n+1) and the second quality indicator data corresponding to time t2n can be predicted.

[0080] In this way, the quality changes of the model in the future can be predicted, which is convenient for early warning service owners to optimize the model and effectively avoid the real impact of model quality problems on online business.

[0081] In some embodiments, the difference between the second quality indicator data and the first quality indicator data is compared, and an alarm is issued when the difference exceeds a preset range, including: comparing the second quality indicator data at a second moment with the first quality indicator data to obtain a second difference; when the second difference exceeds a second preset range, an alarm is issued using a second alarm method.

[0082] Here, the second preset range can be set or adjusted according to actual conditions such as monitoring accuracy or monitoring efficiency.

[0083] Here, the second preset range may be the same as the first preset range, or may be different from the first preset range. For example, the second preset range is smaller than the first preset range. For another example, the second preset range is larger than the first preset range.

[0084] Here, the second warning mode includes but is not limited to voice warning, text warning, image warning, alarm warning, etc. The second warning mode may be the same as the first warning mode, or may be different from the first warning mode.

[0085] When the quality change data of multiple second moments are predicted to exceed the second preset range, a reminder is issued for the quality change of the second moment that is the earliest among the multiple second moments. In this way, the moment that needs attention can be reminded in time, so that the service owner can optimize the current target model in time according to the moment, which helps to reduce the impact of the target model quality change on the target model business.

[0086] In this way, the quality indicators for a period of time in the future are predicted through existing data, and alarms and reminders are issued for quality changes that exceed the second preset range, so as to warn the service owner in advance to optimize the model and reduce the impact of changes in the target model quality on the target model business.

[0087] Figure 3 A schematic diagram of the model quality monitoring architecture is given, such as Figure 3 As shown in the figure, the architecture is divided into several parts: request data collection module, monitoring indicator algorithm library, monitoring baseline generation module, quality real-time monitoring module, monitoring data statistics module, indicator intelligent learning module and monitoring alarm module.

[0088] Among them, the request data collection module: hijacks and forwards traffic through a proxy, collects the request and response data of the model service, and achieves zero intrusion into the business code; can configure the sampling rate, collection cycle, etc. to control the frequency of data collection; configure the structure of the request and response data to correctly parse the input and output data of the model from the data;

[0089] Among them, the monitoring indicator algorithm library is responsible for implementing the algorithms for various calculation model quality indicators; it has a unified interface protocol and plug-in implementation, which is convenient for adding support for new indicator calculation algorithms;

[0090] Among them, the baseline indicator generation module: according to the specified benchmark data and monitoring type, selects the corresponding algorithm from the monitoring indicator algorithm library to calculate the baseline indicator data of the model quality indicator;

[0091] Among them, the quality real-time monitoring module: according to the specified monitoring type, select the corresponding algorithm from the monitoring indicator algorithm library, regularly read the real-time data, and calculate the real-time data of the model quality indicator; this module is a scheduled task, which regularly calculates quasi-real-time indicators;

[0092] Among them, the monitoring data management module is responsible for centrally storing and integrating monitoring indicator data, including request response data, benchmark and real-time indicators and other time series data;

[0093] Among them, the indicator intelligent learning module: by using algorithm models such as regression or time series prediction, it continuously learns the changing rules of fitting indicator data, and can predict the indicator value for a period of time in the future based on the model;

[0094] Among them, the monitoring and alarm module is responsible for comparing the differences between quantitative real-time indicator data and benchmark indicator data; and issuing alarm reminders in a specified manner when the data exceeds the tolerable range.

[0095] Model quality monitoring is a very late link in the entire AI technology implementation process. Before model quality monitoring, some prerequisite work is required: use your own business data to train the model, deploy the model as a service, and then call the service uninterruptedly in actual business. The model monitoring process is as follows:

[0096] Step 1. Configure the model quality monitoring task: specify the monitored service, benchmark data, monitoring indicators, alarm strategy, and alarm method, and start quality monitoring of an AI service-bearing model;

[0097] Step 2. Generate the quality index of the benchmark model: According to the quality monitoring index specified in step 1, select the corresponding calculation algorithm from the monitoring index algorithm library, and use the benchmark data configured in step 1 to calculate the monitoring index of the generated benchmark;

[0098] Step 3. Real-time calculation of model quality indicators: Because it is necessary to calculate on a large amount of data, it will take a certain amount of time and cannot be calculated in quasi-real time. Therefore, it exists in the form of a scheduled task. The entire task is subdivided into two steps:

[0099] 3a. Collect online traffic request and response data by time window, that is, actual production environment data within a period of time, which is used as data input for calculating model quality indicators at this moment;

[0100] 3b. According to the quality monitoring indicators configured in step 1, select the corresponding calculation algorithm from the monitoring indicator algorithm library, and use the online quasi-real-time data set collected in step 3a to calculate the model quality monitoring indicators at the current moment;

[0101] Step 4. Aggregation and statistics of monitoring data: Store the request response data of the production environment, the benchmark indicators produced in step 2, and the real-time indicators produced in step 3, and associate them to facilitate subsequent monitoring and display;

[0102] Step 5. Learn the changing patterns of quality indicators through AI capabilities: Since step 3 is executed regularly, it will continuously produce monitoring indicator data at multiple moments. Through AI regression, time series prediction and other models, learn the changing patterns of fitting indicators over time in a directional or periodic manner, and continuously optimize the model effect over time; and use the model to predict the quality indicators at N moments in the future;

[0103] Step 6. Monitoring data comparison and alarm: Through steps 4 and 5, the real-time model quality indicators and indicators for the next N moments can be continuously obtained; according to the alarm strategy set in step 1, the obtained indicators are compared with the benchmark indicators, and it is determined whether the deviation of the indicators exceeds the set tolerance range. If it exceeds, an alarm is issued according to the alarm method set in step 1, and the service owner is notified to pay attention, and re-iterate training is performed to obtain a model that is more in line with the actual production environment.

[0104] The above-mentioned model quality architecture can provide customized AI model monitoring services with excellent performance and stable operation. In view of the process characteristics of model quality monitoring, an automatic and intelligent model quality monitoring process is given. By decomposing the entire model monitoring process, several core modules are abstracted to be responsible for the functional responsibilities of each stage of model monitoring, and a set of automated execution processes are designed to ensure that the entire model monitoring capability can operate smoothly and automatically. The modular design facilitates the continuous enrichment of the strategies and functions of related core modules in actual business scenarios. At the same time, in the overall process, a monitoring indicator intelligent learning module is designed. By continuously learning and fitting the changing rules of monitoring indicators, the indicator data for a period of time in the future is predicted and monitored, so as to achieve the goal of rapid and early monitoring and alarming, and avoid the real impact of model quality changes on actual business.

[0105] It should be understood that Figure 3 The architecture diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 3 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0106] The present disclosure embodiment discloses a model quality monitoring device, such as Figure 4 As shown, the model quality monitoring device includes: a first acquisition module 401, used to obtain first quality indicator data of the target model; a second acquisition module 402, used to obtain input data and output data of the target model within a preset time period; a first determination module 403, used to determine second quality indicator data of the target model using a target algorithm based on the input data and the output data within the preset time period; a monitoring module 404, used to compare the difference between the second quality indicator data and the first quality indicator data, and to issue an alarm when the difference exceeds a preset range.

[0107] In some embodiments, the first acquisition module 401 includes: a first acquisition submodule, used to acquire monitoring type and benchmark data; a first matching submodule, used to determine a target algorithm matching the monitoring type from an indicator algorithm library; and a first determination submodule, used to determine the first quality indicator data of the target model using the target algorithm based on the benchmark data.

[0108] In some embodiments, the preset time period includes multiple first moments; the first determination module 403 is also used to determine the second quality indicator data corresponding to the multiple first moments respectively based on the input data and the output data corresponding to the multiple first moments respectively, using the target algorithm.

[0109] In some embodiments, the monitoring module 404 includes: a first comparison submodule, used to compare the second quality indicator data corresponding to multiple first moments with the first quality indicator data to obtain the first differences corresponding to the multiple first moments; a first alarm submodule, used to respond to the existence of a first difference that exceeds a first preset range in the first differences corresponding to the multiple first moments, and to issue an alarm using a first alarm method.

[0110] In some embodiments, Figure 5 As shown, the model quality monitoring device also includes: a prediction module 405, which is used to predict the second quality indicator data at the second moment based on the second quality indicator data corresponding to the multiple first moments respectively, using a prediction model selected from a prediction model library, and the second moment is after the multiple first moments and the second moment is outside the preset time period.

[0111] In some embodiments, the monitoring module 404 includes: a second comparison submodule, used to compare the second quality indicator data at the second moment with the first quality indicator data to obtain a second difference; a second alarm submodule, used to use a second alarm method to issue an alarm when the second difference exceeds a second preset range.

[0112] Those skilled in the art should understand that the functions of each processing module in the model quality monitoring device of the embodiment of the present disclosure can be understood with reference to the relevant description of the aforementioned model quality monitoring method. Each processing module in the model quality monitoring device of the embodiment of the present disclosure can be realized by an analog circuit that realizes the functions described in the embodiment of the present disclosure, or can be realized by running software that executes the functions described in the embodiment of the present disclosure on an electronic device.

[0113] The model quality monitoring device of this embodiment can intelligently monitor the quality of the model without the need for human intervention and business intrusion, and can quickly and timely feedback the quality changes of the model, thereby improving the efficiency of model quality monitoring.

[0114] Figure 6 The schematic diagram of the model quality monitoring scenario is shown. Figure 6 It can be seen that the electronic device, such as the cloud server, receives the model quality monitoring request imported from each terminal, and the model quality monitoring request may include the service name, benchmark data, monitoring index, alarm strategy and alarm method of the specified monitored target model; the electronic device obtains the first quality index data of the target model of each terminal based on the model quality monitoring request imported by each terminal; obtains the input data and output data of the target model within the preset time period of each terminal; based on the input data and output data of each terminal within the preset time period, the target algorithm of each terminal is used to determine the second quality index data of each target model; compares the difference between the second quality index data of the target model of each terminal and the first quality index data, and issues an alarm when the difference exceeds the preset range. In addition, the electronic device predicts the second quality index data of the second moment based on the second quality index data corresponding to the multiple first moments, using the prediction model selected from the prediction model library; compares the second quality index data of the second moment with the first quality index data to obtain the second difference; and issues an alarm using the second alarm method when the second difference exceeds the second preset range.

[0115] The present disclosure does not limit the number of terminals or electronic devices, and actual applications may include multiple terminals and multiple electronic devices.

[0116] It should be understood that Figure 6 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 6 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0117] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0118] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0119] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0120] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0121] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0122] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the model quality monitoring method. For example, in some embodiments, the model quality monitoring method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the model quality monitoring method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the model quality monitoring method in any other appropriate manner (eg, by means of firmware).

[0123] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0124] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0125] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0126] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0127] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0128] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on the respective computers and having a client and server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0129] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0130] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model quality monitoring method, comprising: Acquire monitoring type and benchmark data; the monitoring type includes abnormal point detection monitoring or offset detection monitoring; The reference data is data used to determine the first quality indicator data, and the reference data includes reference input data and reference output data; Determine a target algorithm matching the monitoring type from an indicator algorithm library; Determining first quality indicator data of a target model using the target algorithm according to the benchmark data; Acquire first quality indicator data of the target model; Obtaining input data and output data of the target model within a preset time period; Determining second quality indicator data of the target model using a target algorithm based on the input data and the output data within the preset time period; The difference between the second quality indicator data and the first quality indicator data is compared, and an alarm is issued when the difference exceeds a preset range.

2. The method according to claim 1, wherein: The preset time period includes a plurality of first moments; the second quality indicator data of the target model is determined by using a target algorithm based on the input data and the output data within the preset time period, including: Based on the input data and the output data respectively corresponding to the multiple first moments, the second quality indicator data respectively corresponding to the multiple first moments are determined using the target algorithm.

3. The method according to claim 2, wherein: The comparing the difference between the second quality indicator data and the first quality indicator data, and issuing an alarm when the difference exceeds a preset range, includes: Comparing the second quality indicator data respectively corresponding to the plurality of first moments with the first quality indicator data to obtain first differences respectively corresponding to the plurality of first moments; In response to the first differences corresponding to the plurality of first moments respectively having a first difference that exceeds a first preset range, a first alarm method is used to issue an alarm.

4. The method according to claim 2, further comprising: Based on the second quality indicator data corresponding to the multiple first moments respectively, the second quality indicator data at the second moment is predicted using the prediction model selected from the prediction model library. The second moment is after the multiple first moments and is outside the preset time period.

5. The method according to claim 2, wherein: The comparing the difference between the second quality indicator data and the first quality indicator data, and issuing an alarm when the difference exceeds a preset range, includes: Comparing the second quality indicator data at the second moment with the first quality indicator data to obtain a second difference; When the second difference exceeds a second preset range, a second alarm method is used to issue an alarm.

6. A model quality monitoring device, comprising: A first acquisition module is used to acquire first quality indicator data of a target model; wherein the first acquisition module includes: a first acquisition submodule, used to acquire a monitoring type and benchmark data; a first matching submodule, used to determine a target algorithm matching the monitoring type from an indicator algorithm library; a first determination submodule, used to determine the first quality indicator data of a target model using the target algorithm according to the benchmark data; wherein the monitoring type includes anomaly detection monitoring or offset detection monitoring; the benchmark data is data used to determine the first quality indicator data, and the benchmark data includes benchmark input data and benchmark output data; A second acquisition module, used to acquire input data and output data of the target model within a preset time period; A first determination module, configured to determine second quality indicator data of the target model using a target algorithm based on the input data and the output data within the preset time period; The monitoring module is used to compare the difference between the second quality indicator data and the first quality indicator data, and to issue an alarm when the difference exceeds a preset range.

7. The device according to claim 6, wherein: The preset time period includes multiple first moments; the first determination module is also used to determine the second quality indicator data corresponding to the multiple first moments respectively based on the input data and the output data corresponding to the multiple first moments respectively, using the target algorithm.

8. The device according to claim 7, wherein: The monitoring module comprises: A first comparison submodule, configured to compare the second quality indicator data respectively corresponding to the plurality of first moments with the first quality indicator data to obtain first differences respectively corresponding to the plurality of first moments; The first alarm submodule is configured to generate an alarm in a first alarm mode in response to a first difference exceeding a first preset range among the first differences respectively corresponding to the plurality of first moments.

9. The apparatus according to claim 7, further comprising: A prediction module is used to predict the second quality indicator data of the second moment based on the second quality indicator data corresponding to the multiple first moments respectively, using a prediction model selected from a prediction model library, where the second moment is after the multiple first moments and the second moment is outside the preset time period.

10. The device according to claim 7, wherein: The monitoring module comprises: A second comparison submodule, used to compare the second quality indicator data at a second moment with the first quality indicator data to obtain a second difference; The second alarm submodule is used to use a second alarm method to issue an alarm when the second difference exceeds a second preset range.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-5.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images