AI model screening method and device, electronic equipment and storage medium

Abstract

The invention provides an AI model screening method and device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the steps that evaluation indexes corresponding to a plurality of AI models are obtained, the AI models comprise at least one AI model of an AI capacity type, and the evaluation indexes comprise at least one of a calling frequency index of the AI models of the same type, a calling frequency index of the AI models of all types, an accurate value index and a variance index; obtaining an application value evaluation result of each AI model through the evaluation indexes; screening each type of AI models according to an application value evaluation result to obtain each target AI model, wherein each type of AI model corresponds to one target AI model; and determining an AI model screening result according to each target AI model. According to the method, the application value of each AI model is evaluated according to the evaluation indexes, so that the accuracy and reliability of AI model application value evaluation and AI model screening are improved, and the user experience is ensured.

Description

Background Art

[0002] With the development of computer technology, AI models with various AI (Artificial Intelligence) capabilities have been developed and applied in large quantities. Therefore, an AI platform or AI system is needed to manage AI models.

[0003] In related technologies, due to the development of a large number of AI platforms and AI systems, AI capabilities are repeatedly built and fragmented. Therefore, when users need to obtain AI models with corresponding AI capabilities, the corresponding AI platform or AI system may not be able to uniformly manage various AI models, resulting in the inability to recommend the AI ​​model with the highest application value to users.

[0004] Therefore, there is an urgent need for an AI model screening method to uniformly manage various AI models and screen out the most matching AI models for users, improve the accuracy and reliability of AI model screening, and ensure user experience.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present disclosure provides an AI model screening method, device, electronic device and storage medium, which at least to a certain extent overcome the problem in related technologies that the corresponding AI platform or AI system cannot uniformly manage various AI models, resulting in an inability to recommend the AI ​​model with the highest application value to users.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to one aspect of the present disclosure, there is provided an AI model screening method, including: obtaining evaluation indicators corresponding to multiple AI models, wherein the multiple AI models include at least one AI model of an AI capability type, and the evaluation indicators include at least one of an indicator of the number of calls of the same type of AI models, an indicator of the number of calls of each type of AI models, an accuracy indicator, and a variance indicator; performing application value evaluation on the multiple AI models respectively through the evaluation indicators to obtain application value evaluation results of each AI model; screening each type of AI model according to the application value evaluation results to obtain each target AI model, wherein each type of AI model corresponds to a target AI model; and determining the AI ​​model screening results according to each target AI model.

[0009] In some exemplary embodiments, the application value of multiple AI models is evaluated respectively by the evaluation indicator to obtain the application value evaluation results of each AI model, including: when the evaluation indicator includes the call number indicator of the same type of AI model, the type call ratio of the actual call number of each AI model to the actual call number of the same type of model is calculated; when the evaluation indicator includes the call number indicator of each type of AI model, the total call ratio of the actual call number of the AI ​​model with the largest number of calls in the total call number of each type of model is calculated; when the evaluation indicator includes the accuracy indicator, the accuracy difference between the original accuracy of each AI model before the historical call and the actual accuracy after the historical call is calculated; when the evaluation indicator includes the variance indicator, the variance difference between the original variance of each AI model before the historical call and the actual variance after the historical call is calculated; the application value evaluation results of each AI model are determined according to at least one of the type call ratio, the total call ratio, the accuracy difference, and the variance difference.

[0010] In some exemplary embodiments, the application value evaluation result of each AI model is determined according to at least one of the type call ratio, the total call ratio, the accuracy difference, and the variance difference, including: determining the actual application value level of each AI model according to at least one of the type call ratio, the total call ratio, the accuracy difference, and the variance difference, and the actual application value level is used to describe the corresponding application value evaluation result.

[0011] In some exemplary embodiments, each type of AI model is screened according to the application value evaluation results to obtain target AI models, including: taking the one with the highest application value evaluation result in each type of AI model as the target AI model; or, when the evaluation index includes the call count index of the same type of AI model, taking the one with the largest number of historical calls in each type of AI model as the target AI model.

[0012] In some exemplary embodiments, the AI ​​model screening method provided by the embodiments of the present disclosure may also include: when the application value evaluation result of any target AI model does not meet the preset conditions, retraining the any target AI model to obtain the corresponding target AI model that meets the preset conditions.

[0013] In some exemplary embodiments, when the evaluation index includes the variance index, the preset condition may be that the variance difference is lower than a preset threshold.

[0014] In some exemplary embodiments, after the application value of multiple AI models is evaluated separately using the evaluation index and the application value evaluation results of each AI model are obtained, the AI ​​model screening method provided by the embodiments of the present disclosure may also include: obtaining the actual number of calls of each AI model within the current preset time interval at every preset time interval; and adjusting the application value evaluation result according to the actual number of calls within the current preset time interval.

[0015] In some exemplary embodiments, the AI ​​model screening method provided by the embodiments of the present disclosure may also include: obtaining an AI capability request sent by a user; determining at least one AI model to be obtained corresponding to the AI ​​capability request in the AI ​​model screening results based on the AI ​​capability request; and sending at least one AI model to be obtained corresponding to the AI ​​capability request to the user.

[0016] According to another aspect of the present disclosure, an AI model screening device is also provided, including: an evaluation index acquisition module, used to obtain evaluation indexes corresponding to multiple AI models, wherein the multiple AI models include at least one AI model of AI capability type, and the evaluation index includes at least one of the same type of AI model call count index, each type of AI model call count index, accuracy index, and variance index; an application value evaluation module, used to perform application value evaluation on multiple AI models respectively through the evaluation index to obtain application value evaluation results of each AI model; an AI model screening module, used to screen each type of AI model according to the application value evaluation results to obtain each target AI model, wherein each type of AI model corresponds to a target AI model; an AI model screening result determination module, used to determine the AI ​​model screening results according to each target AI model.

[0017] In some exemplary embodiments, the application value evaluation module is used to calculate the type call ratio of the actual number of calls of each AI model to the actual number of calls of the same type of models when the evaluation indicator includes the number of calls of the same type of AI model; when the evaluation indicator includes the number of calls of each type of AI model, calculate the total call ratio of the actual number of calls of the type of AI model with the largest number of calls in the total number of calls of each type of model; when the evaluation indicator includes the accuracy indicator, calculate the accuracy difference between the original accuracy of each AI model before the history was called and the actual accuracy after the history was called; when the evaluation indicator includes the variance indicator, calculate the variance difference between the original variance of each AI model before the history was called and the actual variance after the history was called; determine the application value evaluation result of each AI model according to at least one of the type call ratio, the total call ratio, the accuracy difference, and the variance difference.

[0018] In some exemplary embodiments, the application value assessment module is used to determine the actual application value level of each AI model based on at least one of the type call ratio, the total call ratio, the accuracy difference, and the variance difference, and the actual application value level is used to describe the corresponding application value assessment result.

[0019] In some exemplary embodiments, the AI ​​model screening module is used to select the one with the highest application value evaluation result in each type of AI model as the target AI model; or, when the evaluation index includes the call count index of the same type of AI model, the one with the largest number of historical calls in each type of AI model is used as the target AI model.

[0020] In some exemplary embodiments, the AI ​​model screening device provided by the embodiments of the present disclosure may also include: a retraining module, which is used to retrain any target AI model when the application value evaluation result of any target AI model does not meet the preset conditions, so as to obtain the corresponding target AI model that meets the preset conditions.

[0021] In some exemplary embodiments, when the evaluation index includes an accuracy index, the preset condition is that the accuracy difference is lower than a preset threshold.

[0022] In some exemplary embodiments, the AI ​​model screening device provided by the embodiments of the present disclosure may also include: an application value assessment result adjustment module, which is used to obtain the actual number of calls of each AI model within the current preset time interval at every preset time interval; and adjust the application value assessment result according to the actual number of calls within the current preset time interval.

[0023] In some exemplary embodiments, the AI ​​model screening device provided by the embodiments of the present disclosure may also include: a module for sending AI models to be acquired, used to acquire an AI capability request sent by a user; determining, according to the AI ​​capability request, at least one AI model to be acquired in the AI ​​model screening results corresponding to the AI ​​capability request; and sending at least one AI model to be acquired corresponding to the AI ​​capability request to the user.

[0024] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned AI model screening methods by executing the executable instructions.

[0025] According to another aspect of the present disclosure, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the AI ​​model screening method described in any one of the above is implemented.

[0026] According to another aspect of the present disclosure, a computer program product or computer program is provided, the computer program product or computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the AI ​​model screening method provided in various optional ways in any embodiment of the present disclosure.

[0027] The technical solution provided in the embodiments of the present disclosure can evaluate the application value of each AI model based on evaluation indicators such as the number of calls of the same type of AI model, the number of calls of each type of AI model, the accuracy index, the variance index, etc., so as to achieve unified management of each AI model. The present disclosure improves the accuracy and reliability of the application value evaluation of the AI ​​model, thereby improving the accuracy and reliability of the AI ​​model screening, ensuring the user recommendation effect, and improving the user experience.

[0028] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0030] Figure 1 A schematic diagram showing the system architecture of an AI model screening method in an embodiment of the present disclosure is shown;

[0031] Figure 2 A flow chart of an AI model screening method in an embodiment of the present disclosure is shown;

[0032] Figure 3 A schematic diagram of an AI model screening system in an embodiment of the present disclosure is shown;

[0033] Figure 4 A schematic diagram of an AI model screening system in an embodiment of the present disclosure is shown;

[0034] Figure 5 A flowchart of a method for determining an AI model screening result in an embodiment of the present disclosure is shown;

[0035] Figure 6 A schematic diagram of an AI model screening device in an embodiment of the present disclosure is shown;

[0036] Figure 7A schematic diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0038] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0039] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0040] Figure 1 FIG. 1 shows an exemplary system architecture diagram to which the AI ​​model screening method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0041] In an exemplary embodiment, the terminal device 101 can obtain evaluation indicators corresponding to multiple AI models, wherein the multiple AI models include at least one AI model of an AI capability type, and the evaluation indicators include at least one of an index of the number of calls to the same type of AI models, an index of the number of calls to each type of AI model, an accuracy index, and a variance index.

[0042] Then, the terminal device 101 can evaluate the application value of multiple AI models respectively through the evaluation index to obtain the application value evaluation results of each AI model. And each type of AI model can be screened according to the application value evaluation results to obtain each target AI model, wherein each type of AI model corresponds to a target AI model.

[0043] Finally, the terminal device 101 can determine the AI ​​model screening results according to each target AI model, and send the AI ​​model screening results to the server 103.

[0044] In another exemplary embodiment, the server 103 can obtain evaluation indicators corresponding to multiple AI models, wherein the multiple AI models include at least one AI model of an AI capability type, and the evaluation indicators include at least one of an indicator of the number of calls to the same type of AI models, an indicator of the number of calls to each type of AI models, an accuracy indicator, and a variance indicator.

[0045] Then, the server 103 can evaluate the application value of multiple AI models respectively according to the evaluation index to obtain the application value evaluation results of each AI model. And each type of AI model can be screened according to the application value evaluation results to obtain each target AI model, wherein each type of AI model corresponds to a target AI model.

[0046] Finally, the server 103 can determine the AI ​​model screening results according to each target AI model, and send the AI ​​model screening results to the terminal device 101.

[0047] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0048] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0049] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, wearable devices, augmented reality devices, virtual reality devices, etc.

[0050] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0051] Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0052] Those skilled in the art will know that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.

[0053] Under the above system architecture, an AI model screening method is provided in an embodiment of the present disclosure, which can be executed by any electronic device with computing and processing capabilities.

[0054] In some embodiments, the AI ​​model screening method provided in the embodiments of the present disclosure can be executed by a terminal device of the above-mentioned system architecture; in other embodiments, the AI ​​model screening method provided in the embodiments of the present disclosure can be executed by a server in the above-mentioned system architecture; in other embodiments, the AI ​​model screening method provided in the embodiments of the present disclosure can be implemented by the terminal device and server in the above-mentioned system architecture through interaction.

[0055] Figure 1 A flow chart of an AI model screening method in an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the AI ​​model screening method provided in the embodiment of the present disclosure includes the following steps S202 to S208.

[0056] S202, obtaining evaluation indicators corresponding to multiple AI models, wherein the multiple AI models include at least one AI model of an AI capability type, and the evaluation indicators include at least one of an indicator of the number of calls to the same type of AI models, an indicator of the number of calls to each type of AI model, an accuracy indicator, and a variance indicator.

[0057] The embodiments of the present disclosure do not limit the AI ​​capabilities corresponding to the AI ​​model. For example, the AI ​​capabilities may include traffic prediction capabilities, network status monitoring capabilities, data encryption capabilities, etc. Therefore, the AI ​​capability types of the AI ​​model may correspond to traffic prediction types, network status monitoring types, data encryption types, etc.

[0058] In some embodiments, taking the AI ​​capability type of traffic prediction capability as an example, the traffic prediction capability can be subdivided into the capability to predict various types of traffic. Therefore, the traffic prediction capability can also be regarded as an overall classification of an AI capability, and an overall classification of an AI capability can correspond to multiple AI model types. The AI ​​capability type of one AI model can correspond to the capability to predict any type of traffic.

[0059] In a possible implementation, the application scope indicators of the AI ​​models obtained by each user in a certain AI system or AI platform over a period of time may be collected. For example, the number of users collected may be 10 or more.

[0060] In an exemplary embodiment, the AI ​​models obtained by each user can be multiple AI models to be screened. The above-mentioned application scope indicator can include the overall classification of AI capabilities. Exemplarily, the overall classification of the AI ​​capabilities can be represented by classification labels. The overall classification is used to indicate that after obtaining the overall classification of each AI capability, the AI ​​capabilities can be classified based on the machine learning algorithm combined with the classification labels, so as to confirm the AI ​​capability type corresponding to each AI model.

[0061] In an exemplary embodiment, the above-mentioned application scope index may also include at least one of ranking information of similar AI models and recommended AI model information. The ranking information of the same AI model is used to indicate the ranking of the historical actual call times corresponding to each type of AI model. The recommended AI model information may be a historical AI model screening result.

[0062] In some embodiments, after determining multiple AI models to be screened, the evaluation indicators corresponding to each AI model can be obtained. It should be noted that when the evaluation indicator includes the call count indicator of the same type of AI model, it is necessary to obtain the actual call count of each AI model. When the evaluation indicator includes the call count indicator of each type of AI model, it is necessary to calculate the total call count of each type of model and determine the actual call count of the AI ​​model with the most calls.

[0063] In addition, when the evaluation index includes an accuracy index, it is necessary to determine the original accuracy of each AI model history before it is called, and the actual accuracy of each AI model history after it is called. When the evaluation index includes a variance index, the original variance of each AI model history before it is called, and the actual variance of each AI model history after it is called.

[0064] S204, performing application value evaluation on multiple AI models respectively through evaluation indicators to obtain application value evaluation results of each AI model.

[0065] In some embodiments, application value evaluation is performed on multiple AI models respectively through evaluation indicators to obtain application value evaluation results of each AI model, including: when the evaluation indicator includes the call count indicator of the same type of AI model, the type call ratio of the actual call count of each AI model to the actual call count of the same type of model is calculated; when the evaluation indicator includes the call count indicator of each type of AI model, the total call ratio of the actual call count of the AI ​​model with the largest number of calls in the total call count of each type of model is calculated; when the evaluation indicator includes the accuracy indicator, the accuracy difference between the original accuracy of each AI model before the historical call and the actual accuracy after the historical call is calculated; when the evaluation indicator includes the variance indicator, the variance difference between the original variance of each AI model before the historical call and the actual variance after the historical call is calculated; the application value evaluation result of each AI model is determined according to at least one of the type call ratio, the total call ratio, the accuracy difference, and the variance difference.

[0066] In a possible implementation, when the evaluation index includes an index of the number of calls of the same type of AI model, the value of the type call ratio may be a percentage greater than 0, such as 90%, 75.5%, 61.72%, etc. When the evaluation index includes an index of the number of calls of each type of AI model, the value of the total call ratio may be a percentage greater than, such as 90%, 75.5%, 61.72%, etc. When the evaluation index includes an accuracy index, the value of the accuracy difference may be, for example, 50%, 11.4%, 0%, -20.75%, etc. When the evaluation index includes a variance index, the value of the variance difference may be 20%, 5.2%, 0%, -13.42%, etc.

[0067] It should be noted that the above original variance and actual variance may both refer to mean square absolute error.

[0068] In a possible implementation, a corresponding weight value may be set for at least one of the type call ratio, the total call ratio, the accuracy difference, and the variance difference, respectively, so as to calculate the application value evaluation result according to the corresponding weight value.

[0069] Alternatively, in some embodiments, the application value evaluation result of each AI model is determined according to at least one of the type call ratio, the total call ratio, the accuracy difference, and the variance difference, which may include: determining the actual application value level of each AI model according to at least one of the type call ratio, the total call ratio, the accuracy difference, and the variance difference, and the actual application value level is used to describe the corresponding application value evaluation result.

[0070] In an exemplary embodiment, the actual application value level may include four levels: excellent, medium, good, and poor. Exemplarily, let the type call ratio be ai, let the total call ratio be bi, let the accuracy difference be ci, and let the variance difference be di.

[0071] In this case, when ai>=90% or bi>=90%, and ci>=0% or di<=0, the corresponding actual application value level can be considered excellent. When 80%<=ai<90% or 80%<=bi<90% or -10%<=ci<0%, the corresponding actual application value level can be medium. When 70%<=ai<80% or 70%<=bi<80%, and -20%<=ci<-10%, the corresponding actual application value level can be good. When ai<70% or bi<70%, and ci<-20%, the corresponding actual application value level can be poor.

[0072] It should be noted that in the embodiments of the present disclosure, the actual application value level and the division thresholds corresponding to each actual application value level can be set according to the application scenario or experience, and the embodiments of the present disclosure are not limited to this.

[0073] In some embodiments, after the application value of multiple AI models is evaluated separately through evaluation indicators and the application value evaluation results of each AI model are obtained, the AI ​​model screening method provided by the embodiments of the present disclosure may also include: at every preset time interval, obtaining the actual number of calls of each AI model within the current preset time interval; adjusting the application value evaluation results according to the actual number of calls within the current preset time interval.

[0074] Exemplarily, the type call ratio and / or total call ratio can be re-determined based on the actual number of calls of each AI model within the current preset time interval, thereby re-determining the application value evaluation result.

[0075] Alternatively, for example, when the application value evaluation result is represented by the actual application value level, if the actual number of calls of an AI model of a certain AI capability type within a preset time interval is the type with the largest number of actual calls among all types of AI models, the actual application value level corresponding to the AI ​​model of the AI ​​capability type can be adjusted to excellent.

[0076] The embodiment of the present disclosure does not limit the length of the preset time interval, and the length of the preset time interval can be set according to the application scenario or experience.

[0077] The method provided in the embodiment of the present disclosure can evaluate the application value of the AI ​​model based on the application scope index, call times, accuracy, and variance of the AI ​​capability, so as to recommend the AI ​​model with the highest application value to the user, and can effectively evaluate the functions of the corresponding AI system or AI platform, so as to realize the unified management of various AI models. Therefore, the embodiment of the present disclosure can improve the accuracy and reliability of AI model screening, ensure the user recommendation effect, and improve the user experience.

[0078] S206: Screen each type of AI model according to the application value evaluation results to obtain target AI models, wherein each type of AI model corresponds to a target AI model.

[0079] In some embodiments, each type of AI model is screened according to the application value evaluation results to obtain each target AI model, which may include: taking the one with the highest application value evaluation result in each type of AI model as the target AI model; or, when the evaluation index includes the number of calls of the same type of AI model, taking the one with the largest number of historical calls in each type of AI model as the target AI model.

[0080] In a possible implementation, the average accuracy of each AI model can be calculated, and the AI ​​model with the highest average accuracy and the most actual calls among all types of AI models can be used as the target AI model corresponding to the AI ​​model of this type. In addition, other AI models in this type of AI model can also be deleted.

[0081] For example, if the actual number of calls of an AI model with a lower average accuracy in a certain category of AI models is the highest among the AI ​​models in this category, it can be said that the average accuracy requirement of this category of AI models is not high. In this case, the AI ​​model with the highest actual number of calls can be used as the target AI model corresponding to this category of AI models.

[0082] In an exemplary embodiment, when the application value evaluation result is represented by the actual application value level, the AI ​​model with the highest actual application value level in each type of AI model can be used as the target AI model corresponding to the AI ​​model of that type.

[0083] In an exemplary embodiment, when obtaining recommended AI model information, if the actual application value level of the target AI model corresponding to a certain type of AI model is poor, the target AI model can be changed to a suboptimal AI model included in the recommended AI model information. In addition, when the actual application value levels of all AI models in the type of AI model included in the recommended AI model information are poor, the AI ​​model with the highest actual application value level in the type of AI model can still be used as the target AI model corresponding to the type of AI model.

[0084] S208: Determine AI model screening results according to each target AI model.

[0085] In some embodiments, the AI ​​model screening method provided by the embodiments of the present disclosure may also include: obtaining an AI capability request sent by a user; determining at least one AI model to be obtained in the AI ​​model screening results corresponding to the AI ​​capability request based on the AI ​​capability request; and sending at least one AI model to be obtained corresponding to the AI ​​capability request to the user.

[0086] In an exemplary embodiment, the target AI models corresponding to various AI models can be summarized as the AI ​​model screening result. When an AI capability request sent by a user is received, at least one AI capability specifically requested in the AI ​​capability request can be determined, so that at least one target AI model corresponding to the AI ​​capability request can be obtained from the AI ​​model screening result. The at least one target AI model is returned to the user as the AI ​​model to be obtained.

[0087] In some embodiments, sending at least one AI model to be acquired corresponding to the AI ​​capability request to the user may include: sharing at least one target AI model with the user.

[0088] Exemplarily, the sharing method may include: at least one of standardized sharing, retraining sharing, inference calling, model embedded reference, and image downloading.

[0089] Among them, the standardized sharing method can be a standardized AI capability, and standardized sharing can be used for direct sharing through intelligent asset management. The retraining sharing method can be used to retrain the managed AI capabilities so that they can be reused in different regions, professions, and scenarios. The reasoning call method can be used to call the corresponding AI model through the API (Application Program Interface) interface, so as to perform model reasoning in the cloud. The model embedded reference method can be used to package and pack AI models and embed them into other systems. The image download method can be used to package and download AI models and their operating environment to other systems, and other systems provide computing resources.

[0090] In some embodiments, the AI ​​model screening method provided by the embodiments of the present disclosure may also include: when the application value evaluation result of any target AI model does not meet the preset conditions, retraining any target AI model to obtain the corresponding target AI model that meets the preset conditions.

[0091] In some embodiments, when the evaluation indicator includes an accuracy value indicator, the preset condition is that the accuracy difference is lower than a preset threshold.

[0092] In a possible implementation, a list of AI models with poor actual application value levels may be obtained. Exemplarily, the list of AI models also includes the type call ratio, total call ratio, and accuracy difference corresponding to each AI model with poor actual application value level.

[0093] Afterwards, it can be determined whether each target AI model is included in the AI ​​model list. If at least one target AI model is included in the AI ​​model list, the at least one target AI model can be retrained. Until the accuracy difference after training is higher than the preset threshold. The embodiment of the present disclosure does not limit the size of the preset threshold. Exemplarily, the preset threshold can be -20%.

[0094] In a possible implementation, at least one of the parameters, features, and training data sets of the AI ​​model included in the AI ​​model list may be optimized so that ci>=-20%.

[0095] The method provided by the embodiment of the present disclosure can evaluate the application value of each AI model based on evaluation indicators such as the number of calls of the same type of AI model, the number of calls of each type of AI model, the accuracy index, the variance index, etc., so as to achieve unified management of each AI model. The present disclosure improves the accuracy and reliability of the application value evaluation of the AI ​​model, thereby improving the accuracy and reliability of the AI ​​model screening, ensuring the user recommendation effect, and improving the user experience.

[0096] In some embodiments, a schematic diagram of an AI model screening system provided by an embodiment of the present disclosure can be as follows: Figure 3 shown.

[0097] exist Figure 3 In the AI ​​capability management system, users can send AI capability requests to the AI ​​capability management system through an external system. The AI ​​capability management system can determine the user's intention based on the AI ​​capability request, and feedback the user's intention to the user for verification. When the user confirms, at least one AI model to be acquired that the user needs can be determined based on the user's intention. The AI ​​capability management system includes an AI capability management module, an AI capability application value assessment module, an AI capability sharing module, and a system improvement plan customization module.

[0098] Among them, the AI ​​capability management module can store various AI models and record the application scope indicators, evaluation indicators, etc. corresponding to each AI model respectively, where the evaluation indicators may include the number of calls of each AI model, the accuracy of each AI model, etc.

[0099] Secondly, the AI ​​capability application value assessment module can be used to obtain the evaluation indicators corresponding to multiple AI models, and perform application value assessment on each AI model based on the evaluation indicators. For example, the AI ​​capability application value assessment module can first obtain the application scope indicator, the number of calls of each AI model, and the accuracy of each AI model, so as to calculate the actual application value level corresponding to each AI model.

[0100] Furthermore, the AI ​​capability sharing module can be used to submit an AI capability subscription application for at least one AI capability to be acquired corresponding to the user intent after the user confirms that the parsed user intent is correct. After approval, the user can download the AI ​​model to be acquired corresponding to the AI ​​capability to be acquired to the external system, for example, by means of an image or a model.

[0101] In addition, the system improvement program customization module can be used to retrain AI models with poor actual application value levels, thereby improving the accuracy of AI models. Moreover, the system improvement program customization module can screen various types of AI models based on the AI ​​capability application value evaluation results to obtain the optimal AI model screening results. In addition, the system improvement program customization module can obtain the actual number of calls of each AI model within the current preset time interval at every preset time interval, and adjust the application value evaluation results according to the actual number of calls within the current preset time interval. Therefore, it is possible to continuously optimize the AI ​​model screening results.

[0102] It should be noted that the external system involved in this disclosure is a separate system that can interact with the AI ​​capability management system through an interface to provide efficient and high-quality intelligent services, improve the efficiency of network operation management, and enhance customer perception. In addition, the AI ​​capability application value assessment module in the embodiment of this disclosure can be encapsulated as a tool for various industries and various AI capability management and sharing platforms or systems to apply, thereby realizing the wide application of AI model application value assessment.

[0103] The disclosed embodiments can propose improvement plans for the application value evaluation results based on the AI ​​model from the aspects of improving the accuracy of the AI ​​model, optimizing the selection of AI capabilities with the same functional category, optimizing the recommended plans, etc., and can recommend the AI ​​model with the best application value while meeting the user's needs.

[0104] In some embodiments, a flowchart of a method for determining an actual application value level provided by an embodiment of the present disclosure can be as follows: Figure 4 The actual application value level determination method may include steps S402 to S410.

[0105] S402, randomly collect evaluation indicators corresponding to the AI ​​models of a certain number of users within a period of time.

[0106] S404, calculating ai, bi, ci, and di corresponding to each AI model.

[0107] Exemplarily, ai may be a type call ratio, bi may be a total call ratio, ci may be an accuracy difference, and di may be a variance difference.

[0108] S406: Evaluate the application value of each AI model.

[0109] S408 outputs four practical application value levels: excellent, medium, good and poor.

[0110] S410, obtaining the actual number of calls of each AI model within the current preset time interval at every preset time interval, and adjusting the application value evaluation result according to the actual number of calls within the current preset time interval.

[0111] It should be noted that the implementation method of the above steps can refer to the above S202 to S204, which will not be repeated here.

[0112] In some embodiments, a flowchart of an AI model screening result determination method provided by an embodiment of the present disclosure can be as follows: Figure 5 The actual application value level determination method may include steps S502 to S512.

[0113] S502, obtaining the actual application value level and call count of each AI model.

[0114] S504: Filter each type of AI model according to the actual application value level and the number of calls to obtain each target AI model, and use each target AI model as the AI ​​model screening result.

[0115] S506, obtaining a list of AI models with a poor actual application value level, wherein the list of AI models may further include ai, bi, and ci corresponding to each AI model with a poor actual application value level.

[0116] Exemplarily, ai may be a type call ratio, bi may be a total call ratio, and ci may be an accuracy difference.

[0117] S508, retrain each AI model in the AI ​​model list until ci>=-20%.

[0118] S510, comparing the AI ​​model screening results with the list of AI models with poor actual application value levels.

[0119] S512: If any target AI model in the AI ​​model screening result is in the AI ​​model list, the corresponding type of suboptimal AI model will replace any of the above target AI models. If the actual application value level of the AI ​​model of this type is poor, the AI ​​model screening result will not be changed.

[0120] It should be noted that the implementation method of the above steps can refer to the above S206 to S208, which will not be repeated here.

[0121] Based on the same inventive concept, an AI model screening device is also provided in the embodiments of the present disclosure, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0122] Figure 6 A schematic diagram of an AI model screening device in an embodiment of the present disclosure is shown. Figure 6 As shown, the device comprises:

[0123] An evaluation index acquisition module 601 is used to acquire evaluation indexes corresponding to multiple AI models, wherein the multiple AI models include at least one AI model of an AI capability type, and the evaluation index includes at least one of an index of the number of calls of the same type of AI model, an index of the number of calls of each type of AI model, an accuracy index, and a variance index;

[0124] An application value evaluation module 602 is used to evaluate the application value of multiple AI models respectively according to the evaluation index to obtain the application value evaluation results of each AI model;

[0125] The AI ​​model screening module 603 is used to screen each type of AI model according to the application value evaluation result to obtain each target AI model, wherein each type of AI model corresponds to a target AI model;

[0126] The AI ​​model screening result determination module 604 is used to determine the AI ​​model screening results according to each target AI model.

[0127] In some exemplary embodiments, the application value evaluation module 602 is used to calculate the type call ratio of the actual number of calls of each AI model to the actual number of calls of the same type of models when the evaluation indicator includes the number of calls of the same type of AI model; when the evaluation indicator includes the number of calls of each type of AI model, calculate the total call ratio of the actual number of calls of the type of AI model with the largest number of calls in the total number of calls of each type of model; when the evaluation indicator includes the accuracy indicator, calculate the accuracy difference between the original accuracy of each AI model before the history was called and the actual accuracy after the history was called; when the evaluation indicator includes the variance indicator, calculate the variance difference between the original variance of each AI model before the history was called and the actual variance after the history was called; determine the application value evaluation result of each AI model according to at least one of the type call ratio, the total call ratio, the accuracy difference, and the variance difference.

[0128] In some exemplary embodiments, the application value assessment module 602 is used to determine the actual application value level of each AI model based on at least one of the type call ratio, the total call ratio, the accuracy difference, and the variance difference, and the actual application value level is used to describe the corresponding application value assessment result.

[0129] In some exemplary embodiments, the AI ​​model screening module 603 is used to select the one with the highest application value evaluation result in each type of AI model as the target AI model; or, when the evaluation index includes the call count index of the same type of AI model, the one with the largest number of historical calls in each type of AI model is used as the target AI model.

[0130] In some exemplary embodiments, the AI ​​model screening device provided by the embodiments of the present disclosure may further include:

[0131] The retraining module is used to retrain any target AI model when the application value evaluation result of any target AI model does not meet the preset conditions, so as to obtain the corresponding target AI model that meets the preset conditions.

[0132] In some exemplary embodiments, when the evaluation index includes an accuracy index, the preset condition is that the accuracy difference is lower than a preset threshold.

[0133] In some exemplary embodiments, the AI ​​model screening device provided by the embodiments of the present disclosure may further include:

[0134] The application value assessment result adjustment module is used to obtain the actual number of calls of each AI model within the current preset time interval at every preset time interval; and adjust the application value assessment result according to the actual number of calls within the current preset time interval.

[0135] In some exemplary embodiments, the AI ​​model screening device provided by the embodiments of the present disclosure may also include:

[0136] The module for sending AI models to be obtained is used to obtain an AI capability request sent by a user; determine at least one AI model to be obtained corresponding to the AI ​​capability request in the AI ​​model screening results according to the AI ​​capability request; and send at least one AI model to be obtained corresponding to the AI ​​capability request to the user.

[0137] It should be noted that the evaluation index acquisition module 601, the application value evaluation module 602, the AI ​​model screening module 603, and the AI ​​model screening result determination module 604 correspond to S202 to S208 in the method embodiment, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above method embodiment. It should be noted that the above modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0138] The device provided by the embodiment of the present disclosure can evaluate the application value of each AI model based on evaluation indicators such as the number of calls of the same type of AI model, the number of calls of each type of AI model, the accuracy index, the variance index, etc., so as to achieve unified management of each AI model. The present disclosure improves the accuracy and reliability of the application value evaluation of the AI ​​model, thereby improving the accuracy and reliability of the AI ​​model screening, ensuring the user recommendation effect, and improving the user experience.

[0139] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0140] Refer to the following Figure 7An electronic device 700 according to this embodiment of the present disclosure is described. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0141] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710).

[0142] Among them, the storage unit stores a program code, and the program code can be executed by the processing unit 710, so that the processing unit 710 executes the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 710 can execute the following steps of the above method embodiment: obtain evaluation indicators corresponding to multiple AI models, wherein the multiple AI models include at least one AI model of AI capability type, and the evaluation indicator includes at least one of the same type of AI model call number indicator, each type of AI model call number indicator, accuracy value indicator, and variance indicator; use the evaluation indicator to evaluate the application value of multiple AI models respectively, and obtain the application value evaluation results of each AI model; according to the application value evaluation results, each type of AI model is screened to obtain each target AI model, wherein each type of AI model corresponds to a target AI model; determine the AI ​​model screening results according to each target AI model.

[0143] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .

[0144] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0145] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0146] The electronic device 700 may also communicate with one or more external devices 740 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0147] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0148] In particular, according to an embodiment of the present disclosure, the process described in the above reference flowchart can be implemented as a computer program product, which includes: a computer program, which implements the above-mentioned AI model screening method when executed by a processor.

[0149] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. Exemplarily, a program product capable of implementing the above method of the present disclosure is stored on the computer-readable storage medium. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps described in the above “Exemplary Method” section of this specification according to various exemplary implementations of the present disclosure.

[0150] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0152] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0153] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0154] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0155] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0156] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0157] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope of the present disclosure is indicated by the appended claims.

Claims

1. An artificial intelligence (AI) model screening method, characterized in that: include: Obtaining evaluation indicators corresponding to multiple AI models, wherein the multiple AI models include at least one AI model of an AI capability type, and the evaluation indicators include at least one of an indicator of the number of calls of the same type of AI models, an indicator of the number of calls of each type of AI models, an accuracy indicator, and a variance indicator; The application value of multiple AI models is evaluated respectively by using the evaluation indicators to obtain the application value evaluation results of each AI model; According to the application value assessment results, each type of AI model is screened to obtain various target AI models, where each type of AI model corresponds to a target AI model; The AI ​​model screening results are determined based on each target AI model.

2. The AI ​​model screening method according to claim 1, characterized in that: The application value of the multiple AI models is evaluated respectively by the evaluation indicators to obtain the application value evaluation results of each AI model, including: When the evaluation index includes the same type of AI model call count index, the type call ratio of the actual call count of each AI model to the actual call count of the same type of models is calculated; When the evaluation index includes the call count index of each type of AI model, the total call ratio of the actual call count of the AI ​​model with the largest number of calls to the total call count of each type of model is calculated; When the evaluation index includes the accuracy index, the accuracy difference between the original accuracy of each AI model history before it is called and the actual accuracy after the history is called is calculated; When the evaluation index includes the variance index, the variance difference between the original variance of each AI model history before being called and the actual variance after the history is called is calculated; The application value evaluation result of each AI model is determined according to at least one of the type call ratio, the total call ratio, the accuracy difference, and the variance difference.

3. The AI ​​model screening method according to claim 2, characterized in that: The determining the application value evaluation result of each AI model according to at least one of the type call ratio, the total call ratio, the accuracy difference, and the variance difference, respectively, includes: The actual application value level of each AI model is determined based on at least one of the type call ratio, the total call ratio, the accuracy difference, and the variance difference, and the actual application value level is used to describe the corresponding application value evaluation result.

4. The AI ​​model screening method according to any one of claims 1 to 3, characterized in that: The target AI models are obtained by screening each type of AI model according to the application value evaluation results, including: The AI ​​model with the highest application value evaluation result in each category is used as the target AI model; Alternatively, when the evaluation index includes the call count index of the same type of AI model, the one with the largest number of historical calls in each type of AI model is used as the target AI model.

5. The AI ​​model screening method according to any one of claims 1 to 3, characterized in that: The method further comprises: When the application value evaluation result of any target AI model does not meet the preset conditions, the target AI model is retrained to obtain a corresponding target AI model that meets the preset conditions.

6. The AI ​​model screening method according to claim 5, characterized in that: When the evaluation index includes the accuracy value index, the preset condition is that the accuracy difference is lower than a preset threshold.

7. The AI ​​model screening method according to claim 2 or 3, characterized in that: After evaluating the application value of the multiple AI models respectively by using the evaluation indicators to obtain the application value evaluation results of each AI model, the method further includes: At every preset time interval, the actual number of calls of each AI model within the current preset time interval is obtained; The application value evaluation result is adjusted according to the actual number of calls within the current preset time interval.

8. The AI ​​model screening method according to claim 1, characterized in that: The method further comprises: Get AI capability requests sent by users; Determine, according to the AI ​​capability request, at least one AI model to be acquired in the AI ​​model screening result that corresponds to the AI ​​capability request; Sending at least one AI model to be acquired corresponding to the AI ​​capability request to the user.

9. An AI model screening device, characterized in that: include: An evaluation index acquisition module, used to acquire evaluation indexes corresponding to multiple AI models, wherein the multiple AI models include at least one AI model of an AI capability type, and the evaluation index includes at least one of an index of the number of calls of the same type of AI model, an index of the number of calls of each type of AI model, an accuracy index, and a variance index; An application value evaluation module, used to evaluate the application value of multiple AI models respectively according to the evaluation indicators, and obtain the application value evaluation results of each AI model; An AI model screening module is used to screen each type of AI model according to the application value evaluation results to obtain each target AI model, wherein each type of AI model corresponds to a target AI model; The AI ​​model screening result determination module is used to determine the AI ​​model screening results based on each target AI model.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the AI ​​model screening method described in any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the AI ​​model screening method described in any one of claims 1 to 8 is implemented.

Images