Model information analysis method and device, electronic equipment and computer storage medium

Abstract

The present disclosure provides a model information analysis method and device, electronic equipment and computer storage medium, relates to the technical field of computers, and particularly relates to the technical field of artificial intelligence such as information flow, machine learning and deep learning. The specific implementation scheme is as follows: a target model is acquired; the target model is one of multiple models called at a current moment; dependency relationship information between a target input feature and a predetermined to-be-verified feature is determined; the target input feature is one of multiple input features of the target model; and a model information analysis result is generated according to the dependency relationship information. The present disclosure helps to improve the management efficiency of models and model-related data, and helps to improve the use efficiency of models.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence technologies such as information flow, machine learning, and deep learning. Background Technology

[0002] With the development of computer technology, models are widely used in computer-related and network-related products. While models bring convenience to these products, over time, the accumulation of data related to model structure and input data increases the difficulty of managing the model, its features, and related fields, potentially impacting model efficiency and effectiveness. Therefore, it is necessary to improve model information analysis methods to reduce the processing difficulty of model-related data and enhance model usage. Summary of the Invention

[0003] This disclosure provides a model information analysis method, apparatus, electronic device, and computer storage medium.

[0004] According to one aspect of this disclosure, a model information analysis method is provided, comprising:

[0005] Obtain the target model; the target model is one of several models invoked at the current moment.

[0006] Determine the dependency information between the target input features and the predetermined features to be verified; the target input features are one of multiple input features of the target model;

[0007] Based on dependency information, generate model information analysis results.

[0008] According to another aspect of this disclosure, a model information analysis apparatus is provided, comprising:

[0009] The target model acquisition module is used to acquire the target model; the target model is one of several models that are invoked at the current moment.

[0010] The dependency information determination module is used to determine the dependency information between the target input feature and the predetermined feature to be verified; the target input feature is one of multiple input features of the target model;

[0011] The analysis results module is used to generate model information analysis results based on dependency information.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0013] At least one processor; and

[0014] The memory is communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods in any embodiment of this disclosure.

[0016] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods of any embodiment of this disclosure.

[0017] According to another aspect of this disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the methods in any embodiment of this disclosure.

[0018] According to the technology disclosed herein, it is possible to analyze the input features and features to be verified of the target model, determine the dependency information between the input features and features to be verified of the target model, help to identify features to be verified that are dependent on the input features of the current target model, and then find features that are not related to the model input features from the features to be verified, which helps to simplify the features to be verified, reduce useless features in the features to be verified, help to improve the management efficiency of the model and model-related data, and also help to improve the efficiency of model use.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0021] Figure 1 This is a schematic flowchart of a model information analysis method according to an embodiment of the present disclosure;

[0022] Figure 2 This is a schematic flowchart of a model information analysis method according to another embodiment of the present disclosure;

[0023] Figure 3 This is a flowchart illustrating a model information analysis method according to yet another embodiment of the present disclosure;

[0024] Figure 4 This is a schematic diagram of the model training process according to an example of this disclosure;

[0025] Figure 5 This is a schematic diagram of a model information analysis method based on an example of this disclosure;

[0026] Figure 6 This is a schematic diagram of feature dependency analysis based on an example of this disclosure;

[0027] Figure 7 This is a schematic diagram of a model information analysis device according to an embodiment of the present disclosure;

[0028] Figure 8 This is a schematic diagram of a model information analysis device according to another embodiment of the present disclosure;

[0029] Figure 9 This is a schematic diagram of a model information analysis device according to yet another embodiment of the present disclosure;

[0030] Figure 10 This is a schematic diagram of a model information analysis device according to yet another embodiment of the present disclosure;

[0031] Figure 11 This is a schematic diagram of a model information analysis device according to yet another embodiment of the present disclosure;

[0032] Figure 12 This is a schematic diagram of a model information analysis device according to yet another embodiment of the present disclosure;

[0033] Figure 13 This is a schematic diagram of a model information analysis device according to yet another embodiment of the present disclosure;

[0034] Figure 14 This is a schematic diagram of a model information analysis device according to yet another embodiment of the present disclosure;

[0035] Figure 15 This is a block diagram of an electronic device used to implement the model information analysis method of the embodiments of this disclosure. Detailed Implementation

[0036] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0037] In this embodiment of the disclosure, the model information is analyzed based on the possibility that the model name, the name of the model's features, and the features of the model may be modified during the model's use. The analysis results are generated and can serve as the basis for model management and model data management, thereby helping to improve the model's usage effect and efficiency.

[0038] The model information analysis method provided in this disclosure embodiment, such as Figure 1 As shown, it includes:

[0039] Step S11: Obtain the target model; the target model is one of the multiple models invoked at the current moment;

[0040] Step S12: Determine the dependency information between the target input feature and the predetermined feature to be verified; the target input feature is one of the multiple input features of the target model;

[0041] Step S13: Generate model information analysis results based on dependency information.

[0042] The model information analysis method provided in this embodiment can be implemented for a specific model management platform. For example, a specific user uses a specific model management platform to manage all product-related models and model-related data. The target model can be one of multiple models managed by the specific user on the model management platform. In one specific implementation, the model information analysis method provided in any embodiment of this disclosure can be implemented one by one on each model managed by the specific user on the model management platform.

[0043] In this embodiment, the multiple models invoked at the current moment may include multiple models currently in use. When this embodiment is applied to a model management platform, the multiple models invoked at the current moment may be multiple models managed by the model management platform and still in use. Models that were previously used but have been updated or discontinued for other reasons may be considered not to be among the models invoked at the current moment.

[0044] In one specific implementation, the multiple models invoked at the current moment may include an article recommendation model for recommending articles to users, a question-and-answer information processing model for processing user input questions, a video recommendation model for recommending videos to users, and a trending news filtering model for determining the trending search rankings, etc.

[0045] In this embodiment, features such as target input features and predetermined features to be verified can all be features of fields used as input data for the model. For example, for a user recommendation data prediction model, if information such as the user's network usage time is required as input data, then "network usage time" can be a model input feature. When this user recommendation data prediction model is the target model, "network usage time" can be a target input feature. Similarly, for a text recognition model, if the input data includes the text to be recognized, then "the text to be recognized" can be a target input feature.

[0046] In this embodiment, the predetermined feature to be verified can be a feature that needs to be determined based on the information of the target model to determine whether there are redundant or invalid features. When this embodiment of the disclosure is applied to a model management platform, the feature to be verified can be all the features recorded in the model management platform, that is, all the features accumulated since the user of the model management platform deployed the first model.

[0047] The features to be validated can include the input features of the currently invoked model and the input features of models that are not currently invoked. For example, for a specific user recommendation data prediction model, information such as user gender, network usage time, and geographical location are required as input data. Therefore, "gender," "network usage time," and "geographical location" serve as input features for the user recommendation data prediction model. Meanwhile, the initial version of the user recommendation model is model A, which has already been iterated by the current version of the user recommendation model, thus model A is in an unused state. The user recommendation data prediction model is one of several models currently invoked. Therefore, both the input features of model A and the input features of the user recommendation data prediction model can be used as features to be validated.

[0048] The features to be validated can include input features of the target model and input features of other models, as well as valid and unused features of the target model or other models. For example, for a specific user recommendation data prediction model, information such as user gender, network usage time, and geographical location are required as input data. Therefore, "gender," "network usage time," and "geographical location" are used as input features of the user recommendation data prediction model. However, in the initial stage of using the user recommendation data prediction model, features B, C, and D are also used as input data. Therefore, the features to be validated still include features B, C, and D.

[0049] In this embodiment, determining the dependency information between the target input feature and the predetermined feature to be verified may include determining whether the target input feature directly or indirectly depends on the feature to be verified. If so, the dependency information is then added to the dependency information to indicate that the target input feature depends on the feature to be verified. Alternatively, it may include determining whether the target input feature is directly or indirectly depended upon by the feature to be verified. If so, the dependency information is then added to indicate that the target input feature is depended upon by the feature to be verified.

[0050] In another possible implementation, determining the dependency information between the target input feature and the predetermined feature to be verified can include whether the target input feature depends on the feature to be verified, as well as other related features in the dependency relationship. For example, if the target input feature depends on feature E to be verified, and feature E to be verified depends on feature F to be verified, then the dependency information can be supplemented with the content that the target input feature depends on feature E to be verified, and feature E to be verified depends on feature F to be verified.

[0051] In another possible implementation, determining the dependency information between the target input feature and the predetermined feature to be verified can include whether there is a (direct or indirect) dependency between the target input feature and the feature to be verified. For example, if the target input feature depends on feature E to be verified, and feature E to be verified depends on feature F to be verified, then the dependency information can record that there is a dependency between the target input feature and feature E to be verified, and also that there is a dependency between the target input feature and feature F to be verified. If there is no direct or indirect dependency between the target input feature and feature G to be verified, then the dependency information can record that there is no dependency between the target input feature and feature G to be verified.

[0052] In one implementation, generating model information analysis results based on dependency information may include taking each input feature of the target model as a target input feature, performing dependency analysis to obtain all dependency information when each input feature is taken as a target input feature, and generating model information analysis results based on all dependency information.

[0053] In another possible implementation, generating model information analysis results based on dependency information may include generating corresponding model information analysis results for each of the multiple models called at the current moment, and summarizing the multiple model information analysis results into a single model information analysis result.

[0054] In this embodiment, the input features and features to be verified of the target model can be analyzed to determine the dependency information between the input features and features to be verified of the target model. This helps to identify the features to be verified that are dependent on the input features of the current target model, and then find the features that are not related to the model input features from the features to be verified. This helps to simplify the features to be verified, reduce the useless features in the features to be verified, and help to improve the management efficiency of the model and model-related data, as well as the efficiency of model use.

[0055] In one implementation, determining dependency information between target input features and predetermined features to be verified includes at least one of the following steps performed for each feature to be verified:

[0056] When the acquisition process of the target input feature is related to the first feature to be verified, the dependency information is determined to include that the target input feature depends on the first feature to be verified; and the first feature to be verified is one of the features to be verified.

[0057] When the acquisition process of the first feature to be verified is related to the target input feature, the dependency information is determined to include the dependency of the first feature to be verified on the target input feature.

[0058] In this embodiment, dependency information may include the mutual dependencies between the target input features and any features to be verified.

[0059] The process of obtaining the target input features is related to the features to be verified, and may include the calculation of the target input features directly or indirectly involving the features to be verified. For example, if the calculation formula for the target input features is: Target Input Feature = f(G), then the process of obtaining the target input features is related to feature G. As another example, if the calculation formula for the target input features is: Target Input Feature = F(G), G = f(H), then the process of obtaining the target input features is related to features G and feature H.

[0060] The process of obtaining the target input features is related to the features to be verified, and may also include cases where the target input features include the features to be verified. For example, if the target input feature is a geographical location, specifically including province, city, and county, then the target input feature is related to the features "province", "city", and "county".

[0061] In this embodiment, the dependency information between the target input features and the features to be verified can be determined based on the relevance of the acquisition process, thereby helping to remove features that are not used by any model from the features to be verified based on the dependency information.

[0062] In one implementation, determining the dependency information between the target input feature and the feature to be verified includes:

[0063] When the process of obtaining the dependent feature in the dependency relationship information is related to the second feature to be verified, the dependency relationship information is determined to include the dependent feature in the dependency relationship information depending on the second feature to be verified, and the second feature to be verified is another feature to be verified besides the first feature to be verified.

[0064] In this embodiment, the feature that is depended on in the dependency relationship information can be the feature to be verified when the target input feature depends on the feature to be verified, or the target input feature when the feature to be verified depends on the target input feature.

[0065] In this embodiment, the second feature to be verified can be any feature other than the dependent feature in the current dependency relationship information. For example, if feature I depends on feature H according to the current dependency relationship information, then the dependent feature in the current dependency relationship information is H, and the second feature to be verified is any feature other than feature I and H. If the input features of the target model are included in the features to be verified, then other input features in the target model besides the target input / output features may also be the second feature to be verified.

[0066] For example, based on the current dependency information, feature I depends on feature H, and feature J depends on feature K. Therefore, the dependent features in the current dependency information are H and K, and the second feature to be verified is any feature other than H and K. Alternatively, in this case, the second feature to be verified is any feature other than those already present in the dependency information, i.e., any feature other than I, H, J, and K.

[0067] In another possible implementation, the second feature to be verified can also be another feature in the current dependency information that does not exist in the related dependencies of the dependent feature. For example, according to the current dependency information, feature I depends on feature H, and feature J depends on feature K. Then, for feature H, the related dependency information is "feature I depends on feature H". Features J and K can be regarded as other features that do not exist in the related dependencies of the dependent feature, i.e., the second feature to be verified.

[0068] This embodiment enables the inclusion of the dependency relationship between indirectly dependent target input features and features to be verified in the dependency relationship information, making the dependency relationship information more comprehensive and helping to avoid deleting still-used features when simplifying features to be verified.

[0069] In one implementation, model information analysis results are generated based on dependency information, including:

[0070] Based on dependency information, identify the features to be verified that are dependent on the target input features; the features to be verified that are dependent on the target input features include the features to be verified that are related to all dependency information.

[0071] Based on the features to be verified that are dependent on the target input features, the model information analysis results are generated.

[0072] In this embodiment, the features to be verified that are related to all dependency information can be features existing within the dependency information. For example, the features to be verified include features {H1, H2, H3, H4, H5, K1, K2, K3, K4, K5}. The input features of the target model include features {I1, I2, J1, J2}. The current dependency information includes that feature I1 depends on I2, feature I2 depends on H1, feature H1 depends on H2, feature J1 depends on H2, and feature J2 depends on K1. Therefore, the features related to all dependency information include all features existing within the dependency information, namely features I1, I2, H1, H2, J1, and K1.

[0073] In this embodiment, based on dependency information, it is possible to determine the features to be verified that have a dependency relationship with the input features of the target model, and the features to be verified that do not have a dependency relationship with the input features of the target model.

[0074] For example, the features to be verified include features {H1, H2, H3, H4, H5, K1, K2, K3, K4, K5}, and the features to be verified related to all dependency information include features H1, H2, and K1.

[0075] In this embodiment, model information analysis results about the features to be verified are generated based on dependency information, which helps to manage the model and model-related data based on the model information analysis results, and helps to improve the efficiency of model use.

[0076] In one implementation, such as Figure 2 As shown, model information analysis methods also include:

[0077] Step S21: Based on the model information analysis results, determine the redundant features among the features to be verified; redundant features are features to be verified that are not dependent on the input features of any model among multiple models.

[0078] Step S22: Delete redundant features from the features to be verified to obtain the verified features.

[0079] In this embodiment, based on the model information analysis results, redundant features in the features to be verified are determined. This can be done by identifying features in the features to be verified that are not related to the dependency information as redundant features.

[0080] For example, the features to be verified include features {H1, H2, H3, H4, H5, K1, K2, K3, K4, K5}. The features to be verified that are related to all dependency information include features H1, H2, and K1. The features to be verified that are not related to all dependency information are the other features in the features to be verified besides H1, H2, and K1, namely features H3, H4, H5, K2, K3, K4, and K5.

[0081] In this embodiment, features to be verified that are not dependent on the input features of any model can be obtained, and then undependent features to be verified can be deleted, thereby simplifying the features to be verified. When applied to scenarios such as model management platforms, this can improve the convenience of model and model-related data management and help improve model usage efficiency.

[0082] In one implementation, the model information analysis method further includes:

[0083] Based on the verified features, generate predicted samples;

[0084] The target model is trained based on the estimated samples.

[0085] In this embodiment, simulated prediction samples can be generated based on all input features of the target model and other features that each input feature depends on. For example, the input features of the target model include "province" in geographical location, usage time period code, and user network age, where the "province" feature depends on the "geographical location" feature. Therefore, the information on the "province" in geographical location, the usage time period code, and the user network age are predicted, and at least one set of input data is generated based on the predicted information. The model is then trained using the predicted input data.

[0086] In this embodiment, the verified features are used to generate predicted samples, and the predicted samples are used to train the target model, thereby improving the model training efficiency and the convenience of training operations.

[0087] In one example of this disclosure, such as Figure 4 As shown, based on the validated features, feature configuration is performed to determine the features used for sample prediction. For each model (which can be the target model), sample prediction is performed based on the features obtained from the feature configuration operation. Figure 4 The model prediction process involves accumulating predicted samples and training the model based on the accumulated results. Since features are the input data for the model in machine learning, a ranking service manages all models used and deployed from the initial user adoption to the current time during a single prediction process. The ranking architecture of the management module obtains samples related to the input data (features) from different upstream and downstream sources during a single prediction process through a feature service. It then refines the raw data into features actually used in model prediction through feature configuration. After feature engineering, including feature combination and specific encoding, the samples are used to predict the model's prediction results. Simultaneously, the samples are stored in files for use in training any model, including the target model. Therefore, the feature service is the source of all samples, and the feature configuration file is the list of features actually used by the model. The model's features can be traced and managed by analyzing the feature configuration and feature service.

[0088] In one implementation, the model information analysis method further includes:

[0089] Obtain the configuration file of the target model based on the service address of the target model in the deployment environment;

[0090] Based on the target model's configuration file, determine all input features of the target model.

[0091] In this embodiment, the configuration file can be a file generated during the execution of the target model.

[0092] This embodiment uses a configuration file to obtain all the input features currently used by the target model, ensuring the validity and completeness of the input features.

[0093] In one implementation, such as Figure 3 As shown, model information analysis methods also include:

[0094] Step S31: Determine multiple models based on the list of models to be called;

[0095] Step S32: Determine the redundant model based on multiple models and the full model to be verified;

[0096] Step S33: In the full set of input features related to the full model, delete the redundant model-related features to obtain the features to be verified.

[0097] In this embodiment, the full set of models to be verified can be obtained based on the historical records of the model management platform. The full set of models can include all historical versions of all models and all models that have been iterated and used.

[0098] In this embodiment, the model and its input features are first screened based on the model's validity, thereby reducing the number of features to be verified and improving the efficiency of model information analysis.

[0099] In one specific implementation, each called model in the full model can be used as the target model, and the model information analysis method provided in this embodiment can be executed to summarize the analysis results corresponding to each model to obtain the full model analysis results.

[0100] In one example disclosed herein, model information processing involves related information and operations such as... Figure 5 As shown.

[0101] In this example, the data source may include the Apollo platform (51), the online regression database (52), and the online model environment (53). The Apollo platform can be an open-source configuration management center capable of centrally managing the configurations of different application environments and clusters. It can push changes to configuration information such as the model name to the application in real time and has standardized access control and process governance features. This example primarily obtains partial model configuration information from the Apollo platform, such as the model name and sample storage path.

[0102] The data in the regression database can be the latest model information obtained by the ranking-service module during the automated regression deployment phase. The latest model information can include the model BNS (Backbone Network Service), the APP (Application) using the model, and the product lines involved in the model.

[0103] The online environment of a model can refer to the service address where a single model is deployed online. In this example, the feature configuration file of the model can be obtained through the service address where the model is deployed online. This can be extracted using an extractor.

[0104] In this example, the online model list can be obtained through the Apollo platform (51) and the online regression database (52), which includes multiple models currently being invoked. The full model list (DB) can be obtained from the Paoding database (53). Based on the online and full model lists, the offline model list can be determined; models that are not present in the online model list but exist in the full model list can be added to the offline model list.

[0105] This example uses the feature-service (fs) to obtain all features and their sources from an online machine learning system. For all models in the list of offline models, the models and related data can be logically deleted. Related data can include the model itself, slots (encoded field information) within the model, and the model's features. The essence of this logical deletion is a modification operation; that is, the logical deletion in this example is not a true deletion, but rather an modification of the corresponding deletion flag (is_delete) or status field (status) in the table. For example, 0 (or other numbers) can indicate that the data has not been deleted, and 1 (or other numbers) can indicate that it has been deleted. Logically, the data is deleted, but the data itself still exists in the database.

[0106] Based on the full set of features obtained from the fs service, and after removing the features of the models in the list of offline models, the features to be verified can be obtained.

[0107] Based on the online model list, a traversal operation is performed. For each model in the online model list, the model deployment environment is obtained from the model's online environment. Based on the model deployment environment, the configuration file is downloaded. The configuration file is parsed using an Extractor class to obtain relevant information about the model and features corresponding to the configuration file, thus obtaining the input features for multiple models. This example, through configuration file parsing, can complete the initial mapping of the model-feature-sample relationship. Furthermore, basic model information, such as model name, cube (model deployment environment address) name, QPS (Query Per Second) and instance information for each data center, filesystem information, coarse-ranked incremental topic, and update information, are written to a MySQL database (54) to complete comprehensive model statistics. Basic model information can be obtained not only through configuration file parsing but also through files such as Excel spreadsheets.

[0108] Based on the results of the initial mapping, further sample depth analysis can be performed.

[0109] During in-depth sample analysis, the model's samples can be further analyzed to establish dependencies from operators to sample leaf nodes within the analyzed model, constructing a dependency tree where each node corresponds to a sample field (i.e., the feature mentioned in the previous embodiment). A final mapping between the model and sample fields is established. Building upon the model-slot-sample mapping, a step of expanding operators to leaf nodes is added to ensure that the mapping (i.e., the dependency information mentioned in the previous embodiment) comprehensively includes the relevant features or sample fields. The aforementioned slot can be the encoding corresponding to a field in the target model.

[0110] In this example, multiple models, their input features, and features to be validated are determined based on the data source. Dependency analysis is then performed between the input features of the multiple models and the features to be validated to obtain dependency information. This dependency information, along with all features and models, can be persistently stored in a MySQL database. Figure 5 The example shown can be executed by setting a timed task for model data, transmitting model information to the data management and display interface 55 through the interface, and displaying the model and related data mapping, storage and other operations.

[0111] In another specific example, during the sample depth analysis phase, a feature tree can be constructed based on the hierarchical and combination relationships of features. For example... Figure 6As shown, different slots in the model can correspond to different features, and features can form samples. Features can be further divided into single features and combined features. Combined features can be composed of single features, and there is no upper limit to the level of combination. By expanding combined features to single features, the first-level fields corresponding to the combined features can be expanded to second-level fields, third-level fields, and so on, ultimately expanding to the leaf nodes corresponding to the single features (corresponding to leaf fields). If the sample field that a slot depends on in the configuration file is not a leaf node, it can be expanded to a leaf node to ensure the accuracy of the calculated model-sample mapping. In the mapping or dependency relationship, the relationship between samples and models can be many-to-many, and the relationship between models and slots can be one-to-many.

[0112] For example, a combined feature could include: geographic location - province. If this combined feature corresponds to N levels of fields, the individual features geographic location and province can correspond to N+1 levels of fields. Furthermore, the node corresponding to the province can be a leaf node, while geographic location, which includes province information, can also include information about special administrative regions. Therefore, geographic location is not a leaf node. When expanding other features to the geographic location feature, the geographic location can be further expanded to the leaf nodes corresponding to the province or special administrative region.

[0113] In one specific example, the model information analysis method provided in this disclosure can be applied to model management in a feed stream recommendation system. Since its launch, the feed stream recommendation system has integrated multiple business lines, resulting in a continuous accumulation of the number and types of models. The system faces the following problems: the feed stream system has a large number of features from diverse data sources, and the names of models or model fields vary after being forwarded through multiple systems. The dependency links between model features are complex, requiring layer-by-layer analysis to pinpoint problems, significantly hindering problem localization. Furthermore, the number of fields in the feed stream system only increases, with an unused field rate exceeding 40%. These fields significantly increase the usage cost and resource consumption of the business, and also bring considerable repetitive work to QA (Quality Assurance) regression testing. Model fields are managed only by a specific platform (such as a Protocol Buffer platform), and multiple business lines use the same management platform, lacking a platform to intuitively display the usage status of all models and their features within the recommendation system.

[0114] This public example proposes a model-feature-sample mapping scheme based on configuration extraction. On the basis of model management, it builds a model data access and management platform with high timeliness based on model dimensions. Through offline testing, online regression and other methods, it collects basic data such as model features and samples, and after preliminary analysis and calculation, it establishes a multi-dimensional data analysis capability, including a data collection system for basic model information, resource distribution, feature usage details and other data.

[0115] The model information analysis method provided in this disclosure provides more comprehensive statistical information. In this example, the ranking-service module (the entry point of the ranking module) and the Apollo platform are used as data sources, resulting in comprehensive model information, including the features currently being used by the model. Simultaneously, the feature-service module obtains the complete set of machine learning features, facilitating the parsing and writing of the model's state to the database and visualization through the platform. This data allows for easy removal of redundant features within the system, facilitating feature pruning. When applied to feed stream recommendation systems, which contain tens of thousands of model fields from diverse sources and with varying processing and usage methods, this disclosure clarifies the dependencies between features, facilitating feature source tracing, reducing feature acquisition time during model usage, and thus improving model management and usage efficiency.

[0116] This disclosure also provides a model information analysis device, such as... Figure 7 As shown, it includes:

[0117] The target model acquisition module 71 is used to acquire the target model; the target model is one of multiple models that are called at the current moment.

[0118] The dependency information determination module 72 is used to determine the dependency information between the target input feature and the predetermined feature to be verified; the target input feature is one of multiple input features of the target model;

[0119] The analysis results module 73 is used to generate model information analysis results based on dependency information.

[0120] In one implementation, such as Figure 8 As shown, the dependency information determination module includes:

[0121] The first determining unit 81 is used to determine, when the acquisition process of the target input feature is related to the first feature to be verified, the dependency information includes that the target input feature depends on the first feature to be verified; the first feature to be verified is one of the features to be verified.

[0122] The second determining unit 82 is used to determine, when the acquisition process of the first feature to be verified is related to the target input feature, the dependency information includes the first feature to be verified depending on the target input feature.

[0123] In one implementation, such as Figure 9 As shown, the dependency information determination module includes:

[0124] The third determining unit 91 is used to determine, when the process of obtaining the dependent feature in the dependency relationship information is related to the second feature to be verified, that the dependency relationship information includes the dependent feature in the dependency relationship information depending on the second feature to be verified, and the second feature to be verified is another feature to be verified other than the first feature to be verified.

[0125] In one implementation, such as Figure 10 As shown, the analysis results module includes:

[0126] The first result unit 101 is used to determine the features to be verified that are dependent on the target input features based on the dependency information; the features to be verified that are dependent on the target input features include the features to be verified that are related to all dependency information.

[0127] The second result unit 102 is used to generate model information analysis results based on the features to be verified that are dependent on the target input features.

[0128] In one implementation, such as Figure 11 As shown, the model information analysis device also includes:

[0129] The redundant feature module 111 is used to determine the redundant features in the features to be verified based on the analysis results of the model information; the redundant features are the features to be verified that are not dependent on the input features of any model in the multiple models.

[0130] The verification module 112 is used to delete redundant features from the features to be verified, so as to obtain the verified features.

[0131] In one implementation, such as Figure 12 As shown, the model information analysis device also includes:

[0132] The prediction sample module 121 is used to generate prediction samples based on the verified features;

[0133] Training module 122 is used to train the target model based on the estimated samples.

[0134] In one implementation, such as Figure 13 As shown, the model information analysis device also includes:

[0135] The configuration file acquisition module 131 is used to obtain the configuration file of the target model based on the service address of the target model in the deployment environment.

[0136] The input feature module 132 is used to determine all input features of the target model based on the configuration file of the target model.

[0137] In one implementation, such as Figure 14 As shown, the model information analysis device also includes:

[0138] The model acquisition module 141 is used to determine multiple models based on the list of called models;

[0139] Redundancy model module 142 is used to determine the redundant model based on multiple models and the full model to be verified;

[0140] The feature acquisition module 143 is used to remove redundant model-related features from the full set of input features related to the full model to obtain the features to be verified.

[0141] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0142] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0143] Figure 15 A schematic block diagram of an example electronic device 150 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0144] like Figure 15 As shown, device 150 includes a computing unit 151, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 152 or a computer program loaded from storage unit 158 ​​into random access memory (RAM) 153. RAM 153 may also store various programs and data required for the operation of device 150. The computing unit 151, ROM 152, and RAM 153 are interconnected via bus 154. Input / output (I / O) interface 155 is also connected to bus 154.

[0145] Multiple components in device 150 are connected to I / O interface 155, including: input unit 156, such as a keyboard, mouse, etc.; output unit 157, such as various types of displays, speakers, etc.; storage unit 158, such as a disk, optical disk, etc.; and communication unit 159, such as a network card, modem, wireless transceiver, etc. Communication unit 159 allows device 150 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0146] The computing unit 151 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 151 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 151 performs the various methods and processes described above, such as model information analysis methods. For example, in some embodiments, the model information analysis method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 158. In some embodiments, part or all of the computer program may be loaded and / or installed on device 150 via ROM 152 and / or communication unit 159. When the computer program is loaded into RAM 153 and executed by the computing unit 151, one or more steps of the model information analysis method described above may be performed. Alternatively, in other embodiments, the computing unit 151 may be configured to perform model information analysis methods by any other suitable means (e.g., by means of firmware).

[0147] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0148] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0149] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0150] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0151] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0152] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0153] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0154] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A model information analysis method, comprising: Obtain the target model; The target model is one of several models that are invoked at the current moment; The multiple models invoked at the current moment include an article recommendation model for recommending articles to users, a question-and-answer information processing model for processing user input questions, a video recommendation model for recommending videos to users, and a trending news filtering model for determining the trending search rankings. Determine the dependency information between the target input feature and the predetermined feature to be verified; the target input feature is one of multiple input features of the target model; the dependency information includes at least one of the following: whether there is a direct or indirect dependency between the target input feature and the feature to be verified; determine whether the target input feature directly or indirectly depends on the feature to be verified, and if so, add the content of the target input feature depending on the feature to be verified to the dependency information; Determine whether the target input feature is directly or indirectly dependent on the feature to be verified. If so, add the content about the target input feature being dependent on the feature to be verified to the dependency information. Based on the dependency information, features that are dependent on the target input feature are determined from the predetermined features to be verified, wherein the dependency is used to characterize the relevance of the feature acquisition process, and the features that are dependent on the target input feature are features used to input the model; and Based on the features to be verified that are dependent on the target input features, model information analysis results are generated.

2. The method according to claim 1, wherein, Determining the dependency information between the target input feature and the predetermined feature to be verified includes at least one of the following steps performed for each of the features to be verified: When the acquisition process of the target input feature is related to the first feature to be verified, the dependency information is determined to include the target input feature depending on the first feature to be verified. The first feature to be verified is one of the predetermined features to be verified; When the first feature to be verified is related to the target input feature, the dependency information is determined to include the first feature to be verified depending on the target input feature.

3. The method according to claim 2, wherein, The determination of the dependency information between the target input features and the features to be verified includes: When the process of obtaining the dependent feature in the dependency relationship information is related to the second feature to be verified, it is determined that the dependency relationship information includes the dependent feature in the dependency relationship information depending on the second feature to be verified, and the second feature to be verified is another feature to be verified besides the first feature to be verified.

4. The method according to claim 1, further comprising: Based on the analysis results of the model information, redundant features in the features to be verified are identified; Redundant features are features to be verified that are not dependent on the input features of any of the multiple models. Redundant features are removed from the features to be verified to obtain the verified features.

5. The method according to claim 4, further comprising: Based on the verified features, generate predicted samples; The target model is trained based on the predicted samples.

6. The method according to any one of claims 1-5, further comprising: Based on the service address of the target model in the deployment environment, obtain the configuration file of the target model; Based on the configuration file of the target model, determine all input features of the target model.

7. The method according to any one of claims 1-5, further comprising: Based on the list of models to be invoked, determine the plurality of models; Based on the multiple models and the full model to be verified, a redundant model is determined; In the full set of input features related to the full model, the redundant model features are deleted to obtain the features to be verified.

8. The method according to claim 6, further comprising: Based on the list of models to be invoked, determine the plurality of models; Based on the multiple models and the full model to be verified, a redundant model is determined; In the full set of input features related to the full model, the redundant model features are deleted to obtain the features to be verified.

9. A model information analysis device, comprising: The target model acquisition module is used to acquire the target model; The target model is one of several models that are invoked at the current moment; The multiple models invoked at the current moment include an article recommendation model for recommending articles to users, a question-and-answer information processing model for processing user input questions, a video recommendation model for recommending videos to users, and a trending news filtering model for determining the trending search rankings. A dependency information determination module is used to determine the dependency information between a target input feature and a predetermined feature to be verified; the target input feature is one of multiple input features of the target model; the dependency information includes at least one of the following: whether there is a direct or indirect dependency between the target input feature and the feature to be verified; determining whether the target input feature directly or indirectly depends on the feature to be verified, and if so, adding the content of the target input feature depending on the feature to be verified to the dependency information; Determine whether the target input feature is directly or indirectly dependent on the feature to be verified. If so, add the content about the target input feature being dependent on the feature to be verified to the dependency information. The analysis results module is used to determine, based on the dependency information, the features to be verified that have a dependency relationship with the target input feature from the predetermined features to be verified, wherein the dependency relationship is used to characterize the relevance of the feature acquisition process, and the features to be verified that have a dependency relationship with the target input feature are features used to input the model; and to generate model information analysis results based on the features to be verified that have a dependency relationship with the target input feature.

10. The apparatus according to claim 9, wherein, The dependency information determination module includes: The first determining unit is configured to determine, when the acquisition process of the target input feature is related to the first feature to be verified, that the dependency information includes the target input feature depending on the first feature to be verified; the first feature to be verified is one of the predetermined features to be verified. The second determining unit is configured to determine, when the first feature to be verified acquisition process is related to the target input feature, that the dependency information includes the first feature to be verified depending on the target input feature.

11. The apparatus according to claim 10, wherein, The dependency information determination module includes: The third determining unit is used to determine, when the acquisition process of the dependent feature in the dependency information is related to other features, that the dependency information includes the dependent feature in the dependency information depending on the other features.

12. The apparatus according to claim 9, further comprising: The redundancy feature module is used to determine the redundant features in the features to be verified based on the analysis results of the model information. Redundant features are features to be verified that are not dependent on the input features of any of the multiple models. The verification module is used to delete redundant features from the features to be verified, so as to obtain the verified features.

13. The apparatus of claim 12, further comprising: The prediction sample module is used to generate prediction samples based on the validated features. The training module is used to train the target model based on the estimated samples.

14. The apparatus according to any one of claims 9-13, further comprising: The configuration file acquisition module is used to obtain the configuration file of the target model based on the service address of the target model in the deployment environment; The input feature module is used to determine all input features of the target model based on the configuration file of the target model.

15. The apparatus according to any one of claims 9-13, further comprising: The model acquisition module is used to determine the multiple models based on the list of models to be called; The redundant model module is used to determine the redundant model based on the multiple models and the full model to be verified. The feature acquisition module is used to delete redundant model-related features from the full input features related to the full model to obtain the feature to be verified.

16. The apparatus of claim 14, further comprising: The model acquisition module is used to determine the multiple models based on the list of models to be called; The redundant model module is used to determine the redundant model based on the multiple models and the full model to be verified. The feature acquisition module is used to delete redundant model-related features from the full input features related to the full model to obtain the feature to be verified.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images