Model evaluation method, device, electronic device and storage medium

Abstract

The present application discloses a model evaluation method, device, electronic device and storage medium. The evaluation method includes: obtaining an evaluation sample set and a model to be evaluated, wherein the model to be evaluated includes at least one sub-model; identifying the hierarchical relationship between the sub-models, wherein the hierarchical relationship includes at least one of a serial relationship and a parallel relationship; distributing the evaluation samples to the sub-models according to the hierarchical relationship; and obtaining the evaluation results of the sub-models according to the evaluation samples distributed to the sub-models. Thus, based on the hierarchical relationship between the sub-models, the evaluation samples can be automatically distributed between the sub-models, and then the evaluation results of the sub-models can be obtained, which improves the flexibility and accuracy of the model evaluation.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model evaluation method, device, electronic device and storage medium. Background Art

[0002] At present, with the development of artificial intelligence technology, models are widely used in technical fields such as intelligent recommendation, speech recognition, and image processing, with advantages such as high degree of automation and low labor costs. For example, recommendation models can be used to recommend products, articles, anchors, etc. to users, speech recognition models can be used to convert speech into text information, and image processing models can be used to recognize image categories and object actions in images. However, the model evaluation methods in related technologies are relatively simple, with low flexibility and accuracy. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the above-mentioned technology at least to some extent.

[0004] To this end, one purpose of the present application is to propose a model evaluation method, which can distribute evaluation samples to sub-models according to the hierarchical relationship between sub-models, and obtain the evaluation results of the sub-models according to the evaluation samples distributed to the sub-models. Thus, based on the hierarchical relationship between sub-models, the evaluation samples can be automatically distributed between sub-models, and then the evaluation results of the sub-models can be obtained, which improves the flexibility and accuracy of model evaluation.

[0005] The second objective of this application is to provide a model evaluation device.

[0006] The third objective of the present application is to provide an electronic device.

[0007] A fourth objective of the present application is to provide a computer-readable storage medium.

[0008] To achieve the above-mentioned objectives, the first aspect of the present application proposes a model evaluation method, comprising: obtaining an evaluation sample set and a model to be evaluated, wherein the model to be evaluated includes at least one sub-model; identifying a hierarchical relationship between the sub-models, wherein the hierarchical relationship includes at least one of a serial relationship and a parallel relationship; according to the hierarchical relationship, distributing the evaluation samples to the sub-models; and according to the evaluation samples distributed to the sub-models, obtaining the evaluation results of the sub-models.

[0009] According to the model evaluation method of the embodiment of the present application, the evaluation samples can be distributed to the sub-models according to the hierarchical relationship between the sub-models, and the evaluation results of the sub-models can be obtained according to the evaluation samples distributed to the sub-models. Thus, based on the hierarchical relationship between the sub-models, the evaluation samples can be automatically distributed between the sub-models, and then the evaluation results of the sub-models can be obtained, which improves the flexibility and accuracy of the model evaluation.

[0010] In addition, the model evaluation method proposed in the above embodiment of the present application may also have the following additional technical features:

[0011] In one embodiment of the present application, distributing the evaluation samples to the sub-models according to the hierarchical relationship includes: obtaining the sample distribution conditions and / or the sample distribution ratio of the sub-models; and distributing the evaluation samples to the sub-models according to the hierarchical relationship, the sample distribution conditions and / or the sample distribution ratio.

[0012] In one embodiment of the present application, distributing the evaluation sample to the sub-model according to the hierarchical relationship, the sample distribution condition and / or the sample distribution ratio includes: acquiring attribute information of the evaluation sample; identifying, based on the attribute information, that the sub-model meets the sample distribution condition; and distributing the evaluation sample to the sub-model according to the hierarchical relationship.

[0013] In one embodiment of the present application, the hierarchical relationship includes a serial relationship and a parallel relationship, and distributing the evaluation samples to the sub-models according to the hierarchical relationship includes: distributing the evaluation samples to multiple first sub-models, wherein the hierarchical relationship between the multiple first sub-models is a parallel relationship; for any first sub-model, distributing the evaluation samples distributed to any first sub-model to multiple second sub-models, wherein the hierarchical relationship between the multiple second sub-models is a parallel relationship; wherein the hierarchical relationship between the first sub-model and the second sub-model is a serial relationship, and the distribution time of the first sub-model is earlier than the distribution time of the second sub-model.

[0014] In one embodiment of the present application, the hierarchical relationship only includes a serial relationship, and distributing the evaluation samples to the sub-models according to the hierarchical relationship includes: distributing the evaluation samples to a third sub-model; distributing the evaluation samples distributed to the third sub-model to a fourth sub-model; wherein the hierarchical relationship between the third sub-model and the fourth sub-model is a serial relationship, and the distribution time of the third sub-model is earlier than the distribution time of the fourth sub-model.

[0015] In one embodiment of the present application, the hierarchical relationship only includes parallel relationships, and distributing the evaluation samples to the sub-models according to the hierarchical relationship includes: distributing the evaluation samples to multiple fifth sub-models, wherein the hierarchical relationship between the multiple fifth sub-models is a parallel relationship.

[0016] In one embodiment of the present application, distributing the evaluation sample to the sub-model according to the hierarchical relationship includes: obtaining identification information of the evaluation sample; determining the target sub-model corresponding to the evaluation sample according to the identification information and the hierarchical relationship; and distributing the evaluation sample to the target sub-model.

[0017] In one embodiment of the present application, obtaining the evaluation result of the sub-model according to the evaluation sample distributed to the sub-model includes: obtaining evaluation configuration information of the sub-model; configuring an evaluation environment according to the evaluation configuration information and the evaluation sample distributed to the sub-model; running the evaluation environment, and obtaining the evaluation result of the sub-model based on the evaluation environment.

[0018] To achieve the above-mentioned objectives, the second aspect of the present application proposes a model evaluation device, comprising: an acquisition module, used to acquire an evaluation sample set and a model to be evaluated, wherein the model to be evaluated includes at least one sub-model; an identification module, used to identify the hierarchical relationship between the sub-models, wherein the hierarchical relationship includes at least one of a serial relationship and a parallel relationship; a distribution module, used to distribute the evaluation samples to the sub-models according to the hierarchical relationship; and an evaluation module, used to obtain the evaluation results of the sub-models according to the evaluation samples distributed to the sub-models.

[0019] The model evaluation device of the embodiment of the present application can distribute the evaluation samples to the sub-models according to the hierarchical relationship between the sub-models, and obtain the evaluation results of the sub-models according to the evaluation samples distributed to the sub-models. Thus, based on the hierarchical relationship between the sub-models, the evaluation samples can be automatically distributed between the sub-models, and then the evaluation results of the sub-models can be obtained, which improves the flexibility and accuracy of the model evaluation.

[0020] In addition, the model evaluation device proposed in the above embodiment of the present application may also have the following additional technical features:

[0021] In one embodiment of the present application, the distribution module is also used to: obtain the sample distribution conditions and / or sample distribution ratio of the sub-model; and distribute the evaluation samples to the sub-model according to the hierarchical relationship, the sample distribution conditions and / or the sample distribution ratio.

[0022] In one embodiment of the present application, the distribution module is further used to: obtain attribute information of the evaluation sample; identify the sub-model that meets the sample distribution condition based on the attribute information; and distribute the evaluation sample to the sub-model based on the hierarchical relationship.

[0023] In one embodiment of the present application, the hierarchical relationship includes a serial relationship and a parallel relationship, and the distribution module is further used to: distribute the evaluation samples to multiple first sub-models, wherein the hierarchical relationship between the multiple first sub-models is a parallel relationship; for any first sub-model, distribute the evaluation samples distributed to any first sub-model to multiple second sub-models, wherein the hierarchical relationship between the multiple second sub-models is a parallel relationship; wherein the hierarchical relationship between the first sub-model and the second sub-model is a serial relationship, and the distribution time of the first sub-model is earlier than the distribution time of the second sub-model.

[0024] In one embodiment of the present application, the hierarchical relationship only includes a serial relationship, and the distribution module is further used to: distribute the evaluation sample to the third sub-model; distribute the evaluation sample distributed to the third sub-model to the fourth sub-model; wherein the hierarchical relationship between the third sub-model and the fourth sub-model is a serial relationship, and the distribution time of the third sub-model is earlier than the distribution time of the fourth sub-model.

[0025] In one embodiment of the present application, the hierarchical relationship only includes a parallel relationship, and the distribution module is further used to: distribute the evaluation sample to a plurality of fifth sub-models, wherein the hierarchical relationship between the plurality of fifth sub-models is a parallel relationship.

[0026] In one embodiment of the present application, the distribution module is further used to: obtain identification information of the evaluation sample; determine the target sub-model corresponding to the evaluation sample based on the identification information and the hierarchical relationship; and distribute the evaluation sample to the target sub-model.

[0027] In one embodiment of the present application, the evaluation module is also used to: obtain evaluation configuration information of the sub-model; configure an evaluation environment according to the evaluation configuration information and the evaluation sample distributed to the sub-model; run the evaluation environment, and obtain the evaluation result of the sub-model based on the evaluation environment.

[0028] To achieve the above-mentioned objectives, the third aspect embodiment of the present application proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the model evaluation method as described in the first aspect embodiment of the present application is implemented.

[0029] The electronic device of the embodiment of the present application can distribute the evaluation samples to the sub-models according to the hierarchical relationship between the sub-models through the processor executing the instructions stored in the memory, and obtain the evaluation results of the sub-models according to the evaluation samples distributed to the sub-models. Thus, based on the hierarchical relationship between the sub-models, the evaluation samples can be automatically distributed between the sub-models, and then the evaluation results of the sub-models can be obtained, which improves the flexibility and accuracy of the model evaluation.

[0030] To achieve the above-mentioned objectives, the fourth aspect of the present application proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the model evaluation method as described in the first aspect of the present application is implemented.

[0031] The computer-readable storage medium of the embodiment of the present application can distribute the evaluation samples to the sub-models according to the hierarchical relationship between the sub-models by storing the computer program and being executed by the processor, and obtain the evaluation results of the sub-models according to the evaluation samples distributed to the sub-models. Thus, based on the hierarchical relationship between the sub-models, the evaluation samples can be automatically distributed between the sub-models, and then the evaluation results of the sub-models can be obtained, thereby improving the flexibility and accuracy of the model evaluation.

[0032] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0034] Figure 1 A schematic diagram of a flow chart of a model evaluation method according to an embodiment of the present application;

[0035] Figure 2 is a schematic diagram of a model to be evaluated in a model evaluation method according to an embodiment of the present application;

[0036] Figure 3 is a schematic diagram of a model to be evaluated in a model evaluation method according to another embodiment of the present application;

[0037] Figure 4 is a schematic diagram of a model to be evaluated in a model evaluation method according to another embodiment of the present application;

[0038] Figure 5 A schematic diagram of a process of distributing evaluation samples to sub-models in a model evaluation method according to an embodiment of the present application;

[0039] Figure 6A schematic diagram of a flow chart of distributing evaluation samples to sub-models in a model evaluation method according to another embodiment of the present application;

[0040] Figure 7 is a block diagram of a model evaluation device according to an embodiment of the present application;

[0041] Figure 8 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0043] The model evaluation method, device, electronic device and storage medium of the embodiments of the present application are described below in conjunction with the accompanying drawings.

[0044] Figure 1 The figure is a flowchart of a model evaluation method according to an embodiment of the present application.

[0045] like Figure 1 As shown, the model evaluation method of the embodiment of the present application includes:

[0046] S101, obtaining an evaluation sample set and a model to be evaluated, wherein the model to be evaluated includes at least one sub-model.

[0047] It should be noted that the execution subject of the model evaluation method of the embodiment of the present application may be the model evaluation device of the embodiment of the present application, and the above-mentioned model evaluation device may be configured in any electronic device to execute the model evaluation method of the embodiment of the present application. Among them, the electronic device may be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, etc., and the mobile terminal may be, for example, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens, and / or display screens.

[0048] In the embodiments of the present application, an evaluation sample set may be obtained. It is understandable that the evaluation sample set may include a large number of evaluation samples. It should be noted that the types of evaluation samples are not overly limited. For example, when the model to be evaluated is a recommendation model, the evaluation sample may be user information of a sample user, when the model to be evaluated is a speech recognition model, the evaluation sample may be speech, and when the model to be evaluated is an image processing model, the evaluation sample may be an image.

[0049] In one implementation, a user can log in to an application (APP), website, etc., and perform operations such as clicking and searching on the APP, website, etc., and correspondingly, the relevant server can collect user information in response to the user's click, search, etc. operations to generate an evaluation sample set. It should be noted that the evaluation sample at this time is the user information of the sample user.

[0050] In an embodiment of the present application, a model to be evaluated may be obtained, wherein the model to be evaluated includes at least one sub-model. It should be noted that the type of the model to be evaluated is not too limited, for example, the model to be evaluated includes but is not limited to a recommendation model, a speech recognition model, an image processing model, etc. It should be noted that the number of sub-models included in the model to be evaluated is not too limited.

[0051] S102, identifying a hierarchical relationship between sub-models, wherein the hierarchical relationship includes at least one of a serial relationship and a parallel relationship.

[0052] In an embodiment of the present application, a hierarchical relationship between sub-models may be identified, wherein the hierarchical relationship includes at least one of a serial relationship and a parallel relationship.

[0053] In one embodiment, identifying the hierarchical relationship between sub-models may include identifying the hierarchy to which the sub-models belong, and identifying the hierarchical relationship between the sub-models based on the hierarchy. The hierarchical relationship between sub-models belonging to the same hierarchy is a parallel relationship, and the hierarchical relationship between sub-models belonging to different hierarchies is a serial relationship.

[0054] In one embodiment, the hierarchical relationship between sub-models includes serial relationship and parallel relationship. Figure 2 As shown, the model to be evaluated is a recommendation model, and the model to be evaluated includes a user interface (UI) layer, a recommendation result layer, and an advertising result layer. The UI layer includes sub-models 1 to 5, the recommendation result layer includes sub-models 6 to 10, and the advertising result layer includes sub-models 11 to 15. The hierarchical relationship between sub-models 1 to 5 is a parallel relationship, the hierarchical relationship between sub-models 6 to 10 is a parallel relationship, and the hierarchical relationship between sub-models 11 to 15 is a parallel relationship. In addition, the hierarchical relationship between any two sub-models is a serial relationship. For example, the hierarchical relationship between sub-model 1 and sub-model 7 is a serial relationship.

[0055] In one embodiment, the hierarchical relationship between sub-models only includes serial relationships. Figure 3 As shown, the model to be evaluated includes a UI layer, a recommendation result layer and an advertising result layer. The UI layer includes sub-model 1, the recommendation result layer includes sub-model 6, and the advertising result layer includes sub-model 11. The hierarchical relationship among sub-models 1, 6, and 11 is a serial relationship.

[0056] In one embodiment, the hierarchical relationship between sub-models only includes parallel relationships. Figure 4 As shown, the model to be evaluated includes a UI layer, the UI layer includes sub-models 1 to 5, and the hierarchical relationship between sub-models 1 to 5 is a parallel relationship.

[0057] S103, distributing the evaluation samples to the sub-models according to the hierarchical relationship.

[0058] It should be noted that there is no excessive limitation on the specific method of distributing the evaluation samples to the sub-models. For example, the same evaluation sample can be distributed to different sub-models. For example, the evaluation samples can be evenly distributed to the sub-models whose hierarchical relationships are parallel.

[0059] In one implementation, sub-models whose hierarchical relationship is a parallel relationship do not have the same evaluation samples, and sub-models whose hierarchical relationship is a serial relationship may have the same evaluation samples.

[0060] In one embodiment, distributing the evaluation samples to the sub-models according to the hierarchical relationship may include generating a distribution strategy according to the hierarchical relationship and distributing the evaluation samples to the sub-models according to the distribution strategy. The distribution strategy may include the target sub-model to which the evaluation samples are distributed.

[0061] In one implementation, the evaluation samples are distributed to the sub-models according to the hierarchical relationship, which may include the following three possible implementations:

[0062] Method 1: In response to the hierarchical relationship including a serial relationship and a parallel relationship, the evaluation samples are distributed to multiple first sub-models, wherein the hierarchical relationship between the multiple first sub-models is a parallel relationship. For any first sub-model, the evaluation samples distributed to any first sub-model are distributed to multiple second sub-models, wherein the hierarchical relationship between the multiple second sub-models is a parallel relationship, wherein the hierarchical relationship between the first sub-model and the second sub-model is a serial relationship, and the distribution time of the first sub-model is earlier than the distribution time of the second sub-model.

[0063] Continue with Figure 2 For example, if the distribution time of sub-models 1 to 5 is earlier than the distribution time of sub-models 6 to 10, and the distribution time of sub-models 6 to 10 is earlier than the distribution time of sub-models 11 to 15, the evaluation samples can be distributed to sub-models 1 to 5, the evaluation samples distributed to sub-models 1 to 5 can be distributed to sub-models 6 to 10 respectively, and the evaluation samples distributed to sub-models 6 to 10 can be distributed to sub-models 11 to 15 respectively.

[0064] Method 2: In response to the hierarchical relationship only including a serial relationship, the evaluation samples are distributed to the third sub-model, and the evaluation samples distributed to the third sub-model are distributed to the fourth sub-model, wherein the hierarchical relationship between the third sub-model and the fourth sub-model is a serial relationship, and the distribution time of the third sub-model is earlier than the distribution time of the fourth sub-model.

[0065] Continue with Figure 3 For example, if the distribution time of sub-model 1 is earlier than that of sub-model 6, and the distribution time of sub-model 6 is earlier than that of sub-model 11, the evaluation samples can be distributed to sub-model 1, the evaluation samples distributed to sub-model 1 can be distributed to sub-model 6, and the evaluation samples distributed to sub-model 6 can be distributed to sub-model 11.

[0066] As another possible implementation, in response to the hierarchical relationship only including a serial relationship, the evaluation samples are distributed to the third sub-model and the fourth sub-model respectively, wherein the hierarchical relationship between the third sub-model and the fourth sub-model is a serial relationship. That is, at this time, the distribution time of the third sub-model is the same as the distribution time of the fourth sub-model, which can achieve parallel distribution of evaluation samples and save sample distribution time.

[0067] Continue with Figure 3 For example, the evaluation samples can be distributed to sub-models 1, 6, and 11 respectively.

[0068] Method 3: The hierarchical relationship only includes parallel relationships, and the evaluation samples are distributed to multiple fifth sub-models, wherein the hierarchical relationship between the multiple fifth sub-models is a parallel relationship.

[0069] Continue with Figure 4 For example, the evaluation samples can be distributed to sub-models 1 to 5.

[0070] S104, obtaining the evaluation result of the sub-model according to the evaluation samples distributed to the sub-model.

[0071] It should be noted that there are not too many restrictions on the types of evaluation results. For example, the evaluation results include but are not limited to click-through rate (CTR), conversion rate (CVR), accuracy, error rate, calculation speed, robustness, etc.

[0072] In one embodiment, obtaining the evaluation result of the sub-model according to the evaluation samples distributed to the sub-model may include inputting the evaluation samples distributed to the sub-model into the sub-model, obtaining the output result of the sub-model, and obtaining the evaluation result of the sub-model according to the output result. It should be noted that there are no excessive restrictions on the type of output results. For example, when the model to be evaluated is a live broadcast recommendation model, the output result may include a live broadcast room, an anchor, etc. When the model to be evaluated is a speech recognition model, the output result may include recognized text. When the model to be evaluated is an image processing model, the output result may include image categories, recognized actions, etc.

[0073] In one embodiment, obtaining the evaluation result of the sub-model according to the sub-model distributed to the evaluation sample may include obtaining the evaluation configuration information of the sub-model, configuring the evaluation environment according to the evaluation configuration information and the evaluation sample distributed to the sub-model, running the evaluation environment, and obtaining the evaluation result of the sub-model based on the evaluation environment. It should be noted that there are no excessive restrictions on the types of evaluation configuration information and evaluation environment. For example, when the model to be evaluated is a live broadcast recommendation model, the evaluation configuration information includes but is not limited to recall data and re-ranking methods, wherein the recall data includes but is not limited to live broadcast rooms and / or anchors with high attention, live broadcast rooms and / or anchors watched by sample users, etc., and the evaluation environment may include an AB test (Abtest) environment.

[0074] In summary, according to the model evaluation method of the embodiment of the present application, the evaluation samples can be distributed to the sub-models according to the hierarchical relationship between the sub-models, and the evaluation results of the sub-models can be obtained according to the evaluation samples distributed to the sub-models. Thus, based on the hierarchical relationship between the sub-models, the evaluation samples can be automatically distributed between the sub-models, and then the evaluation results of the sub-models can be obtained, which improves the flexibility and accuracy of the model evaluation.

[0075] Based on any of the above embodiments, Figure 5 As shown, in step S103, the evaluation samples are distributed to the sub-models according to the hierarchical relationship, including:

[0076] S501, obtaining sample distribution conditions and / or sample distribution ratios of sub-models.

[0077] In the embodiment of the present application, sample distribution conditions and / or sample distribution ratios may be pre-set for the sub-models. It is understandable that different sub-models may correspond to different sample distribution conditions, and different sub-models may correspond to different sample distribution ratios.

[0078] In one embodiment, if the sample distribution ratio of the sub-model is 0, it indicates that the sub-model does not meet the sample distribution conditions at this time. If the sample distribution ratio of the sub-model is not 0, it indicates that the sub-model meets the sample distribution conditions at this time.

[0079] Continue with Figure 2 For example, when the model to be evaluated is a recommendation model and the evaluation sample is the user information of the sample user, the sample distribution conditions of sub-models 1 to 7 and 11 to 15 are all that the gender of the sample user is female, and the sample distribution conditions of sub-models 8 to 10 are all that the gender of the sample user is male.

[0080] Continue with Figure 2 For example, the sample distribution ratio of sub-models 1 to 15 is 20%.

[0081] Continue with Figure 2 For example, the sample distribution ratios of sub-models 1 to 5 and 11 to 15 are both 20%, the sample distribution ratios of sub-models 6 to 7 are both 10%, and the sample distribution ratios of sub-models 8 to 10 are both 0.

[0082] Continue with Figure 2 For example, the sample distribution ratios of sub-models 1 to 5 are all 20%, and the sample distribution ratios of sub-models 6 to 15 are all 0.

[0083] In one implementation, the sample distribution ratio of the sub-model may be obtained based on the sample distribution condition of the sub-model.

[0084] For example, continue with Figure 2 For example, if sub-model 1 does not meet the sample distribution conditions, it means that the evaluation samples are not distributed to sub-model 1, and the sample distribution ratio of sub-model 1 is 0;

[0085] If sub-models 1 to 5 all meet the sample distribution conditions and the sample distribution ratios of sub-models 1 to 5 are the same, then the sample distribution ratios of sub-models 1 to 5 are all 20%;

[0086] If sub-models 6 to 9 all meet the sample distribution conditions and the sample distribution ratios of sub-models 6 to 9 are the same, then the sample distribution ratios of sub-models 6 to 9 are all 25%;

[0087] If the sample distribution ratio of sub-model 1 is 20%, sub-models 6 to 9 all meet the sample distribution conditions, and the sample distribution ratios of sub-models 6 to 9 are the same, the 20% evaluation samples distributed to sub-model 1 can be evenly distributed to sub-models 6 to 9, that is, the 5% evaluation samples distributed to the sub-models can be distributed to sub-models 6 to 9 respectively.

[0088] S502, distribute the evaluation samples to the sub-models according to the hierarchical relationship, sample distribution conditions and / or sample distribution ratio.

[0089] In one embodiment, distributing the evaluation samples to the sub-models according to the hierarchical relationship, the sample distribution conditions and / or the sample distribution ratio may include obtaining the attribute information of the evaluation samples, identifying the sub-models that meet the sample distribution conditions according to the attribute information, and distributing the evaluation samples to the sub-models according to the hierarchical relationship. Thus, when the sub-models that meet the sample distribution conditions are identified, the method can distribute the evaluation samples to the sub-models according to the hierarchical relationship, thereby improving the flexibility of sample distribution.

[0090] It should be noted that there are not too many restrictions on the type of attribute information. For example, when the evaluation sample is the user information of the sample user, the attribute information includes but is not limited to the gender, age, country, etc. of the sample user. When the evaluation sample is a picture, the attribute information includes but is not limited to the size, color, saturation, brightness, etc. of the picture.

[0091] Continue with Figure 2 For example, when the model to be evaluated is a recommendation model and the evaluation sample is the user information of the sample user, the sample distribution conditions of sub-models 1 to 7 and 11 to 15 are all that the gender of the sample user is female, and the sample distribution conditions of sub-models 8 to 10 are all that the gender of the sample user is male. If the gender of the sample user is female, it can be identified that sub-models 1 to 7 and 11 to 15 meet the sample distribution conditions, and the evaluation samples can be distributed to sub-models 1 to 7 and 11 to 15 according to the hierarchical relationship. In addition, it can be identified that sub-models 8 to 10 do not meet the sample distribution conditions, and the evaluation samples may not be distributed to sub-models 8 to 10.

[0092] In one embodiment, according to the hierarchical relationship, the evaluation samples are distributed to sub-models 1 to 7 and 11 to 15, which may include distributing the evaluation samples to sub-models 1 to 5, and the sample distribution ratio of sub-models 1 to 5 is 20%; distributing the evaluation samples distributed to sub-models 1 to 5 to sub-models 6 and 7, respectively, and the sample distribution ratio of sub-models 6 and 7 is 50%; distributing the evaluation samples distributed to sub-models 6 and 7 to sub-models 11 to 15, respectively, and the sample distribution ratio of sub-models 11 to 15 is 20%.

[0093] In one embodiment, distributing the evaluation samples to the sub-models according to the hierarchical relationship, sample distribution conditions and / or sample distribution ratios may include distributing the evaluation samples to the sub-models according to the hierarchical relationship and the sample distribution ratios.

[0094] Continue with Figure 2For example, if the sample distribution ratios of sub-models 1 to 5 and 11 to 15 are both 20%, the sample distribution ratios of sub-models 6 to 7 are both 10%, and the sample distribution ratios of sub-models 8 to 10 are both 0, then the evaluation samples can be distributed to sub-models 1 to 5, and the sample distribution ratios of sub-models 1 to 5 are all 20%, the evaluation samples distributed to sub-model 1 are distributed to sub-models 6 to 7, and the sample distribution ratios of sub-models 6 to 7 are all 10%, and the evaluation samples distributed to sub-models 1 to 5 are respectively distributed to sub-models 11 to 15, and the sample distribution ratios of sub-models 11 to 15 are all 20%.

[0095] Therefore, this method can comprehensively consider the hierarchical relationship between sub-models, sample distribution conditions and / or sample distribution ratios, and distribute evaluation samples to sub-models, which can improve the flexibility and accuracy of sample distribution and thus improve the model evaluation effect.

[0096] Based on any of the above embodiments, Figure 6 As shown, in step S103, the evaluation samples are distributed to the sub-models according to the hierarchical relationship, which may include:

[0097] S601, obtaining identification information of the evaluation sample.

[0098] In the embodiment of the present application, identification information may be set in advance for the evaluation samples to distinguish different evaluation samples. It should be noted that the type of identification information is not too limited, for example, the identification information includes but is not limited to text, characters, numbers, etc.

[0099] In one implementation, when the evaluation sample is user information of a sample user, the identification information may be an account number (Identity Document, ID) of the sample user and a Media Access Control (Media Access Control, MAC) address of a terminal device used by the sample user.

[0100] In one embodiment, the identification information of the evaluation sample may be randomly generated. For example, assuming that the evaluation sample set includes 100 evaluation samples, a number from 1 to 100 may be randomly generated for any evaluation sample as its identification information.

[0101] In one embodiment, the identification information of the evaluation sample is only one. Figure 2 For example, for sub-models 1 to 15, the identification information of evaluation sample A is the number 5.

[0102] In one embodiment, there may be multiple identification information of the evaluation sample. For example, the identification information of the evaluation sample may be different for different sub-models. Figure 2For example, for sub-models 1 to 5, the identification information of evaluation sample A is number 5, for sub-models 6 to 10, the identification information of evaluation sample A is number 80, and for sub-models 11 to 15, the identification information of evaluation sample A is number 33.

[0103] S602: Determine the target sub-model corresponding to the evaluation sample according to the identification information and the hierarchical relationship.

[0104] In one embodiment, determining a target sub-model corresponding to an evaluation sample based on identification information and a hierarchical relationship may include determining a value range of the identification information corresponding to the sub-model based on the hierarchical relationship, identifying that the identification information of the evaluation sample is within a target value range, and determining the sub-model corresponding to the target value range as the target sub-model corresponding to the evaluation sample.

[0105] Continue with Figure 2 For example, assuming that the evaluation sample set includes evaluation samples with identification information ranging from 1 to 100, the value ranges of the identification information corresponding to sub-models 1 to 5 are 1 to 20, 21 to 40, 41 to 60, 61 to 80, and 81 to 100. The value ranges of the identification information corresponding to sub-models 6 to 10 are 1 to 20, 21 to 40, 41 to 60, 61 to 80, and 81 to 100. The value ranges of the identification information corresponding to sub-models 11 to 15 are 1 to 20, 21 to 40, 41 to 60, 61 to 80, and 81 to 100.

[0106] If for sub-models 1 to 5, the identification information of evaluation sample A is the number 5, it can be recognized that the identification information of evaluation sample A is within the target value range of 1 to 20, and sub-model 1 is determined as the target sub-model corresponding to evaluation sample A.

[0107] If the identification information of evaluation sample A is the number 80 for sub-models 6 to 10, it can be recognized that the identification information of evaluation sample A is within the target value range of 61 to 80, and sub-model 9 is determined as the target sub-model corresponding to evaluation sample A.

[0108] If for sub-models 11 to 15, the identification information of evaluation sample A is the number 33, it can be recognized that the identification information of evaluation sample A is within the target value range of 21 to 40, and sub-model 12 is determined as the target sub-model corresponding to evaluation sample A.

[0109] S603, distribute the evaluation samples to the target sub-model.

[0110] Continue with Figure 2 For example, if the target sub-model corresponding to the evaluation sample A includes sub-models 1, 9, and 12, the evaluation sample A can be distributed to sub-models 1, 9, and 12 respectively.

[0111] Therefore, the method can determine the target sub-model corresponding to the evaluation sample according to the identification information of the evaluation sample and the hierarchical relationship between the sub-models, and distribute the evaluation sample to the target sub-model.

[0112] In order to implement the above embodiment, the present application also proposes a model evaluation device.

[0113] Figure 7 It is a block diagram of a model evaluation device according to one embodiment of the present application.

[0114] like Figure 7 As shown, the model evaluation device 100 of the embodiment of the present application includes: an acquisition module 110, a recognition module 120, a distribution module 130 and an evaluation module 140.

[0115] The acquisition module 110 is used to acquire an evaluation sample set and a model to be evaluated, wherein the model to be evaluated includes at least one sub-model;

[0116] The identification module 120 is used to identify the hierarchical relationship between the sub-models, wherein the hierarchical relationship includes at least one of a serial relationship and a parallel relationship;

[0117] The distribution module 130 is used to distribute the evaluation samples to the sub-models according to the hierarchical relationship;

[0118] The evaluation module 140 is used to obtain the evaluation result of the sub-model according to the evaluation samples distributed to the sub-model.

[0119] In one embodiment of the present application, the distribution module 130 is also used to: obtain the sample distribution conditions and / or sample distribution ratio of the sub-model; and distribute the evaluation samples to the sub-model according to the hierarchical relationship, the sample distribution conditions and / or the sample distribution ratio.

[0120] In one embodiment of the present application, the distribution module 130 is also used to: obtain attribute information of the evaluation sample; identify the sub-model that meets the sample distribution condition based on the attribute information; and distribute the evaluation sample to the sub-model based on the hierarchical relationship.

[0121] In one embodiment of the present application, the hierarchical relationship includes a serial relationship and a parallel relationship, and the distribution module 130 is further used to: distribute the evaluation samples to multiple first sub-models, wherein the hierarchical relationship between the multiple first sub-models is a parallel relationship; for any first sub-model, distribute the evaluation samples distributed to any first sub-model to multiple second sub-models, wherein the hierarchical relationship between the multiple second sub-models is a parallel relationship; wherein the hierarchical relationship between the first sub-model and the second sub-model is a serial relationship, and the distribution time of the first sub-model is earlier than the distribution time of the second sub-model.

[0122] In one embodiment of the present application, the hierarchical relationship only includes a serial relationship, and the distribution module 130 is further used to: distribute the evaluation sample to the third sub-model; distribute the evaluation sample distributed to the third sub-model to the fourth sub-model; wherein the hierarchical relationship between the third sub-model and the fourth sub-model is a serial relationship, and the distribution time of the third sub-model is earlier than the distribution time of the fourth sub-model.

[0123] In one embodiment of the present application, the hierarchical relationship only includes a parallel relationship, and the distribution module 130 is further used to: distribute the evaluation sample to a plurality of fifth sub-models, wherein the hierarchical relationship between the plurality of fifth sub-models is a parallel relationship.

[0124] In one embodiment of the present application, the distribution module 130 is further used to: obtain identification information of the evaluation sample; determine the target sub-model corresponding to the evaluation sample based on the identification information and the hierarchical relationship; and distribute the evaluation sample to the target sub-model.

[0125] In one embodiment of the present application, the evaluation module 140 is also used to: obtain the evaluation configuration information of the sub-model; configure the evaluation environment according to the evaluation configuration information and the evaluation sample distributed to the sub-model; run the evaluation environment, and obtain the evaluation result of the sub-model based on the evaluation environment.

[0126] It should be noted that for details not disclosed in the model evaluation device in the embodiment of the present application, please refer to the details disclosed in the model evaluation method in the above embodiment of the present application, which will not be repeated here.

[0127] In summary, the model evaluation device of the embodiment of the present application can distribute the evaluation samples to the sub-models according to the hierarchical relationship between the sub-models, and obtain the evaluation results of the sub-models according to the evaluation samples distributed to the sub-models. Therefore, based on the hierarchical relationship between the sub-models, the evaluation samples can be automatically distributed between the sub-models, and then the evaluation results of the sub-models can be obtained, which improves the flexibility and accuracy of the model evaluation.

[0128] In order to implement the above embodiment, Figure 8 As shown, an embodiment of the present application proposes an electronic device 200, including: a memory 210, a processor 220, and a computer program stored in the memory 210 and executable on the processor 220, and when the processor 220 executes the program, the above-mentioned model evaluation method is implemented.

[0129] The electronic device of the embodiment of the present application can distribute the evaluation samples to the sub-models according to the hierarchical relationship between the sub-models through the processor executing the instructions stored in the memory, and obtain the evaluation results of the sub-models according to the evaluation samples distributed to the sub-models. Thus, based on the hierarchical relationship between the sub-models, the evaluation samples can be automatically distributed between the sub-models, and then the evaluation results of the sub-models can be obtained, which improves the flexibility and accuracy of the model evaluation.

[0130] In order to implement the above embodiment, the present application also proposes a computer-readable storage medium storing a computer program, which implements the above model evaluation method when executed by a processor.

[0131] The computer-readable storage medium of the embodiment of the present application can distribute the evaluation samples to the sub-models according to the hierarchical relationship between the sub-models by storing the computer program and being executed by the processor, and obtain the evaluation results of the sub-models according to the evaluation samples distributed to the sub-models. Thus, based on the hierarchical relationship between the sub-models, the evaluation samples can be automatically distributed between the sub-models, and then the evaluation results of the sub-models can be obtained, thereby improving the flexibility and accuracy of the model evaluation.

[0132] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the above-mentioned processes does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0133] In the embodiments provided in the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0134] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0135] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several requests for a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to perform some or all of the steps of the above-mentioned methods of various embodiments of the present application.

[0136] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0137] The above is a detailed introduction to a model evaluation method, training method, device, equipment and medium disclosed in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A model evaluation method, characterized in that: include: Acquire an evaluation sample set and a model to be evaluated, wherein the model to be evaluated includes at least one sub-model; Identifying a hierarchical relationship between the sub-models, wherein the hierarchical relationship is at least one of a serial relationship and a parallel relationship; Distributing the evaluation samples to the sub-models according to the hierarchical relationship; Obtaining an evaluation result of the sub-model according to the evaluation sample distributed to the sub-model; The step of distributing the evaluation samples to the sub-models according to the hierarchical relationship includes: Generate a distribution strategy according to the hierarchical relationship, and distribute the evaluation samples to the sub-models according to the distribution strategy; and Obtaining the sample distribution conditions and sample distribution ratio of the sub-model; The evaluation samples are distributed to the sub-models according to the hierarchical relationship, the sample distribution conditions and the sample distribution ratio.

2. The method according to claim 1, characterized in that The distributing the evaluation samples to the sub-models according to the hierarchical relationship, the sample distribution condition and the sample distribution ratio includes: Acquiring attribute information of the evaluation sample; According to the attribute information, identifying that the sub-model meets the sample distribution condition; The evaluation samples are distributed to the sub-models according to the hierarchical relationship.

3. The method according to any one of claims 1 to 2, characterized in that: The hierarchical relationship is a serial relationship and a parallel relationship, and the step of distributing the evaluation samples to the sub-models according to the hierarchical relationship includes: Distributing the evaluation samples to a plurality of first sub-models, wherein the hierarchical relationship between the plurality of first sub-models is a parallel relationship; For any first sub-model, distribute the evaluation samples distributed to any first sub-model to multiple second sub-models, wherein the hierarchical relationship between the multiple second sub-models is a parallel relationship; The hierarchical relationship between the first sub-model and the second sub-model is a serial relationship, and the distribution time of the first sub-model is earlier than the distribution time of the second sub-model.

4. The method according to any one of claims 1 to 2, characterized in that: The hierarchical relationship only includes a serial relationship, and the step of distributing the evaluation samples to the sub-models according to the hierarchical relationship includes: distributing the evaluation samples to a third sub-model; Distributing the evaluation samples distributed to the third sub-model to the fourth sub-model; The hierarchical relationship between the third sub-model and the fourth sub-model is a serial relationship, and the distribution time of the third sub-model is earlier than the distribution time of the fourth sub-model.

5. The method according to any one of claims 1-2, characterized in that: The hierarchical relationship only includes a parallel relationship, and the distributing the evaluation samples to the sub-models according to the hierarchical relationship includes: The evaluation samples are distributed to a plurality of fifth sub-models, wherein the hierarchical relationship between the plurality of fifth sub-models is a parallel relationship.

6. The method according to claim 1, characterized in that The distributing the evaluation samples to the sub-models according to the hierarchical relationship includes: Obtaining identification information of the evaluation sample; Determining a target sub-model corresponding to the evaluation sample according to the identification information and the hierarchical relationship; Distribute the evaluation samples to the target sub-model.

7. The method according to claim 1, characterized in that The obtaining the evaluation result of the sub-model according to the evaluation sample distributed to the sub-model includes: Obtaining evaluation configuration information of the sub-model; configuring an evaluation environment according to the evaluation configuration information and the evaluation samples distributed to the sub-model; The evaluation environment is run, and an evaluation result of the sub-model is obtained based on the evaluation environment.

8. A model evaluation device, characterized in that: include: An acquisition module, used to acquire an evaluation sample set and a model to be evaluated, wherein the model to be evaluated includes at least one sub-model; An identification module, used to identify the hierarchical relationship between the sub-models, wherein the hierarchical relationship is at least one of a serial relationship and a parallel relationship; A distribution module, used for distributing the evaluation samples to the sub-models according to the hierarchical relationship; An evaluation module, used for obtaining an evaluation result of the sub-model according to the evaluation sample distributed to the sub-model; The distribution module is further used to generate a distribution strategy according to the hierarchical relationship, and distribute the evaluation samples to the sub-models according to the distribution strategy; and Obtaining the sample distribution conditions and sample distribution ratio of the sub-model; The evaluation samples are distributed to the sub-models according to the hierarchical relationship, the sample distribution conditions and the sample distribution ratio.

9. The device according to claim 8, characterized in that The distribution module is further used for: Acquiring attribute information of the evaluation sample; According to the attribute information, identifying that the sub-model meets the sample distribution condition; The evaluation samples are distributed to the sub-models according to the hierarchical relationship.

10. The device according to any one of claims 8 to 9, characterized in that: The hierarchical relationship is a serial relationship and a parallel relationship, and the distribution module is further used for: Distributing the evaluation samples to a plurality of first sub-models, wherein the hierarchical relationship between the plurality of first sub-models is a parallel relationship; For any first sub-model, distribute the evaluation samples distributed to any first sub-model to multiple second sub-models, wherein the hierarchical relationship between the multiple second sub-models is a parallel relationship; The hierarchical relationship between the first sub-model and the second sub-model is a serial relationship, and the distribution time of the first sub-model is earlier than the distribution time of the second sub-model.

11. The device according to any one of claims 8 to 9, characterized in that: The hierarchical relationship only includes a serial relationship, and the distribution module is further used for: distributing the evaluation samples to a third sub-model; Distributing the evaluation samples distributed to the third sub-model to the fourth sub-model; The hierarchical relationship between the third sub-model and the fourth sub-model is a serial relationship, and the distribution time of the third sub-model is earlier than the distribution time of the fourth sub-model.

12. The device according to any one of claims 8 to 9, characterized in that: The hierarchical relationship only includes a parallel relationship, and the distribution module is further used for: The evaluation samples are distributed to a plurality of fifth sub-models, wherein the hierarchical relationship between the plurality of fifth sub-models is a parallel relationship.

13. The device according to claim 8, characterized in that The distribution module is further used for: Obtaining identification information of the evaluation sample; Determining a target sub-model corresponding to the evaluation sample according to the identification information and the hierarchical relationship; Distribute the evaluation samples to the target sub-model.

14. The device according to claim 8, characterized in that The evaluation module is also used to: Obtaining evaluation configuration information of the sub-model; configuring an evaluation environment according to the evaluation configuration information and the evaluation samples distributed to the sub-model; The evaluation environment is run, and an evaluation result of the sub-model is obtained based on the evaluation environment.

15. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the model evaluation method according to any one of claims 1 to 7 is implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the model evaluation method according to any one of claims 1 to 7 is implemented.

Images