Model performance verification method and device

Abstract

The embodiment of the invention provides a model performance verification method and device, and the method comprises the steps: comparing a real first behavior in sample data with a second behavior extracted by a model, determining a difference part between the first behavior and the second behavior as a third behavior, and determining the same part as a fourth behavior. Entities forming the third behavior and the fourth behavior are determined respectively, and the relative importance degree of the entities is determined based on the relative influence of the entities in the event. The importance degree of the entity in the event reflects the weight of the entity, through a dynamic importance degree weighting mechanism, the first entity in the differential behavior is endowed with the weight matched with the actual application scene, and it is ensured that the entity-level difference with larger influence on the model reasoning process obtains the higher weight. And finally, the performance of the model is determined based on the importance weight through a decline function, so that a model performance verification result better conforms to an actual application scene, and the accuracy of model performance verification is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, in particular to a model performance verification method and device. BACKGROUND

[0002] Currently, the verification of the performance of local database models mainly relies on traditional manual testing methods or general benchmark testing tools.

[0003] Manual testing is inefficient and requires a lot of manpower and time costs. Secondly, manual testing is easily affected by the subjective factors of testers, and the accuracy of the test results. Moreover, manual testing is difficult to cover all business scenarios of the database model, and is prone to test blind spots, leading to one-sided test results.

[0004] Although general benchmark testing tools improve the testing efficiency to some extent, they usually use fixed testing strategies and weight allocation methods, which lack flexibility and adaptability. Different business scenarios have different requirements for the capabilities of database models, and fixed testing strategies cannot be dynamically adjusted according to business needs, making it difficult to achieve comprehensive and accurate verification of the capabilities of database models. In addition, these tools have weak capabilities in semantic understanding and context analysis, and cannot dig out potential problems of database models in different business scenarios, thereby affecting the accuracy of model performance verification. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a model performance verification method and device to improve the accuracy of model performance verification. The specific technical solutions are as follows:

[0006] In the first aspect of the present application, a model performance verification method is first provided, which comprises:

[0007] Obtaining sample data for describing an event and all first behaviors; wherein the first behaviors are behaviors that actually occur in the event;

[0008] Analyzing the sample data by a to-be-verified model to obtain all behaviors that occur in the event as second behaviors;

[0009] Determining behaviors in all the first behaviors that do not belong to the second behaviors as third behaviors; and referring to behaviors in all the first behaviors other than the third behaviors as fourth behaviors;

[0010] Determining entities constituting the third behaviors as first entities, and determining entities constituting the fourth behaviors as second entities;

[0011] determining a relative importance of the first entity relative to the second entity as a first relative importance, wherein the relative importance between two entities is positively correlated with a relative size of an impact of the two entities on the event;

[0012] inputting the first relative importance into a decreasing function to obtain the performance of the to-be-verified model.

[0013] In a possible implementation, the determining the relative importance of the first entity relative to the second entity comprises:

[0014] obtaining keywords previously annotated for the sample data as ground truth keywords;

[0015] determining, in the ground truth keywords, a keyword associated with the first entity as a first keyword and determining a heat of the first keyword, and determining, in the ground truth keywords, a keyword associated with the second entity as a second keyword and determining a heat of the second keyword;

[0016] determining the relative importance of the first entity relative to the second entity according to the heat of the first keyword and the heat of the second keyword, wherein the relative importance is positively correlated with the heat of the first keyword and negatively correlated with the heat of the second keyword.

[0017] In a possible implementation, the method further comprises:

[0018] obtaining keywords previously annotated for the sample data as ground truth keywords;

[0019] extracting, by the to-be-verified model, keywords of the sample data as predicted keywords;

[0020] determining, in the ground truth keywords, a keyword that does not belong to the predicted keywords as a missed keyword, and determining a heat of each missed keyword;

[0021] the inputting the first relative importance into a decreasing function to obtain the performance of the to-be-verified model comprises:

[0022] inputting the first relative importance and the heat of each missed keyword into a decreasing function to obtain the performance of the to-be-verified model.

[0023] In a possible implementation, the method further comprises:

[0024] deleting part of the sample data to obtain residual data;

[0025] restore the deleted data in the incomplete data through the to-be-verified model, to obtain restored data;

[0026] In the restored data, determine data that is different from the sample data as difference data;

[0027] The performance of the to-be-verified model is obtained by inputting the first relative importance into a decreasing function, including:

[0028] The performance of the to-be-verified model is obtained by inputting the first relative importance and the data amount of the difference data into a decreasing function.

[0029] In a possible implementation, the method further includes:

[0030] For each of the first behaviors, determine an impact on the event according to a correlation degree between the behavior and a context behavior, wherein the context behavior is a behavior described by context data of the behavior, and the correlation degree is positively correlated with the impact;

[0031] Determine a behavior whose impact on the event is greater than an upper threshold as a third behavior.

[0032] Delete data for describing the third behavior in the sample data, and take the remaining data as incomplete data.

[0033] In a possible implementation, the method further includes:

[0034] For each of the first behaviors, determine an impact on the event according to a correlation degree between the behavior and a context behavior;

[0035] Determine a behavior whose impact on the event is less than a lower threshold as a fourth behavior.

[0036] In the sample data, determine a position of data for describing the fourth behavior as a secondary data position.

[0037] The method further includes:

[0038] Delete data for describing the third behavior in the sample data, and add noise at the secondary data position of the remaining data to obtain incomplete data.

[0039] In a possible implementation, the method further includes:

[0040] determining the behavior described by the difference data as a fifth behavior; and referring to all the behaviors of the first behaviors except the fifth behavior as a sixth behavior;

[0041] determining the entity constituting the fifth behavior as a third entity; and determining the entity constituting the sixth behavior as a fourth entity;

[0042] determining the relative importance of the third entity relative to the fourth entity as a second relative importance;

[0043] The inputting the first relative importance and the data amount of the difference data into the decreasing function to obtain the performance of the to-be-verified model comprises:

[0044] The inputting the first relative importance, the data amount of the difference data and the second relative importance into the decreasing function to obtain the performance of the to-be-verified model.

[0045] In a second aspect of the embodiment of the present application, a model performance verification device is further provided, and the device comprises:

[0046] a first obtaining module, configured to obtain sample data for describing an event and all first behaviors; wherein the first behaviors are behaviors actually occurring in the event;

[0047] a behavior analysis module, configured to analyze the sample data by a to-be-verified model to obtain all behaviors occurring in the event as second behaviors;

[0048] a first determining module, configured to determine a behavior not belonging to the second behaviors from all the first behaviors as a third behavior; and refer to all the first behaviors except the third behavior as fourth behaviors;

[0049] a second determining module, configured to determine an entity constituting the third behavior as a first entity; and determine an entity constituting the fourth behavior as a second entity;

[0050] a third determining module, configured to determine a relative importance of the first entity relative to the second entity as a first relative importance, wherein the relative importance between two entities is positively correlated with the relative size of the influence of the two entities in the event;

[0051] a first inputting module, configured to input the first relative importance into a decreasing function to obtain the performance of the to-be-verified model.

[0052] In a possible implementation, the third determining module comprises:

[0053] The first determining sub-module is configured to obtain keywords pre-labeled for the sample data as ground truth keywords;

[0054] The second determining sub-module is configured to determine, among the ground truth keywords, a keyword associated with the first entity as a first keyword and determine a heat degree of the first keyword, and determine, among the ground truth keywords, a keyword associated with the second entity as a second keyword and determine a heat degree of the second keyword.

[0055] The third determining sub-module is configured to determine, according to the heat degree of the first keyword and the heat degree of the second keyword, a relative importance degree of the first entity relative to the second entity, wherein the relative importance degree is positively correlated with the heat degree of the first keyword and negatively correlated with the heat degree of the second keyword.

[0056] In a possible implementation, the apparatus further includes:

[0057] The second obtaining module is configured to obtain keywords pre-labeled for the sample data as ground truth keywords.

[0058] The third obtaining module is configured to extract, by the to-be-verified model, keywords of the sample data as predicted keywords.

[0059] The fourth determining module is configured to determine, among the ground truth keywords, a keyword that does not belong to the predicted keywords as a missed keyword, and determine a heat degree of each missed keyword.

[0060] The first input module includes:

[0061] The input first sub-module is configured to input the first relative importance degree and the heat degrees of the missed keywords into a decreasing function to obtain a performance of the to-be-verified model.

[0062] In a possible implementation, the apparatus further includes:

[0063] The data deleting module is configured to delete part of data in the sample data and take the remaining data as incomplete data.

[0064] The data restoring module is configured to restore, by the to-be-verified model, the deleted data in the incomplete data to obtain restored data.

[0065] The fifth determining module is configured to determine, among the restored data, data that is different from the sample data as difference data.

[0066] The first input module includes:

[0067] The input second sub-module is configured to input the first relative importance and the data amount of the difference data into a decreasing function to obtain the performance of the to-be-verified model.

[0068] In a possible implementation, the data deletion module comprises:

[0069] The deletion first sub-module is configured to determine, for each of the first behaviors, an impact on the event according to a correlation degree between the behavior and a context behavior, wherein the context behavior is a behavior described by context data of the data describing the behavior, and the correlation degree is positively correlated with the impact.

[0070] The deletion second sub-module is configured to determine, as a third behavior, a behavior whose impact on the event is greater than an upper threshold.

[0071] The deletion third sub-module is configured to delete, from the sample data, data describing the third behavior, and take remaining data as incomplete data.

[0072] In a possible implementation, the apparatus further comprises:

[0073] The sixth determination module is configured to determine, for each of the first behaviors, an impact on the event according to a correlation degree between the behavior and the context behavior.

[0074] The seventh determination module is configured to determine, as a fourth behavior, a behavior whose impact on the event is less than a lower threshold.

[0075] The eighth determination module is configured to determine, in the sample data, a position of data describing the fourth behavior as a secondary data position.

[0076] The deletion third sub-module comprises:

[0077] The deletion first unit is configured to delete, from the sample data, data describing the third behavior, and add noise at the secondary data position of remaining data to obtain incomplete data.

[0078] In a possible implementation, the apparatus further comprises:

[0079] The ninth determination module is configured to determine a behavior described by the difference data as a fifth behavior, and determine, as a sixth behavior, a behavior other than the fifth behavior in all the first behaviors.

[0080] The tenth determination module is configured to determine an entity constituting the fifth behavior as a third entity, and determine an entity constituting the sixth behavior as a fourth entity.

[0081] The eleventh determining module is configured to determine a relative importance degree of the third entity relative to the fourth entity as a second relative importance degree.

[0082] The input second sub-module comprises:

[0083] The input first unit is configured to input the first relative importance degree, the data volume of the difference data and the second relative importance degree into a decreasing function to obtain the performance of the to-be-verified model.

[0084] In another aspect of the embodiments of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the model performance verification method described above.

[0085] In another aspect of the embodiments of the present application, a computer program product is provided, and the computer program product contains instructions which, when executed on a computer, cause the computer to perform the model performance verification method described above.

[0086] The model performance verification method and device provided by the embodiments of the present application compare the first behavior actually occurring in the sample data with the second behavior extracted by the model, determine the difference between the first behavior and the second behavior as a third behavior, and determine the same part as a fourth behavior. The entities constituting the third behavior and the fourth behavior are determined respectively, and the relative importance degree of the entities is determined based on the relative size of the influence of the entities in the event. The importance degree of the entities in the event reflects the weight of the entities. Through the dynamic importance degree weighting mechanism, the first entity in the difference behavior is given a weight matching the actual application scenario, so that the entity level difference having a greater influence on the model reasoning process obtains a higher weight. Finally, the performance of the model is determined based on the importance weight through the decreasing function, so that the model performance verification result is more consistent with the actual application scenario, and the accuracy of the model performance verification is improved. BRIEF DESCRIPTION OF DRAWINGS

[0087] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced.

[0088] Figure 1 FIG. 1 is a first flowchart of the model performance verification method provided in the embodiments of the present application;

[0089] Figure 2 FIG. 2 is a second flowchart of the model performance verification method provided in the embodiments of the present application;

[0090] Figure 3 FIG. 3 is a third flowchart of the model performance verification method provided in the embodiments of the present application.

[0091] Figure 4 A fourth flowchart of a model performance verification method provided in an embodiment of the present application is shown in FIG. 4.

[0092] Figure 5 A fifth flowchart of a model performance verification method provided in an embodiment of the present application is shown in FIG. 5.

[0093] Figure 6 A structure diagram of a model performance verification device provided in an embodiment of the present application is shown in FIG. 6.

[0094] Figure 7 A structure diagram of an electronic device provided in an embodiment of the present application is shown in FIG. 7. DETAILED DESCRIPTION

[0095] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0096] The embodiments of the present application provide a model performance verification method, as shown in FIG. 1. Figure 1 , Figure 1 A first flowchart of a model performance verification method provided in an embodiment of the present application is shown in FIG. 2, which includes the following steps.

[0097] S101, sample data used to describe an event and all first behaviors are obtained.

[0098] The first behaviors are behaviors that actually occur in the event.

[0099] S102, all behaviors occurring in the event are obtained by analyzing the sample data using a model to be verified, as second behaviors.

[0100] S103, behaviors not belonging to the second behaviors are determined from all the first behaviors, as third behaviors; and behaviors other than the third behaviors from all the first behaviors are referred to as fourth behaviors.

[0101] S104, entities constituting the third behaviors are determined as first entities; and entities constituting the fourth behaviors are determined as second entities.

[0102] S105, a relative importance degree of the first entities relative to the second entities is determined as a first relative importance degree.

[0103] The relative importance degree between two entities is positively correlated with the relative size of the influence of the two entities on the event.

[0104] S106, the first relative importance degree is input into a decreasing function to obtain the performance of the model to be verified.

[0105] By applying the above embodiment, the difference between the first behavior and the second behavior is determined as a third behavior, and the same part is determined as a fourth behavior by comparing the first behavior actually occurring in the sample data with the second behavior extracted by the model. The entities constituting the third behavior and the fourth behavior are determined respectively, and the relative importance degree is determined based on the relative size of the impact of the entities in the event. The importance degree of the entity in the event reflects the weight of the entity, and the first entity in the difference behavior is given a weight matching the actual application scenario through a dynamic importance degree weighting mechanism, ensuring that the entity level difference with greater impact on the model reasoning process obtains a higher weight. Finally, the performance of the model is determined based on the importance weight through a decreasing function, so that the model performance verification result is more in line with the actual application scenario, and the accuracy of model performance verification is improved.

[0106] The above steps S101-S106 will be described in detail below:

[0107] For ease of understanding, the event, behavior and entity appearing in this paper will be described first:

[0108] The event contains behaviors, and the behavior is composed of entities. In an event, different entities interact with each other and drive the development of the event through various behaviors.

[0109] The first behavior is the real behavior in the event, and the second behavior is the prediction of the event behavior by the model. By comparing the first behavior and the second behavior, the part (third behavior) where the model prediction is wrong and the part (fourth behavior) where the model prediction is correct can be found.

[0110] The first entity and the second entity correspond to the constituting elements of the third behavior and the fourth behavior respectively. By analyzing the relative importance degree of the first entity relative to the second entity and inputting it into a decreasing function, the performance of the model to be verified can be evaluated. The relative importance degree reflects the relative size of the impact of the entity in the event. If the model prediction of the behavior of the important entity is more wrong, it means that the model performance is poorer; on the contrary, if the model prediction of the behavior of the important entity is accurate, it means that the model performance is better.

[0111] In step S101, the sample data can be various types of data, depending on the application scenario of the model to be verified. For example, in the image recognition field, the sample data can be a series of images, which can include pictures of different categories, different scenes and different styles; in the log analysis field, the sample data can be log lines containing various information; in the plot related data processing field, the sample data can be plot data, such as video data, scripts, novels, etc. For the convenience of description, only the model performance verification method provided by the embodiment of the application is described by taking the sample data as plot data as an example.

[0112] The sample data can be obtained from a public data set or collected by a verification worker according to specific needs, such as crawling web page texts of a specific topic or collecting environmental data using a sensor device, without specific limitation.

[0113] The first behavior is a behavior that actually occurs in an event, is obtained after comprehensively understanding all information and meanings contained in the sample data, and is an action or activity that actually exists and actually occurs in the event process. The first behavior is used as benchmark real data for measuring the performance of the model. When the sample data is plot data, in a possible embodiment, the behaviors in the plot data for describing the event can be obtained in combination with a domain knowledge base and manual annotation.

[0114] In another possible embodiment, the behaviors that actually occur in the event described in the plot data can be represented in the form of triples. Specifically, triple feature extraction is performed to extract triples of "person-scene-action" from the plot data. Specifically, taking plot data as plot video data (i.e., the sample data is plot video data) as an example, through image recognition and speech recognition technologies, the persons in different scenes and the actions performed by the persons are recognized to obtain triples such as "main character (person)-living room (scene)-drinking tea and fanning (action)". Taking plot data as plot text data such as scripts or novels (i.e., the sample data is plot text data) as an example, the corresponding triple information can be extracted by analyzing and parsing the text through natural language processing technology. In other embodiments, the semantics of the plot data can also be represented in forms other than triples, such as extracting "person-scene", "person-action", "person-scene-action-object", and the like to represent the behaviors that actually occur in the event described in the plot data.

[0115] In step S102, the model to be verified is a model that needs to be evaluated for performance, which is a machine learning model or a large language model trained according to local database data. Depending on the application scenario, the model to be verified is also different. For example, in the field of image recognition, the model to be verified can be an image semantic extraction model, an image analysis model, and the like. In the field of log analysis, the model to be verified can be a log analysis model, a log anomaly positioning model, and the like. In the field of plot-related data processing, the model to be verified can be a plot understanding model, a plot summary model, and the like, without further enumeration.

[0116] The model to be verified analyzes and processes the input sample data to extract all behaviors occurring in the event as the second behavior. The second behavior is the result of the model to be verified understanding the sample data, which reflects the semantic understanding ability of the model to be verified in the current state. That is, the performance of the model to be verified depends on the semantic understanding ability of the model to be verified.

[0117] For example, the aforementioned sample data is plot video data, and the to-be-verified model is a plot understanding model. The plot understanding model reads the input plot video data and judges the triplets (person-scene-action) existing in the plot video data. The triplets are the second behaviors output by the to-be-verified model.

[0118] In step S103, the second behaviors extracted by the to-be-verified model are compared with the first behaviors of the pre-acquired sample data. Behaviors that do not belong to the second behaviors among all the first behaviors are determined as third behaviors, that is, the third behaviors are the difference behaviors existing between the first behaviors and the second behaviors, that is, the behaviors that the to-be-verified model fails to correctly predict, reflecting the missing part of the model prediction. Behaviors other than the third behaviors among all the first behaviors are referred to as fourth behaviors, that is, the fourth behaviors are the same behaviors existing between the first behaviors and the second behaviors, that is, the behaviors that the to-be-verified model correctly predicts.

[0119] The existence of the third behaviors indicates that the model does not completely and accurately understand the semantics of the sample data when processing the sample data, and there may be errors, omissions or misunderstandings. For example, it is assumed that the first behaviors of the pre-acquired plot video data include the triplet "main character (person)-living room (scene)-drinking tea and fanning (action)", and the second behaviors extracted by the plot understanding model (that is, the to-be-verified model) are "supporting actor (person)-living room (scene)-drinking tea (action)". It can be seen that the plot understanding model misses the action information of "fanning", and incorrectly analyzes the person information. The incorrect analysis and part of the missing information are the third behaviors, and the third behaviors are "supporting actor (person)-fanning (action)". The accurately analyzed information is the fourth behavior, and the fourth behavior is "living room (scene)-drinking tea (action)".

[0120] In step S104, entities are objects participating in behaviors. The first entities are objects participating in behaviors that the model fails to predict, and the second entities are objects participating in behaviors that the model accurately predicts. For example, in the aforementioned third behaviors, the first entity is "supporting actor (person)", and in the fourth behaviors, the second entity is "living room (scene)".

[0121] In step S105, the relative importance between two entities and the relative size of their impact on events are positively correlated. This means that if an entity has a greater impact on an event, it has a higher relative importance relative to another entity.

[0122] Taking the aforementioned sample data as plot data as an example, the entities include persons and scenes. The plot data can be divided into high-energy plot and smooth plot according to the tension of the plot and the rhythm of the plot development.

[0123] High-energy plot usually contains intense conflict, major turning point of plot development, while smooth plot is relatively flat, mainly to play a transition and lay the foundation. The information carried by the characters and scenes in high-energy plot and the influence on the plot are greater, so the importance of the characters and scenes in high-energy plot is higher than that in smooth plot.

[0124] In addition, in the scene, the character is an entity, the scene is a thing, and the character is more important than the thing. Further subdividing the character category, the character includes the main character and the supporting character. The main character is the core focus of the whole plot and plays a decisive role in the development of the plot; the supporting character mainly develops activities around the main character and plays a role in assisting, contrasting or promoting the plot development of the main character. Based on the above role positioning and functional differences, the importance of the main character is higher than that of the supporting character. The distinction between the importance of the character and the importance of the entity can give different attention weights to the characters and entities according to the specific needs of different scenes. By constructing such a double weight distribution mechanism (i.e. combination of character weight and entity weight), the model can more closely fit the complexity and diversity of the actual application scene in the performance verification process, thereby improving the accuracy of model performance verification.

[0125] For example, in the third behavior, the first entity is "supporting character (character)", and in the fourth behavior, the second entity is "living room (scene)", the scene is a thing, and the influence of the character on the event is greater than that of the thing, so the relative importance of the first entity "supporting character (character)" is higher than that of the second entity "living room (scene)".

[0126] In step S106, the first relative importance is input to the decreasing function to obtain the performance of the model to be verified. That is, the model performance is negatively correlated with the first relative importance, that is, the higher the importance of the first entity, the more problems the model has in processing sample data, and the worse the performance of the model; on the contrary, the lower the importance of the first entity, the more accurate the semantic understanding of the model to the sample data, and the better the performance of the model.

[0127] In one possible embodiment, the first relative importance can be determined according to the heat of the keywords associated with the entity. Specifically, see Figure 2 , Figure 2 The second flowchart of the model performance verification method provided in the embodiment of the present application includes:

[0128] S101, obtaining sample data for describing an event and all first behaviors.

[0129] The first behavior is a behavior that actually occurs in the event.

[0130] S102, obtaining all behaviors in the event by analyzing the sample data through the model to be verified, as second behaviors.

[0131] S103, determining behaviors in all first behaviors that do not belong to the second behaviors, as third behaviors; and determining other behaviors in all first behaviors except the third behaviors as fourth behaviors.

[0132] S104, determining entities constituting the third behaviors as first entities; and determining entities constituting the fourth behaviors as second entities.

[0133] S1051, obtaining keywords pre-labeled for the sample data as true value keywords.

[0134] S1052, in the true value keywords, determining keywords associated with the first entities as first keywords, and determining the heat of the first keywords; and in the true value keywords, determining keywords associated with the second entities as second keywords, and determining the heat of the second keywords.

[0135] S1053, determining the relative importance of the first entities relative to the second entities as a first relative importance according to the heat of the first keywords and the heat of the second keywords.

[0136] The relative importance is positively correlated with the heat of the first keywords and negatively correlated with the heat of the second keywords. The relative importance between the two entities is positively correlated with the relative size of the influence of the two entities in the event.

[0137] S106, inputting the first relative importance into a decreasing function to obtain the performance of the model to be verified.

[0138] Steps S1051 and S1053 are detailed steps of step S105, and steps S101-S104 and S106 have been described in the foregoing, which will not be repeated here.

[0139] In step S1051, the sample data is pre-labeled with keywords by manual or through specific rules when processing the sample data. The keywords are core words that can highlight the event, and can include character keywords, plot keywords, hot keywords, etc. according to different application scenarios.

[0140] For example, as described in the foregoing, when the sample data is plot video data, if the video highlights the dialogue communication between the main character and the supporting character, i.e., the keywords are character keywords and plot keywords, specifically, words such as “main character”, “supporting character”, “dialogue”, etc. are keywords that highlight the event.

[0141] In step S1052, if a certain true value keyword can directly describe the characteristics, behaviors or closely related events of the first entity, this keyword is the first keyword associated with the first entity; similarly, the second keyword associated with the second entity is determined.

[0142] The heat usually reflects the degree of attention that the keyword receives in a certain field and a certain time period. The heat of the keyword can be determined in various ways, such as using the search volume statistics function of a search engine to view the number of times the keyword is searched within a certain time period, or referring to relevant industry reports, social media discussion heat and other data sources to comprehensively evaluate the degree of attention of the keyword among the target audience. The heat can be represented by a specific numerical value, and the higher the numerical value, the higher the heat.

[0143] For example, the first keyword is the main character A, and the second keyword is the supporting actor B. If the network search volume of the supporting actor B is higher than that of the main character A, then the heat of the second keyword is higher than that of the first keyword.

[0144] In step S1053, the relative importance of the first entity with respect to the second entity is determined by comparing the heat of the first keyword (representing the first entity) and the heat of the second keyword (representing the second entity).

[0145] The relative importance is positively correlated with the heat of the first keyword, that is, the higher the heat of the first keyword, the more prominent and the more attention the first entity receives in the event, and the more important the first entity is relative to the second entity. The relative importance is negatively correlated with the heat of the second keyword, that is, the higher the heat of the second keyword, the greater the proportion of the second entity in the whole, and the lower the importance of the first entity relative to the second entity.

[0146] The above is the influence of keyword heat on importance within the same time range. In different time ranges, the mechanism of keyword heat on importance also needs to consider the time factor.

[0147] Since information is time-sensitive, the degree of attention of hot keywords in different time periods is different. For example, the heat of the first keyword is higher than that of the second keyword in numerical value, but there is a case where the first keyword is a keyword whose heat reached a peak a month ago and then gradually decreased, which may represent a topic or concept that a certain industry or a certain group of people focused on in a certain period of time in the past. The second keyword is a keyword that is currently in the heat rising period or has been in a high heat state, reflecting the current focus of the industry or a certain group of people.

[0148] In this case, although the first keyword has a higher degree of heat, the first keyword gradually decreases in attention over time due to the consideration of the time factor. In contrast, the second keyword corresponds to the second entity that occupies a more important position in the event because it fits the current actual demand, social hot spot or development trend. Therefore, the importance of the first entity relative to the second entity in the event is low. That is, when evaluating the relative importance of the entities, the relative importance of the first entity relative to the second entity can be determined according to the degree of heat of the first keyword, the degree of heat of the second keyword and the time distance corresponding to the keywords. Among them, the relative importance is positively correlated with the degree of heat of the first keyword, negatively correlated with the degree of heat of the second keyword, and positively correlated with the time distance corresponding to the keywords. The farther the time distance, that is, the longer the period in which the semantics represented by the keywords is from the current time, the more obvious the decay of the degree of attention over time, and the lower the contribution to the importance of the first entity. Conversely, the closer the time distance, the greater the influence.

[0149] By applying the above embodiment, the true value keyword of the sample data is obtained, and the first keyword associated with the first entity and the second keyword associated with the second entity are determined in the true value keyword, and the degree of heat thereof is determined respectively, thereby providing a quantitative basis for the relative importance of the first entity relative to the second entity. Compared with the previous subjective experience or intuition for evaluating the relative importance of the entities by manual verification, the evaluation process of the relative importance is more scientific, and the evaluation result is more accurate and reliable. Moreover, in this process, the relative importance between different entities is considered to be different in specific scenarios, and the relative importance is positively correlated with the degree of heat of the first keyword and negatively correlated with the degree of heat of the second keyword. The relative importance of the first entity relative to the second entity can be dynamically adjusted according to the change of the degree of heat of the keywords in different scenarios, so that the model performance verification result fits the actual application scenario demand when the model performance is determined according to the relative importance, and the process does not require manual participation, thereby avoiding time waste caused by human operation errors, delays and the like, thereby improving the comprehensiveness, accuracy and efficiency of the model performance verification, and reducing the labor cost.

[0150] In another possible embodiment, the model performance can also be determined according to the accuracy of the keywords extracted by the to-be-verified model, the degree of heat of the keywords and the first relative importance, specifically, see Figure 3 , Figure 3 A third flowchart of the model performance verification method provided in the embodiments of the present application includes:

[0151] S301, obtaining the keywords pre-labeled for the sample data as true value keywords.

[0152] S302, extracting the keywords of the sample data by the to-be-verified model as predicted keywords.

[0153] S303, determining the keywords not belonging to the predicted keywords in the true value keywords as missing keywords; and determining the heat of each missing keyword.

[0154] The foregoing step S106 includes:

[0155] S1061, inputting the first relative importance degree and the heat of each missing keyword into a decreasing function to obtain the performance of the to-be-verified model.

[0156] Step S301 is equivalent to the foregoing step S1051, which will not be described here again.

[0157] In step S302, the sample data is input into the to-be-verified model, and the model analyzes and processes the sample data according to its own algorithm and rules to extract keywords, which are the predicted keywords. The predicted keywords reflect the understanding and judgment of the model on the sample data.

[0158] In step S303, the true value keywords and the predicted keywords are compared to determine the keywords that exist in the true value keywords but do not appear in the predicted keywords as missing keywords. The missing keywords mean that the model does not capture important information in the sample data. The heat of each missing keyword is determined, and the higher the heat, the higher the importance of the keyword in the sample data or the related field.

[0159] In step S1061, the first relative importance degree and the heat of each missing keyword are input into a decreasing function to obtain the performance of the to-be-verified model, that is, the performance of the to-be-verified model is negatively correlated with the first relative importance degree and the heat of each missing keyword.

[0160] By applying the above embodiment, the performance of the model is calculated by inputting the first relative importance degree and the heat of the missing keywords into a decreasing function, which can convert the performance of the model in keyword extraction into a specific quantitative index, avoiding the uncertainty and ambiguity of the traditional subjective evaluation method, making the evaluation of the model performance more objective and accurate. Secondly, the heat of the missing keywords reflects the importance of the information not extracted by the model. By including the heat of the missing keywords in the model performance evaluation, the model can more comprehensively consider the key content that may be missed in actual application, thus more truly reflecting the understanding and processing ability of the model on the sample data, thereby making the accuracy of the model performance verification higher.

[0161] The performance of the to-be-verified model also depends on the reasoning ability and recovery ability of the to-be-verified model based on the context. Based on this, see Figure 4 , Figure 4 The fourth flowchart of the model performance verification method provided in the embodiments of the present application includes:

[0162] S401, deleting part of the data in the sample data, and taking the remaining data as incomplete data.

[0163] S402, restoring the deleted data in the incomplete data by the to-be-verified model to obtain restored data.

[0164] S403, determining the data that is different from the sample data in the restored data as difference data.

[0165] The foregoing step S106 includes:

[0166] S1062, inputting the first relative importance degree and the data amount of the difference data into a decreasing function to obtain the performance of the to-be-verified model.

[0167] In step S401, the main purpose of this step is to simulate the scenario of data loss. In actual application, data may be lost due to various reasons (such as data acquisition device failure, transmission error, human omission, etc.). By actively deleting part of the data in the sample data to generate incomplete data, the actual data loss condition can be simulated, so as to evaluate the performance of the to-be-verified model in processing incomplete data, that is, to verify the reasoning ability of the to-be-verified model.

[0168] In a possible embodiment, a random deletion manner can be used, that is, a certain proportion or quantity of data is randomly selected from the sample data for deletion.

[0169] In another possible embodiment, deletion can also be performed according to specific rules, for example, targeted deletion according to the feature dimension of the data, time sequence, etc. In order to further improve the accuracy of model performance verification, data with a higher importance degree in the sample data can be deleted. Specifically, the foregoing step S401 includes:

[0170] S4011, for each behavior in all the first behaviors, determining the influence on the event according to the association degree between the behavior and the context behavior.

[0171] The context behavior is a behavior described by context data used to describe the behavior, and the association degree is positively correlated with the influence.

[0172] That is, the association degree between the behavior and the context behavior and the influence on the event are positively correlated, that is, the higher the association degree, the greater the influence on the event.

[0173] S4012, determining the behavior whose influence on the event is greater than an upper threshold as a third behavior.

[0174] The upper threshold value can be set by a professional technician according to work experience or set according to industry specifications, and embodiments of the present application do not make specific limitations. That is, only behaviors with a greater impact on the event are screened out.

[0175] S4013, delete data for describing the third behavior in the sample data, and take the remaining data as the incomplete data.

[0176] In this way, a part of behavior data with a greater impact on the event is removed from the original sample data, thereby obtaining incomplete data.

[0177] In actual application, by determining behaviors with an importance (i.e., an impact on the event) higher than the upper threshold value in all first behaviors as the third behaviors, and deleting the corresponding data (i.e., key information in the sample data) to generate incomplete data, the ability of the model to process key information can be more accurately verified. If the to-be-verified model can still accurately complete or make reasonable judgments in the case of missing key information, it has strong generalization and reasoning ability. This method can generate incomplete data in a targeted manner, avoid the dispersion and lack of depth of model capability testing that may be caused by random deletion of data, and make the model performance verification accuracy higher.

[0178] In step S402, the deleted data in the incomplete data is recovered by the to-be-verified model to obtain recovered data, so as to verify the reasoning ability of the to-be-verified model on the data. The incomplete data is input into the to-be-verified model, and the model predicts and fills the missing data part according to the learning algorithm and parameter setting in the model, so as to realize the recovery of the incomplete data.

[0179] In step S403, by comparing the recovered data and the original sample data, the difference between the recovery result of the to-be-verified model and the true data is determined. These difference data can reflect the accuracy and reliability of the to-be-verified model in recovering the data, and the smaller the amount of difference data, the better the recovery effect of the to-be-verified model, i.e., the stronger the reasoning ability, and vice versa. It indicates that the performance of the to-be-verified model is not good.

[0180] In step S1062, the importance of the first entity relative to the second entity and the amount of difference data between the recovered data of the model and the true data are comprehensively considered to evaluate the performance of the to-be-verified model.

[0181] Applying the above embodiments, incomplete data is obtained by deleting some data from the sample data. This incomplete data is then input into the model to be validated for restoration. Finally, the restored data is compared with the original sample data to obtain the difference data. Based on this, when evaluating the performance of the model to be validated, performance is negatively correlated with the first relative importance and the amount of difference data. This evaluation method comprehensively considers the importance of the first entity relative to the second entity, as well as the degree of restoration error. The amount of difference data reflects the number of errors in model restoration, while the first relative importance assigns different weights to different errors. Data restoration errors with higher importance have a greater impact on model performance. This comprehensive evaluation can more fully and realistically reflect the performance of the model to be validated in practical applications, avoiding evaluation bias caused by focusing on only a single indicator, and improving the accuracy of model performance validation.

[0182] In one possible embodiment, the sensitivity and fault tolerance of the model to be verified to unexpected inputs can be further verified, that is, the anti-interference ability of the model to be verified can be verified. This can be achieved by adversarial testing, specifically, through the following methods:

[0183] For each of the first actions, determine its impact on the event based on the degree of association between the action and the context actions.

[0184] The action whose impact on the event is less than the lower threshold is identified as the fourth action.

[0185] The lower threshold can be set by professionals based on their experience or according to industry standards; this embodiment of the invention does not specifically limit this. The lower threshold and the aforementioned upper threshold can be the same or different; this is not specifically limited here. It is understood that when the lower threshold and the upper threshold are different, the upper threshold is greater than the lower threshold. After determining the fourth row, the position of the data used to describe the fourth row in the sample data is determined as the secondary data position.

[0186] The aforementioned step S4013 includes:

[0187] S40131, delete the data used to describe the third behavior in the sample data, and add noise at the minor data positions of the remaining data to obtain incomplete data.

[0188] When the sample data is narrative video data, noise can be interfering audio or video frames, such as white noise, ambient noise (e.g., market noise, traffic noise), or advertising pages. When the sample data is narrative text data, interfering data can be noise words, such as irrelevant modifiers, punctuation marks, etc., which will not be listed exhaustively here.

[0189] By using the above embodiment, in the actual application process of the to-be-verified model, not only the key information may be missing, but also the to-be-verified model may be affected by various noises and interferences. By adding interferences at positions where the data with low importance degree is located, the complex data environment can be simulated, so that the anti-interference ability of the to-be-verified model in the actual application can be verified in the subsequent model performance verification process.

[0190] Based on the above data recovery mechanism (i.e., the word extraction and blank filling mechanism) and the adversarial test, the context reasoning and recovery ability of the to-be-verified model in the noise environment can be verified. Specifically, the context reasoning and recovery ability of the to-be-verified model in the noise environment can be verified by the following methods, including:

[0191] The behavior described by the difference data is determined as a fifth behavior, and all behaviors in the first behaviors except the fifth behavior are referred to as a sixth behavior.

[0192] The entity constituting the fifth behavior is determined as a third entity, and the entity constituting the sixth behavior is determined as a fourth entity.

[0193] The relative importance degree of the third entity relative to the fourth entity is determined as a second relative importance degree.

[0194] The foregoing step S1062 includes:

[0195] S10621, the first relative importance degree, the data amount of the difference data, and the second relative importance degree are input into a decreasing function to obtain the performance of the to-be-verified model.

[0196] That is, the performance of the to-be-verified model is negatively correlated with the first relative importance degree, the data amount of the difference data, and the second relative importance degree.

[0197] The second relative importance degree is used to measure the contribution of each part of the difference data to the whole sample data. When the second importance degree is high, it indicates that the data that needs to be recovered by the to-be-verified model has a greater impact on the context reasoning and recovery ability of the to-be-verified model in the noise environment. If the to-be-verified model makes an error in the recovery of the data, the understanding and reasoning of the sample data will be greatly deviated, and therefore the performance of the to-be-verified model is poor.

[0198] That is, in the performance verification process of the to-be-verified model, the semantic understanding ability of the to-be-verified model is measured by the first relative importance degree, and the context reasoning and recovery ability of the to-be-verified model in the noise environment is measured by the data amount of the difference data and the second relative importance degree.

[0199] The model performance is determined by comprehensively considering the first relative importance, the data amount of the difference data and the second relative importance, which comprehensively considers the semantic understanding ability of the model and the context reasoning and recovery ability of the model in a noisy environment, avoids the limitation of single index evaluation, and can more comprehensively verify the model performance.

[0200] In order to more clearly describe the model performance verification method provided by the embodiment of the application, the sample data is taken as the plot data, and the model to be verified is taken as a plot understanding model, and the model performance verification method is described by taking the plot understanding model as an example in combination with the drawings. Referring to FIG. 1, the model performance verification method provided by the embodiment of the application includes the following steps. Figure 5 , Figure 5 FIG. 5 is a fifth flow diagram of the model performance verification method provided by the embodiment of the application.

[0201] The plot content is taken as the input of the entire model performance verification process, and the data processing includes plot content analysis, triple extraction, keyword extraction and context word extraction, that is, the steps of extracting semantics by the model to be verified, extracting keywords, deleting part of the data and adding interference data.

[0202] The data processing result is obtained, and the data processing result includes high-energy plots and smooth plots obtained by plot content analysis, “person-action-scene” obtained by triple extraction, content keywords obtained by keyword extraction and content obtained by context word extraction.

[0203] The plot content analysis is also a preliminary analysis of the plot content, the high-energy plot is obtained by identifying the climax or key part in the plot, and the smooth plot is obtained by identifying the smooth transition plot in the plot.

[0204] The triple extraction is also the extraction of triple information of persons, actions and scenes from the plot. The keyword extraction is also the extraction of keywords in the plot. The context word extraction is also the extraction of related words according to the context to form the content after word extraction.

[0205] After obtaining the data processing result, the data post-processing is performed on the data processing result. Specifically, the processed data is processed by the attention module through the attention mechanism, the plot content is associated with the real-time hot spot, that is, the real-time hot spot association is performed, and the hot keyword is highlighted. The noise data is introduced through the noise injection module, which can be used for robustness test of the model. After the data is post-processed, the model is evaluated, that is, the performance of the model to be verified is verified, the semantic extraction, hot keyword extraction and completion test (that is, the data recovery mechanism) of the post-processed data are performed by the model to be verified, so as to perform the semantic integrity evaluation, the ability evaluation whether the model to be verified can accurately identify the hot keyword in the current application scene (that is, the hit rate calculation) and the completion test.

[0206] The model score of the to-be-verified model is calculated by weighted scoring. Specifically, in the process of semantic integrity evaluation, semantic weighted scoring is performed, which is equivalent to determining the performance of the to-be-verified model according to the first relative importance in the foregoing step. In the hit rate calculation process, keyword weighted scoring is performed, which is equivalent to determining the performance of the to-be-verified model according to the data amount of the difference data in the foregoing step. In the completion test process, word digging scoring is performed, which is equivalent to determining the performance of the to-be-verified model according to the second relative importance in the foregoing step.

[0207] The semantic weighted score, the keyword weighted score and the word digging score are weighted and summarized to obtain the model score of the to-be-verified model as the output of the entire model performance verification process.

[0208] Corresponding to the foregoing model performance verification method, an embodiment of the present application further provides a model performance verification device, which is described with reference to Figure 6 , Figure 6 A structural schematic diagram of the model performance verification device provided in the embodiment of the present application comprises:

[0209] The first obtaining module 601 is configured to obtain sample data for describing an event and all first behaviors; wherein the first behaviors are behaviors actually occurring in the event;

[0210] The behavior analysis module 602 is configured to analyze all behaviors occurring in the event by the to-be-verified model to obtain the second behaviors;

[0211] The first determining module 603 is configured to determine behaviors not belonging to the second behaviors among all the first behaviors as the third behaviors; and determine other behaviors except the third behaviors among all the first behaviors as the fourth behaviors;

[0212] The second determining module 604 is configured to determine entities constituting the third behaviors as the first entities; and determine entities constituting the fourth behaviors as the second entities;

[0213] The third determining module 605 is configured to determine a relative importance of the first entities relative to the second entities as the first relative importance, wherein the relative importance between two entities is positively correlated with the relative size of the influence of the two entities in the event;

[0214] The first input module 606 is configured to input the first relative importance to a decreasing function to obtain the performance of the to-be-verified model.

[0215] By applying the above embodiment, by comparing the first behavior actually occurring in the sample data with the second behavior extracted by the model, a difference part between the first behavior and the second behavior is determined as a third behavior, and a same part is determined as a fourth behavior. Entities constituting the third behavior and the fourth behavior are respectively determined, and a relative importance degree thereof is determined based on a relative size of an influence of the entities in the event. The importance degree of the entities in the event reflects a weight of the entities, and by a dynamic importance degree weighting mechanism, a first entity in the difference behavior is given a weight matched with an actual application scenario, so as to ensure that an entity level difference having a greater influence on a model reasoning process obtains a higher weight. Finally, a performance of the model is determined based on the importance weight by a decreasing function, so that a model performance verification result is more consistent with an actual application scenario, and an accuracy of model performance verification is improved.

[0216] In a possible implementation, the third determining module comprises:

[0217] The first determining sub-module is configured to acquire a keyword pre-labeled for the sample data as a true value keyword.

[0218] The second determining sub-module is configured to determine, in the true value keyword, a keyword associated with the first entity as a first keyword, and determine a heat degree of the first keyword; and determine, in the true value keyword, a keyword associated with the second entity as a second keyword, and determine a heat degree of the second keyword.

[0219] The third determining sub-module is configured to determine, according to the heat degree of the first keyword and the heat degree of the second keyword, a relative importance degree of the first entity relative to the second entity; wherein the relative importance degree is positively correlated with the heat degree of the first keyword, and negatively correlated with the heat degree of the second keyword.

[0220] In a possible implementation, the apparatus further comprises:

[0221] The second acquiring module is configured to acquire a keyword pre-labeled for the sample data as a true value keyword.

[0222] The third acquiring module is configured to extract, by the model to be verified, a keyword of the sample data as a predicted keyword.

[0223] The fourth determining module is configured to determine, in the true value keyword, a keyword not belonging to the predicted keyword as a missed keyword; and determine a heat degree of each missed keyword.

[0224] The first input module comprises:

[0225] The input first submodule is configured to input the first relative importance degree and the heat degree of each missing keyword into a decreasing function to obtain the performance of the to-be-verified model.

[0226] In a possible implementation, the apparatus further includes:

[0227] The data deletion module is configured to delete part of data in the sample data and take the remaining data as incomplete data.

[0228] The data recovery module is configured to recover the deleted data in the incomplete data by using the to-be-verified model to obtain recovered data.

[0229] The fifth determination module is configured to determine, in the recovered data, data that is different from the sample data as difference data.

[0230] The first input module includes:

[0231] The input second submodule is configured to input the first relative importance degree and the data amount of the difference data into a decreasing function to obtain the performance of the to-be-verified model.

[0232] In a possible implementation, the data deletion module includes:

[0233] The deletion first submodule is configured to determine, for each of the first behaviors, an impact on the event according to the association degree between the behavior and the context behavior, wherein the context behavior is a behavior described by context data of data used to describe the behavior, and the association degree is positively correlated with the impact.

[0234] The deletion second submodule is configured to determine, as a third behavior, a behavior that has an impact on the event greater than an upper threshold.

[0235] The deletion third submodule is configured to delete, in the sample data, data used to describe the third behavior and take the remaining data as incomplete data.

[0236] In a possible implementation, the apparatus further includes:

[0237] The sixth determination module is configured to determine, for each of the first behaviors, an impact on the event according to the association degree between the behavior and the context behavior.

[0238] The seventh determination module is configured to determine, as a fourth behavior, a behavior that has an impact on the event less than a lower threshold.

[0239] An eighth determining module is configured to determine, in the sample data, a position where data describing the fourth behavior is located as a secondary data position;

[0240] The third deleting sub-module comprises:

[0241] A first deleting unit is configured to delete, in the sample data, data describing the third behavior and add noise at the secondary data position of the remaining data to obtain incomplete data.

[0242] In a possible implementation, the apparatus further comprises:

[0243] A ninth determining module is configured to determine a behavior described by the difference data as a fifth behavior, and determine all behaviors in the first behaviors except the fifth behavior as a sixth behavior;

[0244] A tenth determining module is configured to determine an entity constituting the fifth behavior as a third entity, and determine an entity constituting the sixth behavior as a fourth entity;

[0245] An eleventh determining module is configured to determine a relative importance degree of the third entity relative to the fourth entity as a second relative importance degree;

[0246] The second inputting sub-module comprises:

[0247] A first inputting unit is configured to input the first relative importance degree, a data amount of the difference data and the second relative importance degree into a decreasing function to obtain the performance of the to-be-verified model.

[0248] Embodiments of the present application also provide an electronic device, as shown in the accompanying drawings, comprising a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702 and the memory 703 complete mutual communication through the communication bus 704, Figure 7

[0249] The memory 703 is configured to store a computer program;

[0250] The processor 701 is configured to execute the program stored in the memory 703 to implement the following steps:

[0251] Obtain sample data describing an event and all first behaviors, wherein the first behaviors are behaviors actually occurring in the event;

[0252] Obtain all behaviors occurring in the event by analyzing the sample data through a to-be-verified model as second behaviors;

[0253] ​Determine the behavior that does not belong to the second behavior among all the first behaviors as a third behavior; and determine the behavior other than the third behavior among all the first behaviors as a fourth behavior;

[0254] Determine the entity constituting the third behavior as a first entity; and determine the entity constituting the fourth behavior as a second entity;

[0255] Determine the relative importance of the first entity relative to the second entity as a first relative importance; wherein the relative importance between two entities is positively correlated with the relative size of the influence of the two entities on the event;

[0256] Input the first relative importance into a decreasing function to obtain the performance of the to-be-verified model.

[0257] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0258] The communication interface is used for communication between the terminal and other devices.

[0259] The memory can include a Random Access Memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0260] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0261] In yet another embodiment provided by the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program, when executed by a processor, implements the model performance verification method according to any one of the above embodiments.

[0262] In yet another embodiment provided by the present application, a computer program product containing instructions, which when executed on a computer, causes the computer to perform the model performance verification method according to any one of the above embodiments.

[0263] In the above embodiments, the implementation can be wholly or partially in software, hardware, firmware or any combination thereof. When implemented in software, the implementation can be in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer program instructions generate the processes or functions according to the embodiments of the present application, wholly or partially. The computer can be a general purpose computer, a special purpose computer, a computer network or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD) or a semiconductor medium (for example, solid state disk (SSD)) etc.

[0264] It should be noted that, in this document, the terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0265] Each of the embodiments in the specification is described in a relevant manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0266] The above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for verifying model performance, characterized in that, The method includes: Obtain sample data to describe the event and all first behaviors; wherein the first behaviors are behaviors that actually occur in the event; The sample data is analyzed using the model to be verified to obtain all the behaviors that occurred in the event, which are then used as the second behavior. Among all the first actions, identify the actions that do not belong to the second action and designate them as the third action; and designate all other actions among all the first actions besides the third action as the fourth action. The entity constituting the third action is identified as the first entity; the entity constituting the fourth action is identified as the second entity. The relative importance of the first entity relative to the second entity is determined as the first relative importance; wherein, the relative importance between the two entities is positively correlated with the relative magnitude of the influence of the two entities in the event; The first relative importance is input into a decreasing function to obtain the performance of the model to be verified.

2. The method according to claim 1, characterized in that, Determining the relative importance of the first entity relative to the second entity includes: Obtain the keywords that have been pre-annotated to the sample data and use them as truth keywords; Among the truth keywords, the keyword associated with the first entity is determined as the first keyword, and the popularity of the first keyword is determined; among the truth keywords, the keyword associated with the second entity is determined as the second keyword, and the popularity of the second keyword is determined. Based on the popularity of the first keyword and the popularity of the second keyword, the relative importance of the first entity relative to the second entity is determined; wherein the relative importance is positively correlated with the popularity of the first keyword and negatively correlated with the popularity of the second keyword.

3. The method according to claim 1, characterized in that, The method further includes: Obtain the keywords that have been pre-annotated to the sample data and use them as truth keywords; Keywords from the sample data are extracted using the model to be validated and used as predicted keywords. Among the true keywords, keywords that do not belong to the predicted keywords are identified as omitted keywords; and the popularity of each omitted keyword is determined. The step of inputting the first relative importance into a decreasing function to obtain the performance of the model to be verified includes: The first relative importance and the popularity of each of the omitted keywords are input into a decreasing function to obtain the performance of the model to be verified.

4. The method according to claim 1, characterized in that, The method further includes: Some data is deleted from the sample data, and the remaining data is regarded as incomplete data. The deleted data in the incomplete data is restored by using the model to be verified to obtain the restored data; In the restored data, data that differs from the sample data are identified as difference data; The step of inputting the first relative importance into a decreasing function to obtain the performance of the model to be verified includes: The first relative importance and the amount of difference data are input into a decreasing function to obtain the performance of the model to be verified.

5. The method according to claim 4, characterized in that, The step of deleting some data from the sample data and using the remaining data as incomplete data includes: For each of all the first behaviors, the impact on the event is determined based on the degree of correlation between the behavior and the context behaviors; wherein, the context behaviors are behaviors described by context data used to describe the behavior, and the degree of correlation is positively correlated with the impact; The action that has an impact on the event greater than the upper threshold is identified as the third action; The data describing the third behavior is deleted from the sample data, and the remaining data is treated as incomplete data.

6. The method according to claim 5, characterized in that, The method further includes: For each of the first behaviors, the impact on the event is determined based on the degree of association between the behavior and the context behaviors; The action whose impact on the event is less than a lower threshold is identified as the fourth action. In the sample data, the position of the data used to describe the fourth behavior is determined as the secondary data position; The step of deleting the data describing the third behavior from the sample data and treating the remaining data as incomplete data includes: The data describing the third behavior is deleted from the sample data, and noise is added to the secondary data positions of the remaining data to obtain incomplete data.

7. The method according to claim 4, characterized in that, The method further includes: The behavior described by the difference data is identified as the fifth behavior; and all other behaviors among the first behaviors besides the fifth behavior are referred to as the sixth behavior. The entity constituting the fifth action is identified as the third entity; and the entity constituting the sixth action is identified as the fourth entity. The relative importance of the third entity relative to the fourth entity is determined as the second relative importance. The step of inputting the first relative importance and the amount of difference data into a decreasing function to obtain the performance of the model to be verified includes: The first relative importance, the amount of the differential data, and the second relative importance are input into a decreasing function to obtain the performance of the model to be verified.

8. A model performance verification device, characterized in that, The device includes: The first acquisition module is used to acquire sample data describing the event and all first behaviors; wherein, the first behaviors are behaviors that actually occur in the event; The behavior analysis module is used to analyze the sample data through the model to be verified to obtain all behaviors that occur in the event, as the second behavior; The first determining module is used to determine, among all the first actions, the actions that do not belong to the second action, as the third action; and to refer to all the other actions among all the first actions other than the third action as the fourth action. The second determining module is used to determine the entity constituting the third behavior as the first entity; and to determine the entity constituting the fourth behavior as the second entity. The third determining module is used to determine the relative importance of the first entity relative to the second entity, as the first relative importance, wherein the relative importance between the two entities is positively correlated with the relative magnitude of the influence of the two entities in the event; The first input module is used to input the first relative importance into the decrementing function to obtain the performance of the model to be verified.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-7.

Images