Method, device, electronic device and storage medium for evaluating object performance

Abstract

The present disclosure provides a method and device for evaluating the performance of an object, electronic equipment and storage medium, relating to the technical field of artificial intelligence, in particular to the field of natural language processing. The specific implementation scheme is: processing original text data by a target object to be evaluated to obtain an importance index of each character data in the original text data; determining original evidence text in the original text data according to the importance index of each character data in the original text data; determining reference evidence text corresponding to the original text data; and determining a performance index of the target object according to the original evidence text and the reference evidence text to evaluate the performance of the target object.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more particularly to the field of natural language processing. More specifically, this disclosure provides a method, apparatus, electronic device, storage medium, and computer program product for evaluating the performance of an object. Background Technology

[0002] Deep learning models have achieved significant success in many natural language processing tasks, but they are often used as black boxes, with their internal processing mechanisms remaining opaque to users. This makes users distrustful of the outputs of deep learning models, increasing the difficulty of implementation, especially in specialized fields such as medicine and law. Furthermore, when models exhibit poor performance or robustness, it is difficult to improve and optimize them due to a lack of understanding of their internal mechanisms. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, storage medium, and computer program product for evaluating the performance of an object.

[0004] According to one aspect of this disclosure, a method for evaluating the performance of an object is provided, comprising: processing raw text data using the target object to be evaluated to obtain importance indices for each of multiple character data in the raw text data; determining the original evidence text in the raw text data based on the importance indices for each of the multiple character data in the raw text data; determining the reference evidence text corresponding to the raw text data; and determining the performance indices of the target object based on the original evidence text and the reference evidence text, so as to evaluate the performance of the target object.

[0005] According to another aspect of this disclosure, an apparatus for evaluating the performance of an object is provided, comprising: an importance determination module, an original evidence determination module, a reference evidence determination module, and a performance determination module. The importance determination module processes original text data using the target object to be evaluated to obtain importance indices for each of multiple character data in the original text data. The original evidence determination module determines the original evidence text in the original text data based on the importance indices for each of the multiple character data in the original text data. The reference evidence determination module determines the reference evidence text corresponding to the original text data. The performance determination module determines the performance index of the target object based on the original evidence text and the reference evidence text, thereby evaluating the performance of the target object.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods provided in this disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods provided in this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods provided in this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is a schematic diagram illustrating an application scenario of the method and apparatus for evaluating the performance of an object according to embodiments of this disclosure;

[0012] Figure 2 This is a schematic flowchart of a method for evaluating the performance of an object according to an embodiment of the present disclosure;

[0013] Figure 3A This is a schematic flowchart of a method for evaluating the performance of an object according to another embodiment of the present disclosure;

[0014] Figure 3B This is a schematic diagram of a method for evaluating the performance of an object according to another embodiment of the present disclosure;

[0015] Figure 4A This is a schematic flowchart of a method for evaluating the performance of an object according to another embodiment of the present disclosure;

[0016] Figure 4B This is a schematic diagram of a method for evaluating the performance of an object according to another embodiment of the present disclosure;

[0017] Figure 5 This is a schematic structural block diagram of an apparatus for evaluating the performance of an object according to embodiments of the present disclosure; and

[0018] Figure 6 This is a structural block diagram of an electronic device used to implement the method for evaluating the performance of an object according to embodiments of the present disclosure. Detailed Implementation

[0019] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0020] Figure 1 This is a schematic diagram illustrating an application scenario of the method and apparatus for evaluating the performance of an object according to embodiments of this disclosure.

[0021] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0022] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0023] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0024] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as performance index values ​​of the generated target objects) to the terminal devices.

[0025] It should be noted that the method for evaluating the performance of an object provided in this disclosure embodiment can generally be executed by server 105. Correspondingly, the apparatus for evaluating the performance of an object provided in this disclosure embodiment can generally be located in server 105. The method for evaluating the performance of an object provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the apparatus for evaluating the performance of an object provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0026] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0027] Figure 2 This is a schematic flowchart of a method for evaluating the performance of an object according to an embodiment of the present disclosure.

[0028] like Figure 2 As shown, the method 200 for evaluating the performance of an object may include operations S210 to S240.

[0029] In operation S210, the original text data is processed using the target object to be evaluated to obtain the importance index of each character data in the original text data.

[0030] For example, the target object may include a deep learning model, which can be a model used to implement Natural Language Processing (NLP) tasks. NLP tasks may include, for example, sentiment analysis, text similarity calculation, and reading comprehension. Sentiment analysis represents analyzing the positive and negative sentiment, text similarity calculation may include determining similar and dissimilar results or similarity values, and reading comprehension may include determining the response text for the question text from candidate texts. The deep learning model may be BERT, etc., and this disclosure does not limit the deep learning model.

[0031] For example, the target object may also include model interpretation methods, which are used to explain the principles or basis of the output results of deep learning models. Model interpretation methods may include, for example, attention-based algorithms, integrated gradient algorithms, LIME (Local Interpretable Model-agnostic Explanations), SHAP (SHapley Additive explanation), etc. This disclosure does not limit the model interpretation methods.

[0032] For example, the construction of the raw text data is related to the task being handled by the target audience. For instance, for a sentiment analysis task, the raw text data could include a single sentence, such as "A particularly professional photography shop with excellent service." For text similarity calculation, the raw text data could include two sentences, such as "wind-resistant plants" and "weak wind-resistant plants." For reading comprehension tasks, the raw text data could include a question and a paragraph, with the paragraph containing the answer to the question.

[0033] For example, raw text data includes multiple character data, each of which can correspond to a sentence, a word segment, or a character. When each character data corresponds to a single character, the granularity of the character data is small, thus ensuring accuracy.

[0034] For example, importance metrics can characterize the degree of importance of each character in the original text data to the output of a deep learning model. For instance, in a text similarity calculation task, it is necessary to determine the similarity between the text data "wind-resistant plants" and the text data "weak wind-resistant plants". The character data "weak" has a higher importance metric in these two text data, indicating that the character data "weak" has a greater impact on the similarity between the two text data.

[0035] For example, raw text data can be input into a deep learning model. During the processing of this raw text data, the deep learning model can obtain the weights corresponding to each character in the raw text data based on an attention mechanism, and these weights can be used as an importance indicator. Alternatively, during the processing of raw text data, the deep learning model can obtain the gradients returned by each character to determine the importance indicator.

[0036] It should be noted that the embodiments disclosed herein are used to evaluate performance, and the importance index obtained by this operation is an estimated value, which may differ from the true importance of character data to the model prediction.

[0037] In operation S220, an original evidence text in the original text data is determined according to the respective importance indicators of a plurality of character data in the original text data.

[0038] For example, the plurality of character data may be sorted in descending order of the importance indicators to obtain a sequence, and then a predetermined number of leading character data in the sequence or a predetermined proportion of leading character data in the sequence is determined as the original evidence text.

[0039] For example, the original text data is "a very professional photography shop with good service attitude", and the predetermined number is 4, where the first four character data in the sequence are "de", "photography", "image", and "shop", then "de photography image shop" may be determined as the original evidence text.

[0040] In operation S230, a reference evidence text corresponding to the original text data is determined.

[0041] For example, the original text data may be labeled in advance, and the labeled character data is used as the reference evidence text. For example, for the original text "a very professional photography shop with good service attitude", the pre-labeled reference evidence text may include "professional shop" and "good attitude".

[0042] In operation S240, a performance indicator of a target object is determined according to the original evidence text and the reference evidence text, so as to evaluate the performance of the target object.

[0043] For example, the performance indicator may include accuracy, recall rate, F1 (F1-score) value, MAP (Mean Average Precision) value, and the like.

[0044] For example, the performance of the target object may include the level of accuracy of a deep learning model in processing the original text data. For example, for a sentiment analysis task, whether the deep learning model accurately obtains the positive and negative sentiments expressed in the original text data. For example, for a similarity calculation task, whether the deep learning model accurately obtains the similarity result between two texts. For example, for a reading comprehension task, whether the deep learning model accurately marks the start position and the end position of the answer text for the question in a paragraph.

[0045] For another example, the performance of the target object may further include rationality, and the rationality may represent whether the original evidence text obtained by using the deep learning model and a model interpretation method conforms to the reference evidence text. Further, if the reference evidence text is manually confirmed, the rationality may represent whether the original evidence text obtained by using the deep learning model and the model interpretation method conforms to human cognition.

[0046] For example, the performance of the target object can also include loyalty, which can characterize the degree of consistency between the original evidence text determined by the deep learning model and the model interpretation method and the reference evidence text. This consistency can be understood as the degree of fit between the model interpretation method and the processing mechanism of the deep learning model.

[0047] For example, the original evidence text and the reference evidence text can be compared to determine the degree of similarity between them. This degree of similarity can reflect the performance of the target object. For example, if the degree of similarity is high, the performance of the target object can be high. This high performance may include at least one of the following: high accuracy of the deep learning model, high rationality of the model interpretation method, or high fidelity of the model interpretation method.

[0048] This disclosure allows for the evaluation of the performance of deep learning models and model interpretation methods by comparing reference evidence text with original evidence text, thereby explaining the model's processing mechanism.

[0049] It should be noted that during the evaluation of the target object, a dataset containing multiple original text data can be used to improve the accuracy of the evaluation results by increasing the amount of data. For example, the dataset can include a Chinese dataset and an English dataset, each containing a sentiment analysis task dataset, a text similarity calculation task dataset, and a reading comprehension task dataset. For example, one original text data in the Chinese sentiment analysis dataset is "a particularly professional photography shop with good service attitude".

[0050] The following provides a detailed explanation of the method for determining importance indicators. In this embodiment, the operation of processing the original text data using the target object to be evaluated to obtain the importance indicators of each character data in the original text data may include the following operations: processing the original text data using a deep learning model to obtain intermediate results and output results; the output results are determined based on the intermediate results, and then the intermediate results are processed using a model interpretation method to obtain the importance indicators of each character data in the original text data.

[0051] For example, during the processing of raw text data, deep learning models can generate intermediate results such as gradients, weights, and token vectors. Each token vector corresponds to a character in the text data processed by the model. These intermediate results can be processed using model interpretation methods to obtain importance indicators. This disclosure does not limit the working principle of the model interpretation method.

[0052] For example, the performance of different target objects can be accurately evaluated by controlling variables. For instance, the first target object includes a first deep learning model and a first model interpretation method; the second target object includes a first deep learning model and a second model interpretation method; and the third target object includes a second deep learning model and a first model interpretation method. Method 200 can be executed for the first, second, and third target objects respectively, thereby obtaining performance P1, performance P2, and performance P3 corresponding to the first target object. By comparing performance P1 and performance P2, the performance (e.g., reasonableness and fidelity) of the first and second model interpretation methods can be obtained. By comparing performance P1 and performance P3, the performance (e.g., reasonableness, fidelity, and accuracy) of the first and second deep learning models can be obtained.

[0053] The target objects in this embodiment include deep learning models and model interpretation methods. In actual processing, the performance of deep learning models and model interpretation methods can be evaluated separately by controlling variables, thereby improving the accuracy of the evaluation.

[0054] Figure 3A This is a schematic flowchart of a method for evaluating the performance of an object according to another embodiment of the present disclosure. Figure 3B This is a schematic diagram of a method for evaluating the performance of an object according to another embodiment of the present disclosure.

[0055] like Figure 3A As shown below, the method for evaluating the loyalty of a target object will be described in detail. The method 300 for evaluating the performance of an object in this embodiment may include operations S310 and S320, and may also include operations S331, S332 and S341. Operations S310 and S320 can be referred to as operations S210 and S220 above, respectively, and will not be repeated here.

[0056] In operation S310, the original text data is processed using the target object to be evaluated to obtain the importance index of each character data in the original text data.

[0057] In operation S320, the original evidence text in the original text data is determined based on the importance index of each character data in the original text data.

[0058] In operation S331, the original text data is subjected to predetermined processing to obtain at least one processed text data.

[0059] For example, the pre-processing may include at least one of a first pre-processing, a second pre-processing, and a third pre-processing.

[0060] For example, the first pre-defined processing involves performing at least one of the following operations on the first category of character data in the original text data: adding, deleting, and modifying. The first category of character data can be data with an importance index below a threshold. For example, if the original text data is "a particularly professional photography shop with good service", the processed text data obtained after the first pre-defined processing is "a particularly professional barbecue shop with good service" or "a particularly professional shop with good service".

[0061] For example, the second pre-processing step involves synonym replacement of second-category character data in the original text data. Second-category character data can be data whose importance index is higher than a threshold. For instance, a thesaurus can be pre-configured, allowing the search for synonyms corresponding to the second-category character data, and then using these synonyms to replace the second-category character data. For example, if the original text data is "a photography shop I particularly like, with good service," the processed text data after the second pre-processing step would be "a photography shop I'm particularly satisfied with, with good service."

[0062] For example, the third pre-processing step involves modifying the sentence structure of the original text data. For instance, a sentence template library can be pre-configured, and then the original text data can be analyzed to obtain a first sentence template. Next, a second sentence template corresponding to the first sentence template can be found in the library, and then the second sentence template can replace the first sentence template. For example, if the original text data is "A particularly professional photography shop with good service," the processed text data after the third pre-processing step would be "The photography shop is very professional and has good service."

[0063] As can be seen, the above-mentioned pre-processing can produce processed text data that is quite similar to the original text data. Then, after determining the reference evidence text using the processed text data, the performance of the target object can be evaluated based on the original evidence text against the reference evidence text.

[0064] In operation S332, the target character data in the processed text data is determined as reference evidence text.

[0065] In one example, the above operation S332 can be implemented by annotating the processed text data to obtain reference evidence text. The annotation method can be referred to above, and will not be repeated in this embodiment.

[0066] In another example, operation S332 can be implemented as follows: The target object is used to process the processed text data to obtain the importance index of each character data in the processed text data. Then, based on the importance index of each character data in the processed text data, the target character data in the processed text data is determined. For example, a method similar to operations S310 to S320 can be used to process the processed text data, and then the multiple character data in the processed text data can be sorted from high to low according to their importance index to obtain a sequence. Then, a predetermined number of character data or a predetermined proportion of character data in the sequence are determined as reference evidence text. This example uses the same target object to process both the original and processed text data. Because the processing method is the same, comparing the original evidence text and the reference evidence text obtained after processing the target object can effectively evaluate whether the model interpretation method fits the internal processing mechanism of the deep learning model.

[0067] Based on operations S331-S332 above, since the processed text data obtained through the predetermined processing is quite similar to the original text data, after processing the processed text data using the target object to obtain reference evidence text, if the consistency between the reference evidence text and the original evidence text is high, it indicates that the evidence determined by the model interpretation method conforms to the processing mechanism of the deep learning model, that is, the loyalty of the target object is high. It can be seen that this example evaluates loyalty based on the consistency between the reference evidence text and the original evidence text, proposing a new index for evaluating the loyalty of the target object.

[0068] In operation S341, the average accuracy across all classes is determined based on the original evidence text and the reference evidence text, and then the first performance metric is determined based on the average accuracy across all classes.

[0069] For example, the first performance metric is the loyalty mentioned above, which represents the consistency between the first importance metric and the second importance metric. The first importance metric is the importance metric of each character data in the original text data obtained based on the interpretation method, while the second importance metric is the actual importance metric in the process of deep learning model processing the original text data.

[0070] For example, the overall average accuracy can be calculated using the following formula:

[0071]

[0072] Where MAP represents the average accuracy across all classes, X O X represents the list of tokens corresponding to the original text data. P Represents the list of tokens corresponding to the processed text data, |X P| represents X P The number of tokens in the data. Including X O The top i tokens with higher importance in the list are selected, and each token corresponds to a character data. G(x, Y) is used to determine whether token x belongs to list Y, where if x∈Y, then G(x, Y)=1.

[0073] like Figure 3B As shown, in practical applications, the performance of different target objects can be accurately evaluated by controlling variables.

[0074] For example, the first target object includes a first deep learning model and a first model interpretation method; the second target object includes a first deep learning model and a second model interpretation method; and the third target object includes a second deep learning model and a second model interpretation method. Method 300 can be executed for the first target object, the second target object, and the third target object respectively, thereby obtaining the performance corresponding to the first target object, the performance corresponding to the first target object, and the performance corresponding to the first target object.

[0075] For example, the original text data 301 can be processed using a first target object to obtain the original evidence text 3031. Then, the original text data 301 can undergo predetermined processing to obtain the processed text 302. This processed text 302 can then be processed using the first target object to obtain the reference evidence text 3041. The original evidence text 3031 and the reference evidence text 3041 are then used to determine the index value 3051 (i.e., MAP_1 in the figure). Similarly, the original text data 301 can be processed using a second target object to obtain the original evidence text 3032. This processed text 302 can then be processed using the second target object to obtain the reference evidence text 3042. The original evidence text 3032 and the reference evidence text 3042 are then used to determine the index value 3052 (i.e., MAP_2 in the figure). Similarly, the original text data 301 can be processed using the third target object to obtain the original evidence text 3033. The processed text 302 can be processed using the third target object to obtain the reference evidence text 3043. The index value 3053 (i.e., MAP_3 in the figure) can be determined using the original evidence text 3033 and the reference evidence text 3043.

[0076] Next, the fidelity of the first model interpretation method and the second model interpretation method can be determined by comparing index value 3051 and index value 3052, and the fidelity of the first deep learning model and the second deep learning model can be determined by comparing index value 3052 and index value 3053.

[0077] It should be noted that, Figure 3BTaking a single raw text dataset as an example, in actual evaluation, a dataset consisting of multiple raw text datasets can be used for evaluation. During processing, actions can be performed on each raw text data point within the dataset. Figure 3B The process shown yields the MAP value corresponding to a single original text data point. The average MAP value of each original text data point in the dataset can be used as the mean MAP value corresponding to the target object. For two target objects, each includes a deep learning model and a model interpretation method, and the deep learning model or the model interpretation method is the same in both target objects. Therefore, by comparing the mean MAP values ​​of the two target objects, the loyalty relationship between the two target objects can be determined.

[0078] Figure 4A This is a schematic flowchart of a method for evaluating the performance of an object according to another embodiment of the present disclosure. Figure 4B This is a schematic diagram of a method for evaluating the performance of an object according to another embodiment of the present disclosure.

[0079] like Figure 4A As shown below, the method for evaluating the rationality of the target object will be described in detail. The method 400 for evaluating the performance of the object in this embodiment may include operations S410 to S430, and also includes operation S442. Operations S410 to S420 can be referred to as operations S210 to S220 above, and will not be repeated here.

[0080] In operation S410, the original text data is processed using the target object to be evaluated to obtain the importance index of each character data in the original text data.

[0081] In operation S420, the original evidence text in the original text data is determined based on the importance index of each character data in the original text data.

[0082] In operation S430, the reference evidence text corresponding to the original text data is determined.

[0083] For example, you can refer to operation S230 above.

[0084] For example, the original text data can be pre-annotated at the word level. Taking the original text data as "a particularly professional photography shop with good service," when performing sentiment analysis, it is necessary to determine whether the sentiment of the original text is positive or negative. This requires pre-annotating the words used to determine the sentiment. For example, the following two reference evidence texts can be annotated for this original text: "professional shop" and "good service."

[0085] It should be noted that the labeled reference evidence text can satisfy sufficiency, completeness and conciseness. Sufficiency means that the evidence itself can fully support the prediction, for example, "specialty store" can determine that the sentiment is positive. Completeness means that the text outside the evidence cannot support the prediction. For example, after deleting the reference evidence text from the original text data, the remaining text is "special photography, service", and the positive or negative direction of the sentiment cannot be determined based on the remaining text. Conciseness means that all characters in the evidence are necessary to support the prediction, for example, deleting any character in the reference evidence text will lead to the inability to predict the sentiment.

[0086] In operation S442, the accuracy and recall rate are determined according to the original evidence text and the reference evidence text, and then the second performance indicator in the performance is determined according to the accuracy and recall rate.

[0087] For example, the second performance indicator is the rationality mentioned above.

[0088] For example, the following case is taken as an example for description: the original text data is "a very professional photography store, with good service attitude", the target object processes the original text data to obtain the original evidence text "photography store", and the reference evidence texts are "professional store" and "good attitude".

[0089] The F1 value of "photography store" and "professional store" can be calculated, the F1 value of "photography store" and "good attitude" can be calculated, the F1 value of "photography store" and "professional store with good attitude" can be calculated, and then the F1 value with the highest score is taken as the target F1 value corresponding to the target object. The target F1 value can evaluate the degree of fit between the original evidence text and the reference evidence text, so as to determine whether the evidence is reasonable.

[0090] The above F1 value can be calculated through the accuracy and recall rate, where the accuracy can represent the ratio between the number of target characters and the number of characters in the original text data, the recall rate can represent the ratio between the number of target characters and the number of characters in the reference evidence text, and the target characters can be the intersection of the original text data and the reference evidence text. Take calculating the F1 value of "photography store" and "professional store" as an example. The target character is "store", the percentage of "store" in "photography store" is 0.25, so the accuracy is 0.25. The "store" in "professional store" appears in "photography store", the percentage of "store" in "professional store" is 0.333, so the recall rate is 0.333.

[0091] As Figure 4B shown, in practical applications, the performance of different target objects can be accurately evaluated by means of control variables.

[0092] For example, the first target object includes a first deep learning model and a first model interpretation method; the second target object includes a first deep learning model and a second model interpretation method; and the third target object includes a second deep learning model and a second model interpretation method. Method 400 can be executed for the first target object, the second target object, and the third target object respectively, thereby obtaining the performance corresponding to the first target object, the performance corresponding to the first target object, and the performance corresponding to the first target object.

[0093] For example, the original text data 401 can be processed using the first target object to obtain the original evidence text 4021. The original text data 401 can be annotated to obtain the reference evidence text 403 within the original text data 401. Then, the indicator value 4041 (i.e., F1_1 in the figure) can be determined using the original evidence text 4021 and the reference evidence text 403. Similarly, the original text data 401 can be processed using the second target object to obtain the original evidence text 4022. The indicator value 4042 (i.e., F1_2 in the figure) can be determined using the original evidence text 4022 and the reference evidence text 403. Similarly, the original text data 401 can be processed using the third target object to obtain the original evidence text 4023. The indicator value 4043 (i.e., F1_3 in the figure) can be determined using the original evidence text 4023 and the reference evidence text 403.

[0094] Next, the rationality of the first model interpretation method and the second model interpretation method can be determined by comparing index values ​​4041 and 4042, and the rationality of the first deep learning model and the second deep learning model can be determined by comparing index values ​​4042 and 4043.

[0095] It should be noted that, Figure 4B Taking a single raw text dataset as an example, in actual evaluation, a dataset consisting of multiple raw text datasets can be used for evaluation. During processing, actions can be performed on each raw text data point within the dataset. Figure 4B The process shown yields the F1 score corresponding to a single original text data point. The average of the F1 scores for each original text data point in the dataset can be used as the average F1 score for the target object. For two target objects, each includes a deep learning model and a model interpretation method, and both target objects have the same deep learning model or the same model interpretation method. Therefore, by comparing the average F1 scores of the two target objects, the relative reasonableness of the two target objects can be determined.

[0096] Figure 5 This is a schematic structural block diagram of an apparatus for evaluating the performance of an object according to an embodiment of the present disclosure.

[0097] like Figure 5As shown, the apparatus 500 for evaluating the performance of an object may include: an importance determination module 510, an original evidence determination module 520, a reference evidence determination module 530, and a performance determination module 540.

[0098] The importance determination module 510 is used to process the original text data using the target object to be evaluated, and obtain the importance index of each character data in the original text data.

[0099] The original evidence determination module 520 is used to determine the original evidence text in the original text data based on the importance index of each character data in the original text data.

[0100] The reference evidence determination module 530 is used to determine the reference evidence text corresponding to the original text data.

[0101] The performance determination module 540 is used to determine the performance metrics of the target object based on the original evidence text and the reference evidence text, in order to evaluate the performance of the target object.

[0102] According to another embodiment of this disclosure, the reference evidence determination module includes: an acquisition submodule and a first determination submodule. The acquisition submodule is used to acquire processed text data; wherein the processed text data is obtained by performing predetermined processing on the original text data; and the first determination submodule is used to determine target character data in the processed text data as reference evidence text.

[0103] According to another embodiment of this disclosure, the predetermined processing includes at least one of the following: performing at least one of the following processing on the first category of character data in the original text data: adding, deleting, and modifying; performing synonym replacement on the second category of character data in the original text data; wherein the importance index of the second category of character data is higher than the importance index of the first category of character data; and modifying the sentence structure of the original text data.

[0104] According to another embodiment of this disclosure, the first determining submodule includes a processing unit and a determining unit. The processing unit is used to process the processed text data using the target object to obtain the importance index of each of the multiple character data in the processed text data; the determining unit is used to determine the target character data in the processed text data based on the importance index of each of the multiple character data in the processed text data.

[0105] According to another embodiment of this disclosure, the performance determination module includes a second determination submodule and a third determination submodule. The second determination submodule is used to determine the average accuracy across all classes based on the original evidence text and the reference evidence text. The third determination submodule is used to determine a first performance index in the performance based on the average accuracy across all classes. The target objects include a deep learning model and a model interpretation method. The first performance index characterizes the consistency between a first importance index and a second importance index. The first importance index is the importance index of each character data in the original text data obtained by the interpretation device. The second importance index is the actual importance index in the process of the deep learning model processing the original text data.

[0106] According to another embodiment of this disclosure, the performance determination module includes a fourth determination submodule and a fifth determination submodule. The fourth determination submodule is used to determine precision and recall based on the original evidence text and the reference evidence text; the fifth determination submodule is used to determine a second performance indicator in the performance based on precision and recall.

[0107] According to another embodiment of this disclosure, the target object includes a deep learning model and a model interpretation method; the importance determination module includes a first processing submodule and a second processing submodule. The first processing submodule is used to process the original text data using the deep learning model to obtain intermediate results and output results; the output results are determined based on the intermediate results; the second processing submodule is used to process the intermediate results using the model interpretation method to obtain the importance index of each character data in the original text data, and the importance index characterizes the degree of importance of the character data to the output results.

[0108] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0109] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0110] According to embodiments of this disclosure, this disclosure also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method for evaluating the performance of an object.

[0111] According to embodiments of this disclosure, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described method for evaluating the performance of an object.

[0112] According to embodiments of this disclosure, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for evaluating the performance of an object.

[0113] Figure 6 This is a structural block diagram of an electronic device used to implement the method for evaluating the performance of an object according to embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0114] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0115] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0116] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as methods for evaluating object performance. For example, in some embodiments, the method for evaluating object performance may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the method for evaluating object performance described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the method for evaluating object performance by any other suitable means (e.g., by means of firmware).

[0117] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0119] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0121] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0122] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0123] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0124] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for evaluating the performance of an object, wherein the target object to be evaluated includes a deep learning model and a model interpretation method; the method includes: The method of processing raw text data using the deep learning model and the model interpretation method to obtain importance indices for each character data in the raw text data includes: processing the raw text data using the deep learning model to obtain intermediate results for determining the output result; and processing the intermediate results using the model interpretation method to obtain importance indices for each character data in the raw text data, wherein the importance indices characterize the degree of importance of the character data to the output result. Based on the importance index of each character data in the original text data, the original evidence text in the original text data is determined; Determining the reference evidence text corresponding to the original text data includes: performing predetermined processing on the original text data to obtain processed text data; processing the processed text data using the deep learning model and the model interpretation method to obtain the importance index of each of the multiple character data in the processed text data; determining the target character data in the processed text data based on the importance index of each of the multiple character data in the processed text data, and using it as the reference evidence text; and Based on the original evidence text and the reference evidence text, the performance metrics of the target object are determined, including the overall average precision, accuracy, or recall rate.

2. The method according to claim 1, wherein, The predetermined process includes at least one of the following: Perform at least one of the following processes on the first category of character data in the original text data: add, delete, and modify; Synonym replacement is performed on the second category of character data in the original text data; wherein the importance index of the second category of character data is higher than the importance index of the first category of character data; and Modify the sentence structure of the original text data.

3. The method according to any one of claims 1 to 2, wherein, The step of determining the performance indicators of the target object based on the original evidence text and the reference evidence text includes: Based on the original evidence text and the reference evidence text, determine the overall average accuracy rate; and Based on the overall average accuracy, determine the first performance metric in the performance metrics; The target object includes a deep learning model and a model interpretation method. The first performance metric represents the consistency between a first importance metric and a second importance metric. The first importance metric is the importance metric of each character data in the original text data obtained based on the model interpretation method. The second importance metric is the actual importance metric in the process of the deep learning model processing the original text data.

4. The method according to claim 1, wherein, The reference evidence text is obtained through annotation, and determining the performance indicators of the target object based on the original evidence text and the reference evidence text includes: Based on the original evidence text and the reference evidence text, determine the precision and the recall; and A second performance metric is determined based on the accuracy and the recall rate.

5. An apparatus for evaluating the performance of an object, wherein the target object to be evaluated includes a deep learning model and a model interpretation method; the apparatus comprises: An importance determination module is used to process the original text data using the deep learning model and the model interpretation method to obtain importance indices for each of the multiple character data in the original text data. This includes: processing the original text data using the deep learning model to obtain intermediate results for determining the output result; and processing the intermediate results using the model interpretation method to obtain importance indices for each of the multiple character data in the original text data, wherein the importance indices characterize the degree of importance of the character data to the output result. The original evidence determination module is used to determine the original evidence text in the original text data based on the importance index of each of the multiple character data in the original text data; A reference evidence determination module is used to determine reference evidence text corresponding to the original text data, including: performing predetermined processing on the original text data to obtain processed text data; processing the processed text data using the deep learning model and the model interpretation method to obtain importance indices for each of multiple character data in the processed text data; determining target character data in the processed text data based on the importance indices of each of the multiple character data in the processed text data, and using it as the reference evidence text; and The performance determination module is used to determine the performance indicators of the target object based on the original evidence text and the reference evidence text. The performance indicators include the overall average correctness, accuracy, or recall rate.

6. The apparatus according to claim 5, wherein, The predetermined process includes at least one of the following: Perform at least one of the following processes on the first category of character data in the original text data: add, delete, and modify; Synonym replacement is performed on the second category of character data in the original text data; wherein the importance index of the second category of character data is higher than the importance index of the first category of character data; and Modify the sentence structure of the original text data.

7. The apparatus according to any one of claims 5 to 6, wherein, The performance determination module includes: The second determining submodule is used to determine the overall average accuracy rate based on the original evidence text and the reference evidence text; and The third determining submodule is used to determine the first performance index in the performance based on the average accuracy of the entire class. The target object includes a deep learning model and a model interpretation method. The first performance metric represents the consistency between a first importance metric and a second importance metric. The first importance metric is the importance metric of each character data in the original text data obtained based on the model interpretation method. The second importance metric is the actual importance metric in the process of the deep learning model processing the original text data.

8. The apparatus according to claim 5, wherein, The reference evidence text is obtained through annotation, and the performance determination module includes: The fourth determining submodule is used to determine the precision and the recall rate based on the original evidence text and the reference evidence text; and The fifth determining submodule is used to determine the second performance indicator in the performance based on the accuracy and the recall.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 4.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 4.

Images