Evaluation method and device of element extraction model, electronic equipment and storage medium

Abstract

The invention provides an evaluation method and device of an element extraction model and electronic equipment. The method comprises the steps of obtaining a target text slice in a target call text; extracting target element information corresponding to a preset element from the target text slice through an element extraction model; according to an element extraction result of the historical text slice, correcting the target element information to obtain an element extraction result of the target text slice; wherein the historical text slices are processed text slices; evaluating the element extraction result of the target text slice from the at least two evaluation dimensions to obtain at least two dimension evaluation results; and determining a model evaluation result of the element extraction model according to the at least two dimension evaluation results. According to the mode, the element extraction result can be evaluated from at least two evaluation dimensions, so that the model evaluation result is comprehensively determined according to the evaluation results of different dimensions, and the accuracy and reliability of the model evaluation result are ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to an evaluation method, device, electronic device and storage medium for a factor extraction model. Background Art

[0002] Elements refer to the necessary factors or components that make up things. Elements are indispensable factors that make up things, and they are also the basic units that make up the system. They are the driving force behind the generation, change, and development of the system. In different fields, the specific meaning of "elements" may be different. For example, in mechanics, "elements" can refer to basic geometric elements such as points, lines, and surfaces that make up the geometric bodies of mechanical parts; in economics, production factors refer to various social resources required for social production and operation activities, such as labor, land, capital, entrepreneurial talent, etc. In the field of information processing, elements mainly refer to data elements, which can specifically refer to data resources that exist in electronic form, participate in production and operation activities through computing, and play an important role.

[0003] In the field of information processing, the development of big data has led to an increasing number of data elements in terms of types and quantity. Accordingly, various factor extraction models have emerged. Therefore, how to accurately and efficiently evaluate the factor extraction model has become a problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a method and device for evaluating a factor extraction model, an electronic device, and a computer-readable storage medium.

[0005] In a first aspect, the present disclosure provides a method for evaluating a factor extraction model, comprising:

[0006] Obtaining a target text slice in a target call text; wherein, by means of a preset sliding window, the target call text is sequentially divided into a plurality of text slices, and the target text slice is the text slice currently being processed;

[0007] Extracting target element information corresponding to preset elements from the target text slice through the element extraction model;

[0008] According to the element extraction result of the historical text slice, the target element information is corrected to obtain the element extraction result of the target text slice; wherein the historical text slice is a processed text slice;

[0009] Evaluate the element extraction results of the target text slice from at least two evaluation dimensions to obtain at least two dimensional evaluation results;

[0010] According to the evaluation results of the at least two dimensions, a model evaluation result of the factor extraction model is determined.

[0011] In a second aspect, the present disclosure provides an evaluation device for a factor extraction model, comprising:

[0012] A slice acquisition module, adapted to acquire a target text slice in a target call text; wherein the target call text is sequentially divided into a plurality of text slices by a preset sliding window, and the target text slice is a text slice currently being processed;

[0013] An element extraction module, adapted to extract target element information corresponding to a preset element from the target text slice through the element extraction model;

[0014] A correction module, adapted to correct the target element information according to the element extraction result of the historical text slice to obtain the element extraction result of the target text slice; wherein the historical text slice is a processed text slice;

[0015] A dimension evaluation module, adapted to evaluate the element extraction result of the target text slice from at least two evaluation dimensions to obtain at least two dimensional evaluation results;

[0016] The model evaluation module is adapted to determine the model evaluation result of the factor extraction model according to the evaluation results of the at least two dimensions.

[0017] In a third aspect, the present disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can execute the above-mentioned method for evaluating the factor extraction model.

[0018] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned method for evaluating the factor extraction model when executed by a processor.

[0019] In a fifth aspect, the present disclosure provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned method for evaluating the factor extraction model.

[0020] In the evaluation method of the feature extraction model provided in the embodiment of the present disclosure, on the one hand, the target feature information extracted in the target text slice can be corrected according to the feature extraction result of the historical text slice, so as to obtain more comprehensive information content in combination with the historical text slice, thereby improving the accuracy of the feature extraction result and avoiding extraction errors caused by the limited length of the target text slice. On the other hand, the feature extraction result can be evaluated from at least two evaluation dimensions, so as to comprehensively determine the model evaluation result according to the evaluation results of different dimensions, so as to ensure the accuracy and reliability of the model evaluation result.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:

[0023] Figure 1 A flowchart of a method for evaluating a factor extraction model provided by an embodiment of the present disclosure;

[0024] Figure 2 A flowchart of an evaluation method of a factor extraction model provided for an example of the present disclosure;

[0025] Figure 3 A block diagram of an evaluation device for a factor extraction model provided in an embodiment of the present disclosure;

[0026] Figure 4 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0028] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0029] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0030] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0031] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0032] In the technical solution disclosed in this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution complies with relevant national laws and regulations (for example, the "Information Security Technology Personal Information Security Specification", etc.). For example: corresponding prescribed measures are taken for personal information access control; prescribed restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; when using personal information, clear identity orientation is eliminated to avoid precise positioning of specific individuals.

[0033] In the field of information processing, the development of big data has led to an increasing number of data elements in terms of types and quantity. Accordingly, various factor extraction models have emerged, making it difficult to accurately and efficiently evaluate the performance of factor extraction models. In order to solve the above problem, in this application, the factor extraction results can be evaluated from at least two evaluation dimensions, so that the model evaluation results can be comprehensively determined based on the evaluation results of different dimensions to ensure the accuracy and reliability of the model evaluation results.

[0034] The evaluation method of the factor extraction model according to the embodiment of the present disclosure can be performed by an electronic device such as a terminal device or a server, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a cloud server capable of cloud computing. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory.

[0035] Figure 1 A flowchart of an evaluation method for a factor extraction model provided by an embodiment of the present disclosure. Figure 1 , the method comprising:

[0036] Step S110: Obtain a target text slice in the target call text; wherein, the target call text is divided into a plurality of text slices in sequence through a preset sliding window, and the target text slice is the text slice currently being processed.

[0037] The target call text may be a translated text of the call recording content to be processed (i.e., the elements to be extracted). The text slices may be obtained by dividing the target call text according to a preset sliding window and a preset sliding step. By dividing the target call text into multiple text slices for processing, the amount of data processed at a single time can be reduced and the processing speed can be improved.

[0038] Correspondingly, when the target call text is a real-time data stream, the target text slice can be: a partial text segment corresponding to the window length of the current sliding window in the real-time data stream. Among them, the real-time data stream is a technology that connects the data source with the data analysis algorithm so that the data can be analyzed during the generation process. The target call text in this application can be a real-time dynamically generated data stream, or it can be an archived call log, call archive file, etc., and this application does not limit this.

[0039] Among them, the sliding window is an algorithm whose basic idea is to maintain a window of a preset length, which slides on the data sequence (i.e., real-time data stream) in a preset order (such as from left to right or from early to late data timestamps), and the distance of each slide is called the sliding step. Among them, the window length and sliding step of the sliding window can be flexibly set according to actual needs, and this application does not limit this.

[0040] In some possible implementations, the sliding window can be slid in the target call text in the order of data timestamps from early to late, and the text segment intercepted by the sliding window each time is used as the text slice corresponding to the current sliding window. In a specific example, the window length of the sliding window can be determined according to the call round, for example, each sliding window is fixed to correspond to the time length of N call rounds; wherein N is a natural number greater than 1. N can specifically be a value such as 2, 3, 4, etc. For example, the time length of each sliding window can be set to correspond to 3 call rounds, that is, the real-time text segment corresponding to the current sliding window contains the conversation content of 3 call rounds. Among them, the call round refers to: the number of conversation rounds included in the call. Usually, the conversation content completed once by each party to the call is regarded as a call round. For example, a question and answer can be regarded as a call round.

[0041] In an optional implementation, the target text slice corresponding to the current sliding window can be obtained in the following manner: dynamically monitor channel switching events; dynamically determine the number of call turns from the start time of the current sliding window to the current time based on the monitored channel switching events; when the number of call turns reaches N, determine the current time as the end time of the current sliding window, and obtain the text segment corresponding to the current sliding window.

[0042] In addition, the sliding window may also be a window of fixed text length, such as a window of fixed number of text lines. This application does not limit the specific implementation details.

[0043] Step S120: extracting target element information corresponding to preset elements from the target text slice through the element extraction model.

[0044] The target text slice is input into the feature extraction model to extract the target feature information corresponding to the preset feature contained in the target text slice. The number of preset features can be one or more. The target feature information is used to characterize the specific information of the preset feature, such as the feature name and feature content.

[0045] Step S130: According to the element extraction result of the historical text slice, the target element information is corrected to obtain the element extraction result of the target text slice; wherein the historical text slice is a processed text slice.

[0046] The historical text slice refers to the text slice that has been processed (i.e., the element information has been extracted) in the target call text. Accordingly, there can be multiple historical text slices, and the specific value depends on the specific position of the currently processed target text slice in the target call text.

[0047] Typically, the feature extraction results of historical text slices may also contain associated information corresponding to preset features, for example, the specific feature value of the preset feature and relevant context information of the preset feature. Therefore, by correcting the target feature information through the feature extraction results of historical text slices, the feature extraction results of the target text slices can be more accurate.

[0048] Step S140: Evaluate the element extraction results of the target text slice from at least two evaluation dimensions to obtain at least two dimensional evaluation results.

[0049] Among them, at least two evaluation dimensions are used to evaluate the extraction performance of the factor extraction model from different perspectives. Accordingly, compared with a single evaluation result corresponding to a single dimension, the evaluation results of at least two dimensions can complement each other, thereby improving accuracy.

[0050] Step S150: Determine a model evaluation result of the factor extraction model based on the evaluation results of at least two dimensions.

[0051] Among them, preset operations, such as averaging operations, weighted operations, etc., can be performed on the evaluation results of at least two dimensions to obtain the final model evaluation results.

[0052] In the evaluation method of the feature extraction model provided in the embodiment of the present disclosure, on the one hand, the target feature information extracted in the target text slice can be corrected according to the feature extraction result of the historical text slice, so as to obtain more comprehensive information content in combination with the historical text slice, thereby improving the accuracy of the feature extraction result and avoiding extraction errors caused by the limited length of the target text slice. On the other hand, the feature extraction result can be evaluated from at least two evaluation dimensions, so as to comprehensively determine the model evaluation result according to the evaluation results of different dimensions, so as to ensure the accuracy and reliability of the model evaluation result.

[0053] In addition, those skilled in the art may also make various modifications and variations to the method in this application:

[0054] In some possible implementations, when the target element information is corrected according to the element extraction result of the historical text slice to obtain the element extraction result of the target text slice, it can be implemented in the following ways:

[0055] First, obtain the first element key-value pair contained in the target element information, and extract the first element value in the first element key-value pair. The first element key-value pair refers to: element information stored in the form of a key-value pair contained in the target element information. Specifically, the first element key-value pair may include: a first element key (key) for representing the specific name of the element, and a first element value (value) for representing the specific content of the element. For example, the first element key may be "user A's mobile phone number", and correspondingly, the first element value may be a numeric string a.

[0056] Then, obtain the second element key-value pair contained in the element extraction result of the historical text slice, and extract the second element value matching the first element value from the second element key-value pair. The second element key-value pair refers to: element information stored in the form of a key-value pair contained in the element extraction result of the historical text slice. Accordingly, in the case where there are multiple second element key-value pairs, respectively match the second element value in each second element key-value pair with the first element value mentioned above, and extract the successfully matched second element value.

[0057] Next, the second element key corresponding to the second element value is determined, wherein the second element key corresponding to the successfully matched second element value is the element key in the second element key-value pair to which the successfully matched second element value belongs.

[0058] Finally, in the case where the first element key in the first element key-value pair does not match the second element key corresponding to the second element value, the first element key is corrected according to the second element key. Since the first element value in the first element key-value pair matches the second element value in the second element key-value pair, under normal circumstances, the first element key in the first element key-value pair and the second element key in the second element key-value pair should also match. For example, assume that the first element value and the second element value are both numeric strings a, and the first element key is "user A's mobile phone number", but the second element key is "user B's mobile phone number". Therefore, it can be determined that the first element key in the first element key-value pair does not match the second element key corresponding to the second element value. At this time, the first element key can be corrected according to the second element key. For example, the first element key can be corrected from "user A's mobile phone number" to "user B's mobile phone number".

[0059] In the application scenario of the present application, in order to improve the processing speed, the length of a single text slice is limited, and when a single text slice is divided, the associated information corresponding to an element may be split. For example, the name information of an element may be divided in the previous text slice, and the specific content of the element may be divided in the current text slice, which leads to incorrect element extraction results and causes the element key and element value to not correspond. In order to solve the above problems, the element extraction results of historical text slices are corrected, and the final accurate results can be obtained by combining historical text slices, correcting the problem of misrecognition caused by limited text slice content.

[0060] In some possible implementations, when determining the second element key corresponding to the second element value, it can be implemented in the following manner: if there are multiple second element key-value pairs containing the second element value, select a target element key-value pair from the multiple second element key-value pairs according to the frequency of occurrence and / or the time of occurrence of the multiple second element key-value pairs; determine the second element key corresponding to the second element value according to the target element key-value pair. For example, still taking the above example as an example, assuming that the second element value is a numeric string a, and the number of second element key-value pairs containing the numeric string a is multiple, at this time, determine the frequency of occurrence and / or the time of occurrence of multiple second element key-value pairs containing the second element value, and select the second element key-value pair with the highest frequency of occurrence and / or the latest time of occurrence as the target element key-value pair, so as to determine the second element key corresponding to the second element value according to the target element key-value pair. For example, among the multiple second element key-value pairs containing the numeric string a, the element key "user B's mobile phone number" appears the most times, and the element key "user D's identity identification code" appears less times, then the second element key-value pair corresponding to "user B's mobile phone number" is used as the target element key-value pair. By using the frequency and / or time of occurrence, occasional interference information can be filtered out to improve the accuracy of the final result.

[0061] In some possible implementations, the first element key and / or the second element key include at least one of the following: an identity name for characterizing the user identity information of the call user, an attribute name for characterizing the object attribute of the transaction object, and a role name for characterizing the transaction role of the call user in the transaction process. Among them, the identity name for characterizing the user identity information of the call user can be determined by factors such as the user's name and occupation. The attribute name for characterizing the object attribute of the transaction object is used to characterize the amount, physical meaning, etc. of the transaction object, for example, it can be the total amount of the transaction object, the single amount, etc. The role name for characterizing the transaction role of the call user in the transaction process may include: the first role (such as the buyer) for obtaining the transaction object, the second role (such as the seller) for providing the transaction object, etc. In addition, in the field of telecommunications fraud, the role name may also include: the deceived party, the fraudster. In short, the present application does not limit the specific form of the element key, and those skilled in the art can flexibly set it according to the actual business scenario.

[0062] In some possible implementations, the element extraction results of the target text slice are evaluated from at least two evaluation dimensions, and the evaluation results of at least two dimensions include: from the slice evaluation dimension, the element extraction results of each text slice are evaluated to obtain the slice dimension evaluation results; from the call evaluation dimension, the element extraction results of the target call text are evaluated to obtain the call dimension evaluation results; from the element evaluation dimension, the element extraction results of each element are evaluated to obtain the element dimension evaluation results. In this implementation, the evaluation can be performed from the three dimensions of slice, call, and element. Among them, the evaluation granularity of the three dimensions is different: the element granularity is the smallest, which is convenient for analyzing the extraction accuracy of each element in units of elements; the slice granularity is in the middle, and the extraction accuracy of each slice can be analyzed in units of slices; the call granularity is the largest, and the extraction results of multiple slices in a call can be combined to determine the extraction results of the call, which is convenient for comprehensively measuring the accuracy of extraction from a macro perspective. It can be seen that by adopting three evaluation dimensions with different granularities, it is possible to evaluate from three different dimensions of large, medium, and small, so as to take into account the accuracy of local extraction and the accuracy of global extraction.

[0063] In some possible implementations, the dimension evaluation result of any evaluation dimension is determined according to the recall rate, the accuracy rate, the first weight coefficient for characterizing the importance of the recall rate, and the second weight coefficient for characterizing the importance of the accuracy rate. Among them, the recall rate is an indicator for evaluating the performance of the model, especially in binary or multi-classification, and is often used to detect the ability of the model to find relevant instances. In this embodiment, the recall rate is mainly used to characterize the number and proportion of elements recognized by the model. The accuracy rate is used to evaluate the accuracy of the model. The larger the recall rate, the more comprehensive the elements recognized by the model. However, when the recall rate is too large, due to the large number of elements recognized, the time-consuming subsequent processing and extraction of these elements will also be longer. Therefore, in actual situations, the weight coefficients of the recall rate and the accuracy rate can be flexibly configured according to business needs. For example, when the target call text is a first type of call text, the first weight coefficient may be greater than the second weight coefficient; when the target call text is a second type of call text, the first weight coefficient may be less than the second weight coefficient; wherein the text review method for the first type of target call text is usually a non-manual review method, and the text review method for the second type of target call text is usually a manual review method.

[0064] Normally, in the manual review method, it is necessary to manually judge whether the factor extraction results are correct. Therefore, if the recall rate is too high, the manual review cost will be too high and the time consumption will be too high. Therefore, in the manual review method, making the first weight coefficient of the recall rate smaller than the second weight coefficient of the accuracy rate can reduce the impact of the recall rate on the model evaluation results, thereby avoiding a large number of false recalls and reducing the cost of manual verification. In the non-manual review method, there is no need to manually judge whether the factor extraction results are correct. Therefore, even if the recall rate is very high, it will not bring too high manual review costs. In addition, a higher recall rate helps to improve the accuracy of the model. Therefore, in the non-manual review method, making the first weight coefficient of the recall rate larger than the second weight coefficient of the accuracy rate helps to improve the accuracy of the model.

[0065] For ease of understanding, the technical implementation details in the above embodiment are described in detail below using a specific example:

[0066] This example is mainly used in the field of NLP natural language processing, especially for evaluating the fine-tuning effect of large models under specific tasks (such as extracting elements from multi-round conversations in scenarios such as bank customer service and anti-fraud alarms). In business scenarios such as bank customer service and anti-fraud alarms, extracting elements from multi-round conversations is a current technical difficulty. In order to improve the extraction effect, it is necessary to continuously iterate and optimize the model. However, how to evaluate the effect of the fine-tuned model is an important test in this process. Existing technologies often lack a systematic and comprehensive evaluation process and method. Therefore, this example designs a process and method for evaluating the effect of the element extraction model to solve this technical problem.

[0067] As can be seen, this example aims to solve the following problems:

[0068] (1) Lack of a unified evaluation process: The relevant technologies lack a unified and standard evaluation process, which makes it difficult to judge the consistency and comparability of evaluation results in different studies or applications.

[0069] (2) Single evaluation method: Traditional evaluation methods mainly rely on single performance indicators such as accuracy and recall, and fail to fully consider the comprehensive performance and timeliness of the model in actual application scenarios.

[0070] (3) Ignoring the characteristics of multi-round dialogues: Existing evaluation methods fail to fully consider the complexity and real-time nature of multi-round dialogues, resulting in the evaluation results being unable to accurately reflect the actual performance of the model in actual business scenarios.

[0071] Figure 2 FIG. 4 is a flowchart showing the evaluation method of the factor extraction model provided in this example. Figure 2 As shown, this example specifically includes the following steps:

[0072] Step S201: configuring extraction rules for elements to be extracted.

[0073] Specifically, configure the element description elements_text (including the element name and explanation) to be extracted, and also configure the element rule elements_rule (specifying the length, candidate values, etc. that each element value should conform to).

[0074] Step S202: Obtain a target text slice in the target call text, and extract target element information corresponding to the preset element from the target text slice through the element extraction model.

[0075] This step aims to achieve slice extraction. For the existing ASR translation text of multiple conversations, for each conversation, according to a certain text window w and step size s (for example: the window size is 10 texts, and 2 texts are slid down each time), the number of dialogues selected each time is determined, and the prompt words are formed with elements_text. The large model is used for feature extraction to obtain the slice extraction result predict_cut_i, for example:

[0076]

[0077] The slice extraction result predict_cut_i is the text slice mentioned above. The value of i corresponding to the current text slice (ie, the target text slice) changes dynamically.

[0078] It can be seen that this step can use the pre-configured element rule elements_rule to extract the target element information corresponding to the preset element from the target text slice.

[0079] Step S203: verify the extracted target element information.

[0080] This step aims to implement feature value verification: for the extraction results of each slice, the extracted value of each feature is verified according to the feature value rules. For example, according to the feature value rules, the bank card number should meet 16 to 19 digits, and the ID card number should meet 18 digits. Therefore, if the feature value of the ID card number "1**10919****6016" does not meet 18 digits, the extraction result of the ID card number needs to be set to blank to obtain the corrected result of the slice:

[0081]

[0082] Step S204: According to the element extraction result of the historical text slice, the verified target element information is corrected to obtain the element extraction result of the target text slice.

[0083] This step aims to achieve the modification of element names: for elements involving roles, such as: victim transfer method, fraudster transfer method, victim bank card number, fraudster bank card number, victim account opening bank, fraudster account opening bank, etc., it is necessary to use the existing extraction results predict_cut_i (i = 1), predict_cut_i (i = 2) ... predict_cut_i (i = n) to perform element pair statistics temp_statistics, for example: {"victim transfer method": "xx application"} appears 2 times, {"victim bank {"Card Number":"62****000009319****"} appears once, and {"Fraudster Bank Card Number":"62****000009319****"} appears three times. Then, the roles with the same element value in predict_cut_i(i=n+1) are corrected. For example, {"Victim Bank Card Number":"62****000009319****"} extracted when i=n+1 should be corrected to {"Fraudster Bank Card Number":62****000009319****"}.

[0084] Among them, {"victim's transfer method":"xx application"} is a set of key-value pairs mentioned above, "victim's transfer method" is the key, and "xx application" is the value. Similarly, {"scammer's bank card number":"62****000009319****"} is a set of key-value pairs, "scammer's bank card number" is the key, and "62****000009319****" is the value.

[0085] Through correction processing, it is possible to combine historical situations and determine that the element key-value combination that appears many times is the correct key-value combination, thereby avoiding the problem of misextraction caused by incorrect combinations of element keys and element values.

[0086] Step S205: Evaluate the element extraction result of each text slice from the slice evaluation dimension to obtain a slice dimension evaluation result.

[0087] Specifically, this step aims to calculate the slice granularity index: after processing all the slice results of each call according to the above steps, compare each slice with the extraction result predict_cut_i and the label label_cut_i of the slice, and calculate the evaluation index score_cut_i of each slice. Among them, score_cut_i=(w*R+(1-w)*P) / T, where R is the recall rate, P is the accuracy rate, T is the response time, and w is the weight of the recall rate (i.e. the preference coefficient of the business personnel), and further calculate the average score score_cut_avg of the slice granularity of all phone data (at this time, the average score of a single call is not distinguished, only the overall average score of all slices is considered).

[0088] Step S206: Evaluate the element extraction result of the target call text from the call evaluation dimension to obtain a call dimension evaluation result.

[0089] Specifically, this step aims to achieve the summary of slicing results: summarize all slicing results (after processing) of each call respectively, deduplicate according to the key-value pair granularity, obtain the extraction result predict_case_i of the case granularity of each call, and obtain the annotation label label_case_i of the case granularity.

[0090] Next, calculate the case granularity index: for each call, compare the extraction result predict_case_i with the annotation label label_case_i, calculate the evaluation index score_case_i of each call, and score_case_i = w*R+(1-w)*P, where R is the recall rate, P is the accuracy rate, and w is the weight of the recall rate (that is, the business personnel preference coefficient), and further calculate the case granularity average score score_case_avg of all telephone data.

[0091] Step S207: Evaluate the factor extraction result of each factor from the factor evaluation dimension to obtain the factor dimension evaluation result.

[0092] Specifically, this step aims to calculate the element granularity index: for each call, the extraction result predict_case_i is compared with the annotation label label_case_i, the extraction accuracy of each element element_score_i is calculated, and the element accuracy element_score_avg of all phone data is further calculated, for example: [

[0094] {'victim's bank card number':0.35},

[0095] {'victim's name':0.25},

[0096] {'Victim's bank account': 0.16},

[0097] {'victim's account name':0.15},

[0098] {'Scammer's payment method':0.05},

[0099] {'Victim's transfer method': 0.03},

[0100] {'The fraudster opened an account at the bank': 0.02}

[0101] ].

[0102] Based on the performance of each element in element_score_avg, data optimization can be performed on elements with too low scores.

[0103] Step S208: performing preset calculations on the evaluation results of multiple dimensions to obtain a model evaluation result of the factor extraction model.

[0104] Among them, indicators can be recorded and compared: slice granularity indicators, case granularity indicators, and element granularity indicators can be recorded, and compared according to different versions to determine the model effect and its changing trend.

[0105] The evaluation method of the feature extraction model in this example can be implemented based on the evaluation system. The functional block diagram of the evaluation system may include the following modules:

[0106] Configuration management module: responsible for managing the element description (elements_text) and element rules (elements_rule) to be extracted.

[0107] Slice extraction module: Receives ASR-translated texts from multiple conversations, slices them according to the set text window size and step size, and uses a large model to extract features from each text segment to obtain slice extraction results.

[0108] Feature verification and correction module: Verify the feature value of each slice extraction result to ensure compliance with feature rules, and correct the feature values ​​of the involved roles to ensure the accuracy and consistency of the feature values.

[0109] Evaluation index calculation module: calculate slice granularity indicators, including recall rate, accuracy rate and response time, and obtain the average score of slice granularity; calculate case granularity indicators, conduct a comprehensive evaluation on each call, and obtain the average score of case granularity; calculate element granularity indicators, evaluate the extraction accuracy of each element, and obtain the average element accuracy rate.

[0110] This example can at least achieve the following technical effects: (1) Improve extraction accuracy: The element name correction mechanism reduces errors caused by role confusion and enhances the accuracy of the extraction results. (2) Improve the comprehensiveness of model evaluation: The present invention introduces evaluation indicators in three dimensions: slice granularity, case granularity, and element granularity, which can more comprehensively evaluate the performance of the model at different granularity levels; by comparing indicators of different versions, the changing trend of the model effect can be clearly observed, providing strong support for model tuning. (3) Enhance business adaptability: Configure element descriptions and element rules according to actual needs to enhance the flexibility and customizability of the system. (4) Promote continuous optimization of the model: Based on the analysis results of the element granularity indicators, the present invention can optimize data for elements with too low scores, thereby improving the overall performance of the model; through continuous model tuning and iterative upgrades, the present invention can keep pace with business needs and achieve long-term application value.

[0111] Among them, at least the following technical key points are included: (1) Element configuration and management realizes flexible configuration of extracted elements, including element names, explanations and element value rules; provides an element rule verification mechanism to ensure that the extracted element values ​​meet the preset standards. (2) Element value verification and correction: The extracted element values ​​are strictly verified to ensure the accuracy and compliance of the data; and the intelligent correction of the role-related elements is realized to improve the availability of data. (3) Multi-dimensional evaluation index calculation: Multi-dimensional evaluation indicators of slice granularity, case granularity and element granularity are designed; a comprehensive evaluation of the model effect is provided to facilitate business personnel to tune and make decisions. (4) Specific implementation and strategy of slicing extraction: The specific implementation method and strategy of slicing multi-round dialogues using text windows and step strategies. (5) Algorithm and logic of element value verification and correction: The algorithm and specific implementation logic for verifying and correcting the extracted element values. (6) Calculation method and application of multi-dimensional evaluation indicators: The calculation method of multi-dimensional evaluation indicators such as slice granularity, case granularity and element granularity and their application in model effect evaluation.

[0112] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0113] In addition, the present disclosure also provides a factor extraction device, an electronic device, and a computer-readable storage medium, all of which can be used to implement an evaluation method for any factor extraction model provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0114] Figure 3 A block diagram of an evaluation device for a factor extraction model provided in an embodiment of the present disclosure.

[0115] Reference Figure 3 The present disclosure provides an evaluation device 30 for a factor extraction model, and the evaluation device 30 for a factor extraction model includes:

[0116] The slice acquisition module 31 is adapted to acquire a target text slice in the target call text; wherein the target call text is sequentially divided into a plurality of text slices by a preset sliding window, and the target text slice is the text slice currently being processed;

[0117] An element extraction module 32, adapted to extract target element information corresponding to a preset element from the target text slice through the element extraction model;

[0118] A correction module 33, adapted to correct the target element information according to the element extraction result of the historical text slice to obtain the element extraction result of the target text slice; wherein the historical text slice is a processed text slice;

[0119] A dimension evaluation module 34, adapted to evaluate the element extraction result of the target text slice from at least two evaluation dimensions to obtain at least two dimensional evaluation results;

[0120] The model evaluation module 35 is adapted to determine the model evaluation result of the factor extraction model according to the evaluation results of the at least two dimensions.

[0121] In an optional implementation, the correction module is specifically used to:

[0122] Acquire a first element key-value pair included in the target element information, and extract a first element value in the first element key-value pair;

[0123] Obtaining a second element key-value pair contained in the element extraction result of the historical text slice, and extracting a second element value matching the first element value from the second element key-value pair;

[0124] Determine a second element key corresponding to the second element value;

[0125] In the case where the first element key in the first element key-value pair does not match the second element key corresponding to the second element value, the first element key is modified according to the second element key.

[0126] In an optional implementation, the correction module is specifically used to:

[0127] If there are multiple second element key-value pairs including the second element value, selecting a target element key-value pair from the multiple second element key-value pairs according to the occurrence frequency and / or occurrence time of the multiple second element key-value pairs;

[0128] A second element key corresponding to the second element value is determined according to the target element key-value pair.

[0129] In an optional implementation, the first element key and / or the second element key includes at least one of the following:

[0130] An identity name used to characterize the user identity information of the calling user, an attribute name used to characterize the object attribute of the transaction object, and a role name used to characterize the transaction role of the calling user in the transaction process.

[0131] In an optional implementation, the dimension evaluation module is specifically used to:

[0132] From the slice evaluation dimension, the element extraction result of each text slice is evaluated to obtain the slice dimension evaluation result;

[0133] From the call evaluation dimension, evaluating the element extraction result of the target call text to obtain a call dimension evaluation result;

[0134] From the factor evaluation dimension, the factor extraction result of each factor is evaluated to obtain the factor dimension evaluation result.

[0135] In an optional implementation, the dimension evaluation result of any evaluation dimension is determined according to the recall rate, the precision rate, a first weight coefficient for characterizing the importance of the recall rate, and a second weight coefficient for characterizing the importance of the precision rate;

[0136] Wherein, when the target call text is a call text of the first type, the first weight coefficient is greater than the second weight coefficient;

[0137] When the target call text is a call text of the second type, the first weight coefficient is less than the second weight coefficient;

[0138] Among them, the text review method of the first type of target call text is a non-manual review method, and the text review method of the second type of target call text is a manual review method.

[0139] Each module in the evaluation device of the above-mentioned factor extraction model can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0140] Figure 4 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0141] Reference Figure 4 An embodiment of the present disclosure provides an electronic device, which includes: at least one processor 701; at least one memory 702, and one or more I / O interfaces 703, connected between the processor 701 and the memory 702; wherein the memory 702 stores one or more computer programs that can be executed by the at least one processor 701, and the one or more computer programs are executed by the at least one processor 701 so that the at least one processor 701 can execute the above-mentioned evaluation method of the factor extraction model.

[0142] Each module in the above electronic device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0143] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned method for evaluating the factor extraction model when executed by a processor / processing core. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0144] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned method for evaluating the factor extraction model.

[0145] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0146] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0147] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0148] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0149] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0150] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0151] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0152] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0153] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0154] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A method for evaluating a factor extraction model, characterized in that: include: Obtaining a target text slice in a target call text; wherein, by means of a preset sliding window, the target call text is sequentially divided into a plurality of text slices, and the target text slice is the text slice currently being processed; Extracting target element information corresponding to preset elements from the target text slice through the element extraction model; According to the element extraction result of the historical text slice, the target element information is corrected to obtain the element extraction result of the target text slice; wherein the historical text slice is a processed text slice; Evaluate the element extraction results of the target text slice from at least two evaluation dimensions to obtain at least two dimensional evaluation results; According to the evaluation results of the at least two dimensions, a model evaluation result of the factor extraction model is determined.

2. The method according to claim 1, characterized in that The step of correcting the target element information according to the element extraction result of the historical text slice to obtain the element extraction result of the target text slice includes: Acquire a first element key-value pair included in the target element information, and extract a first element value in the first element key-value pair; Obtaining a second element key-value pair contained in the element extraction result of the historical text slice, and extracting a second element value matching the first element value from the second element key-value pair; Determine a second element key corresponding to the second element value; In the case where the first element key in the first element key-value pair does not match the second element key corresponding to the second element value, the first element key is modified according to the second element key.

3. The method according to claim 2, characterized in that Determining the second element key corresponding to the second element value includes: If there are multiple second element key-value pairs including the second element value, selecting a target element key-value pair from the multiple second element key-value pairs according to the occurrence frequency and / or occurrence time of the multiple second element key-value pairs; A second element key corresponding to the second element value is determined according to the target element key-value pair.

4. The method according to claim 2 or 3, characterized in that: The first element key and / or the second element key includes at least one of the following: An identity name used to characterize the user identity information of the calling user, an attribute name used to characterize the object attribute of the transaction object, and a role name used to characterize the transaction role of the calling user in the transaction process.

5. The method according to claim 1, characterized in that The element extraction result of the target text slice is evaluated from at least two evaluation dimensions, and the evaluation results of at least two dimensions are obtained, including: From the slice evaluation dimension, the element extraction result of each text slice is evaluated to obtain the slice dimension evaluation result; From the call evaluation dimension, evaluating the element extraction result of the target call text to obtain a call dimension evaluation result; From the factor evaluation dimension, the factor extraction result of each factor is evaluated to obtain the factor dimension evaluation result.

6. The method according to claim 5, characterized in that The dimension evaluation result of any evaluation dimension is determined according to the recall rate, the precision rate, a first weight coefficient used to characterize the importance of the recall rate, and a second weight coefficient used to characterize the importance of the precision rate; Wherein, when the target call text is a call text of the first type, the first weight coefficient is greater than the second weight coefficient; When the target call text is a call text of the second type, the first weight coefficient is less than the second weight coefficient; Among them, the text review method of the first type of target call text is a non-manual review method, and the text review method of the second type of target call text is a manual review method.

7. An evaluation device for a factor extraction model, characterized in that: include: A slice acquisition module, adapted to acquire a target text slice in a target call text; wherein the target call text is sequentially divided into a plurality of text slices by a preset sliding window, and the target text slice is a text slice currently being processed; An element extraction module, adapted to extract target element information corresponding to a preset element from the target text slice through the element extraction model; A correction module, adapted to correct the target element information according to the element extraction result of the historical text slice to obtain the element extraction result of the target text slice; wherein the historical text slice is a processed text slice; A dimension evaluation module, adapted to evaluate the element extraction result of the target text slice from at least two evaluation dimensions to obtain at least two dimensional evaluation results; The model evaluation module is adapted to determine the model evaluation result of the factor extraction model according to the evaluation results of the at least two dimensions.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor so that the at least one processor can execute the evaluation method of the factor extraction model as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the evaluation method of the factor extraction model according to any one of claims 1 to 6.

10. A computer program product, characterized in that It includes a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the evaluation method of the factor extraction model as described in any one of claims 1-6.

Images