Sample type identification method, device, storage medium, and program product

Abstract

Embodiments of the present application provide a sample type identification method and device, a storage medium and a program product. In the embodiments of the present application, under the guidance of configuration information such as positive and negative sample identification rules required for positive and negative sample identification based on rules corresponding to each interaction intention index configured through configuration operation, target prompt words and intention knowledge base required for positive and negative sample identification based on sample identification models, and in combination with rule-based sample identification processing and sample identification model-based identification processing, multiple interaction information generated in at least one interaction process included in a sample set is subjected to sample identification to obtain first and second identification results, so as to determine whether each interaction information is a positive sample or a negative sample based on the first and second identification results, improve the accuracy of the system in identifying positive and negative sample types, reduce the use of manual work, and ensure the balance between the performance and efficiency of positive and negative sample type identification.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a sample type identification method, device, storage medium and program product. Background Technology

[0002] Currently, many large models rely on machine learning algorithms for autonomous learning and information extraction when processing business classification metrics (i.e., labeling positive and negative samples related to business metrics). However, this approach typically lacks the ability to fine-tune classification criteria, resulting in lower classification accuracy in specific domains or complex scenarios. Because the model's learning process is primarily based on the features of the sample dataset and fails to fully incorporate human annotation knowledge, its understanding and application of industry characteristics are insufficient.

[0003] While some traditional manual classification systems offer relatively high accuracy, they are inefficient and difficult to scale. These systems often rely heavily on manual intervention, affecting the timeliness and accuracy of classification. Furthermore, manual classification is susceptible to subjective factors, making it difficult to guarantee the stability and consistency of the results.

[0004] Therefore, the above solutions still suffer from an imbalance between performance and efficiency. Summary of the Invention

[0005] This application provides a sample type identification method, device, storage medium, and program product to improve the accuracy of the system in identifying sample types, reduce manual intervention, and ensure a balance between performance and efficiency in sample type identification.

[0006] This application provides a sample type identification method, comprising: responding to a configuration operation; configuring at least one configuration information adapted to a target service provided by a target application; each configuration information including an interaction intent indicator; the interaction intent indicator corresponding to a positive and negative sample identification rule required for rule-based positive and negative sample identification, and a target prompt word and intent knowledge base required for positive and negative sample identification based on a sample identification model; collecting a sample set, the sample set containing multiple interaction information generated during at least one interaction process between the user and the target application, wherein the at least one interaction process is initiated by the user for the target service; and performing rule-based positive and negative sample identification processing on each interaction information according to the positive and negative sample identification rule and the interaction intent indicator to obtain a first identification result for each interaction information. The first identification result indicates whether the corresponding interaction information is a positive sample that matches the interaction intent indicator or a negative sample that does not match the interaction intent indicator. The target prompt word, the interaction intent indicator, and each interaction information are input into the sample type identification model. Under the guidance of the target prompt word and combined with the intent knowledge base, the model identifies whether each interaction information matches the interaction intent indicator to obtain the second identification result for each interaction information. The second identification result indicates whether the corresponding interaction information is a positive sample that matches the interaction intent indicator or a negative sample that does not match the interaction intent indicator. For each interaction information, if the first identification result and the second identification result of the interaction information are consistent, the interaction information is determined to be either a positive sample or a negative sample represented by either the first identification result or the second identification result.

[0007] This application also provides an electronic device, including: a memory and a processor; the memory for storing a computer program; and the processor, coupled to the memory, for executing the computer program to implement the steps in the above method.

[0008] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps in the above-described method.

[0009] This application also provides a computer program product, which includes a computer program / instructions that, when executed by a processor, enable the processor to perform the steps in the above-described method.

[0010] In this embodiment, through configuration operations, the interaction intent indicators adapted to the target service provided by the target application, the positive and negative sample recognition rules required for rule-based positive and negative sample recognition corresponding to the interaction intent indicators, and the target prompt words and intent knowledge base required for positive and negative sample recognition based on the sample recognition model can be configured. This configuration information guides the subsequent sample type recognition process, thereby improving the accuracy of positive and negative sample recognition. Furthermore, guided by the configuration information, and combining rule-based sample recognition processing and model-based recognition processing, sample recognition is performed on multiple interaction information generated in at least one interaction process within the sample set, yielding a first recognition result and a second recognition result. Based on the first and second recognition results, it is possible to determine whether each interaction information is a positive or negative sample, further improving the system's accuracy in sample type recognition, reducing manual intervention, and ensuring a balance between performance and efficiency in sample type recognition. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0012] Figure 1 A flowchart illustrating a sample type identification method provided in an exemplary embodiment of this application;

[0013] Figure 2a A schematic diagram of an evaluation page provided for an exemplary embodiment of this application;

[0014] Figure 2b A schematic diagram of another evaluation page provided for another exemplary embodiment of this application;

[0015] Figure 2c A schematic diagram of another evaluation page provided for another exemplary embodiment of this application;

[0016] Figure 2d A schematic diagram of another evaluation page provided for another exemplary embodiment of this application;

[0017] Figure 2e A schematic diagram of another evaluation page provided for another exemplary embodiment of this application;

[0018] Figure 2f A schematic diagram of another evaluation page provided for another exemplary embodiment of this application;

[0019] Figure 2g A schematic diagram of another evaluation page provided for another exemplary embodiment of this application;

[0020] Figure 2hA schematic diagram of another evaluation page provided for another exemplary embodiment of this application;

[0021] Figure 2i A schematic diagram of another evaluation page provided for another exemplary embodiment of this application;

[0022] Figure 2j A schematic diagram of another evaluation page provided for another exemplary embodiment of this application;

[0023] Figure 2k A schematic diagram of another evaluation page provided for another exemplary embodiment of this application;

[0024] Figure 2m A schematic diagram of another evaluation page provided for another exemplary embodiment of this application;

[0025] Figure 2l A schematic diagram of another evaluation page provided for another exemplary embodiment of this application;

[0026] Figure 3 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0029] To address the technical issues such as the performance-efficiency imbalance in existing sample type identification processes, this application embodiment configures the interaction intent indicators adapted to the target service provided by the target application, the corresponding rule-based positive and negative sample identification rules, and the target prompt words and intent knowledge base required for positive and negative sample identification based on the sample identification model. This configuration information guides the subsequent positive and negative sample type identification process, thereby improving the accuracy of positive and negative sample identification. Furthermore, guided by the configuration information, and combining rule-based sample identification processing and model-based identification processing, sample identification is performed on multiple interaction information generated in at least one interaction process within the sample set, yielding a first identification result and a second identification result. Based on these results, it is possible to determine whether each interaction information is a positive or negative sample. Determining the positive and negative sample type through both identification methods further improves the system's accuracy in sample type identification, reduces manual intervention, and ensures a balance between performance and efficiency in positive and negative sample type identification.

[0030] The following describes in detail a solution provided by an embodiment of this application, with reference to the accompanying drawings.

[0031] Figure 1 A flowchart illustrating a sample type identification method provided for an exemplary embodiment of this application. Figure 1 As shown, the method includes:

[0032] 101. Response configuration operation, configure at least one configuration information that is compatible with the target service provided by the target application. Each configuration information includes an interaction intent indicator. The interaction intent indicator corresponds to the positive and negative sample recognition rules required for rule-based positive and negative sample recognition, as well as the target prompt words and intent knowledge base required for positive and negative sample recognition based on the sample recognition model.

[0033] 102. Collect a sample set containing multiple interaction information generated during at least one interaction between the user and the target application, wherein at least one interaction is initiated by the user for the target service;

[0034] 103. Based on the positive and negative sample identification rules and the interaction intent index, perform rule-based positive and negative sample identification processing on each interaction information to obtain the first identification result of each interaction information. The first identification result indicates whether the corresponding interaction information is a positive sample that matches the interaction intent index or a negative sample that does not match the interaction intent index.

[0035] 104. Input the target prompt, interaction intent index and each interaction information into the sample type recognition model. Under the guidance of the target prompt, combined with the intent knowledge base, identify whether each interaction information is compatible with the interaction intent index to obtain the second recognition result of each interaction information. The second recognition result indicates whether the corresponding interaction information is a positive sample that is compatible with the interaction intent index or a negative sample that is not compatible with the interaction intent index.

[0036] 105. For each interactive information, if the first recognition result and the second recognition result of the interactive information are consistent, determine that the interactive information is a positive sample or a negative sample represented by either the first recognition result or the second recognition result.

[0037] Typically, to improve a model's reasoning ability, an initial model can be trained using a sample set adapted to its reasoning capabilities, and then fine-tuned based on evaluation data of the reasoning results. This sample set contains multiple samples. Alternatively, to improve rule-based online task processing capabilities, the task processing results for samples can be evaluated based on rules, and the corresponding rules can be continuously optimized and adjusted based on the evaluation results. Taking a sample recognition model for positive and negative sample type identification (binary classification) as an example, when training the sample recognition model based on samples, the initial model is trained using both positive and negative samples, which are two sample types. Positive samples are those with the target attribute or feature, meaning samples that the model is expected to identify as "yes," "existing," or "belongs to," etc. Negative samples are those without the target attribute or feature, meaning samples that the model is expected to identify as "no," "absent," or "does not belong to," etc. Furthermore, to further improve the accuracy of the inference results of this sample recognition model, it can be further fine-tuned based on both positive and negative samples. Positive and negative samples are obtained by identifying the positive and negative sample types of each sample in the sample set and labeling them based on the identified sample types. Therefore, the accuracy of positive and negative sample type identification determines the accuracy of the classification reasoning ability of the sample identification model. In other words, the accurate identification of positive and negative sample types is the key to ensuring the accuracy of the reasoning results of the sample identification model.

[0038] Sample recognition models can be used to classify and recognize samples from multiple domains. Classifying and recognizing samples from different domains requires training the model with samples from that domain to enable it to classify and recognize samples within that domain. Taking the interaction domain as an example, multiple interaction messages generated during at least one interaction process can be used as positive and negative samples to train the sample recognition model, enabling it to recognize the positive and negative samples contained within the interaction information. An interaction process can be, for example, an interaction between a user and any application, which can be called the target application. Target applications can be, for example, service applications, social applications, real estate transaction applications, etc. More specifically, the target application can provide multiple services (also known as businesses), and the interaction process can be initiated by the user for any one of the services provided by the target application, which can be called the target service. Target services can be, for example, customer service services, keyword search services, etc. Taking customer service services as an example, the interaction process is a chat process between the user and customer service. In this case, an interaction process can be a single question-and-answer session, including any question raised by the user and the customer service representative's response; an interaction process can also be multiple question-and-answer sessions within a single conversation between the user and customer service representative. Additionally, an interaction process can be at least one historical interaction between a user and the target application, or at least one current interaction between a user and the target application. A current interaction process refers to an interaction process ongoing within the current time period (e.g., today, within a few hours), while a historical interaction process refers to an interaction process that occurred before the current time period. Furthermore, user interactions with customer service can take place through the chat page provided by the customer service service, with the communication content between the user and customer service serving as the interaction information. Taking keyword search service as an example, the interaction process is a user-based information search process using keywords. Users can search based on keywords through the search page provided by the keyword search service, with the keywords serving as the interaction information.

[0039] Regardless of the target service, the interactive information generated during a user's interaction with that service typically carries a corresponding interaction intent, also known as an interaction intent metric. Each service may contain multiple interaction intent metrics. Taking customer service as an example, when a user is a merchant partnered with the target application, the merchant can purchase a membership to the application. This allows the merchant to join the application as a member and promote their products or services through posting and other means, thereby attracting customers. During this process, the merchant may have questions or needs that require communication with customer service, allowing customer service to resolve these issues or provide the necessary services. For example, if the merchant receives few or no inquiries about their products or services, the user can report this through the chat. Similarly, if a post receives few or no clicks, the user can report this through the chat. Furthermore, if a user needs to recharge their membership, they can inform customer service through the chat, allowing customer service to provide a recharge option. Finally, if a user needs to renew their subscription, they can inform customer service through the chat. For example, when a user needs to post something, they can ask customer service to help them post it through the chat page. Correspondingly, the interaction intent indicators included in customer service services include, but are not limited to: "few / no phone calls", "few / no post clicks", "want to recharge", "want to renew", "help post".

[0040] Based on this, in order to further improve the accuracy of positive and negative sample type identification, positive and negative samples that are compatible with each interaction intent indicator can be identified from the dimension of interaction intent indicators. This will facilitate the training or fine-tuning of the sample identification model based on the positive and negative samples that are compatible with each interaction intent indicator, or the fine-tuning of the rules corresponding to the rule-based online task based on the samples and negative samples that are compatible with each interaction intent indicator.

[0041] The following details the process of identifying positive and negative samples that match each interaction intent metric from the perspective of interaction intent metrics.

[0042] In this embodiment, at least one configuration information adapted to the target service provided by the target application can be configured in response to a configuration operation, so as to identify positive and negative samples based on the configuration information. One configuration information may be configuration information adapted to each interaction intent metric included in the target service provided by the target application.

[0043] The configuration operation can be performed by the user based on the information configuration page provided by the target application. The configuration operation can be manual configuration or voice control configuration, etc. Manual configuration means that the user manually fills in the configuration information in the information item or manually selects the appropriate configuration information from the drop-down list associated with each information item. Voice control operation means that the user uses voice to fill in the configuration information in the information item or uses voice to select the appropriate configuration information from the drop-down list associated with each information item.

[0044] Each configuration piece of information includes an interaction intent metric, which corresponds to a positive or negative sample identification rule required for rule-based positive and negative sample identification. Each configuration piece of information typically contains one interaction intent metric, while the target service may contain one or more interaction intent metrics. The positive and negative sample identification rules required for rule-based positive and negative sample identification include positive sample identification rules and negative sample identification rules. The positive sample identification rules define the identification criteria for positive samples to guide the positive sample identification process in rule-based positive and negative sample identification, and the negative sample identification rules define the identification criteria for negative samples to guide the negative sample identification process in rule-based positive and negative sample identification.

[0045] Optionally, positive and negative sample identification rules can be defined using regular expressions (Regex) to accurately identify positive and negative samples. Regular expressions are powerful tools for string searching and manipulation. They define a series of rules to match text that conforms to a certain pattern; that is, they are used to define identification rules and identify and extract text that matches a specific pattern. In simple terms, regular expressions are tools that use combinations of special characters to describe search patterns. For example, "*" represents any number of characters, and a*b can match any string that starts with 'a' and ends with 'b'. Taking "few phone calls intent" as an example, a regular expression can be used to identify the string "few phone calls / no phone calls," and samples containing this string can be identified as positive samples, while samples not containing this string can be identified as negative samples.

[0046] In this embodiment, the interaction intent indicator also corresponds to target prompt words and an intent knowledge base required for positive and negative sample identification based on the sample recognition model. The intent knowledge base contains knowledge information related to each intent indicator, and the specific meaning of each intent indicator can be learned through the knowledge information related to the intent indicator. Taking "few calls intent" as an example, the knowledge information related to "few calls intent" can be the definition of "few calls or no calls" and instances of "few calls". The definition of "few calls or no calls" can be, for example, "few calls means receiving fewer calls from users than a set number, and no calls means not receiving any calls from users". An instance of "few calls" can be, for example, "the merchant receives fewer than N calls". The target prompt words include, but are not limited to, at least one or more combinations of the first prompt word, the second prompt word, the third prompt word, and the fourth prompt word. Among them, the first prompt word is the task prompt word, which is used to prompt the sample recognition model about the specific task content in the entire process of positive and negative sample identification. The first prompt word can be, for example, "combining business knowledge and / or intent indicator knowledge, based on the interaction information between the merchant and customer service, determine whether the interaction information contains the intent of 'few calls / no calls'". The second prompt is the model role prompt, used to indicate the role the sample recognition model plays in the positive and negative sample identification process and the specific tasks of that role. For example, the second prompt could be: "You are an intent recognition expert, and you need to combine the interaction information between the merchant and customer service (the merchant's current question and historical dialogue information) to determine whether the interaction information contains the intent of 'few calls / no calls'." The third prompt is the positive sample identification rule, used to guide the sample recognition model to classify the interaction information as a positive sample if it identifies the intent of 'few calls / no calls'. For example, the third prompt could be: "If the interaction information contains the keyword 'few calls / no calls,' the interaction information is a positive sample." The fourth prompt is the negative sample prompt, used to guide the sample recognition model to classify the interaction information as a negative sample if it does not identify the intent of 'few calls / no calls'. For example, the fourth prompt could be: "If the interaction information contains the keyword 'few calls / no calls,' the interaction information is a negative sample."

[0047] In one optional embodiment, in response to a configuration operation, configuring at least one configuration information adapted to the target service provided by the target application includes: for each configuration information, displaying a configuration interface, the configuration interface including: an interaction intent indicator corresponding to the configuration information, and an information configuration item corresponding to the interaction intent indicator, the information configuration item including a positive and negative sample recognition rule configuration item, a sample recognition model configuration item, a prompt word configuration item, and an intent knowledge base configuration item; in response to an editing operation on the positive and negative sample recognition rule configuration item, obtaining the positive sample recognition rules and negative sample recognition rules required for positive and negative sample recognition based on the rules; in response to a selection operation on the sample recognition model configuration item, obtaining the sample recognition model corresponding to the interaction intent indicator; in response to an editing operation on the prompt word configuration item, obtaining the target prompt word required for positive and negative sample recognition based on the sample recognition model; in response to an editing operation on the intent knowledge base configuration item, obtaining the intent knowledge base required for positive and negative sample recognition based on the sample recognition model. In this embodiment, the configuration interface for each configuration information and the interaction intent indicator corresponding to each configuration information are pre-set. When displaying the configuration interface for each configuration information, the interaction intent indicator contained in the configuration information and the information configuration item corresponding to the interaction intent indicator are displayed simultaneously.

[0048] In another optional embodiment, in response to a configuration operation, configuring at least one configuration information adapted to the target service provided by the target application includes: displaying a configuration interface for each configuration information, the configuration interface including: intent indicator configuration items; in response to a configuration operation on the intent indicator configuration items, determining the interactive intent indicators included in the configuration information; displaying information configuration items on the configuration interface, the information configuration items including positive and negative sample recognition rule configuration items, sample recognition model configuration items, prompt word configuration items, and intent knowledge base configuration items; in response to an edit operation on the positive and negative sample recognition rule configuration items, obtaining the positive sample recognition rules and negative sample recognition rules required for rule-based positive and negative sample recognition; in response to a selection operation on the sample recognition model configuration items, obtaining the sample recognition model corresponding to the interactive intent indicator; in response to an edit operation on the prompt word configuration items, obtaining the target prompt words required for rule-based positive and negative sample recognition; and in response to an edit operation on the intent knowledge base configuration items, obtaining the intent knowledge base required for rule-based positive and negative sample recognition. In this embodiment, the configuration interface for each configuration information is pre-defined, but the interaction intent corresponding to the configuration information is not pre-defined. Users need to configure the intent indicator information in the configuration interface in real time to determine the intent indicator configuration items. In addition, since the information configuration items contained in the configuration page corresponding to each configuration information may be different, when displaying the configuration interface for each configuration information, the configuration page may only display the intent indicator items. After configuring the intent indicator through the intent indicator items, the information configuration items corresponding to that intent indicator are displayed.

[0049] In this embodiment, a sample set also needs to be collected. The sample set can be a sample set corresponding to an interaction intent indicator. Optionally, the interaction intent indicator corresponds to a sample set query command line, which can be used to find and collect sample sets related to the interaction intent from the data source. Taking a JSONPath expression as an example, the sample set query command line is a query language used to find and collect specific elements in JSON data. Assuming the following JSON data: {"name":"John","age":"30"}, the JSONPath expression $.extend.intentions can extract "John". For another example, in the interaction domain, the sample set contains multiple interaction information generated during at least one interaction between the user and the target application. Assuming the following JSON data:

[0050]

[0051] Based on the above JSON data, the JSONPath expression `$.extend.intentions` can be used to extract the following: "Merchant": "Why do I have so few phone calls?", "Customer Service": "Hello manager, I see you just started your promotion, so the number of calls is indeed a bit low. I suggest you refresh the post to improve the ranking, and add some keywords to increase exposure. Currently, the backend traffic and exposure are quite good; please continue to observe for a while." It should be noted that this question-and-answer pair can be considered a sample.

[0052] Furthermore, after collecting the sample set corresponding to the interaction intent indicator, positive and negative sample identification processing can be performed on the sample set of the interaction intent indicator. The positive and negative sample identification processing of the sample set of the interaction intent indicator can be initiated through the controls on the interaction page. For the specific process of positive and negative sample identification processing initiated through the controls on the interaction interface, please refer to the relevant description in the following scenario embodiments, which will not be repeated here.

[0053] In this embodiment of the application, positive and negative samples can be identified in the sample set corresponding to the interaction intent index through two positive and negative sample identification methods. The two positive and negative sample identification methods include: rule-based positive and negative sample identification and sample identification model-based positive and negative sample identification. The two sample identification methods are described in detail below.

[0054] Method 1: Rule-based positive and negative sample identification

[0055] In this embodiment of the application, the positive sample identification rule in the positive and negative sample identification rules is used to describe the keywords in the interaction intent index, and the negative sample identification rule in the positive and negative sample identification rules is used to describe the keywords that do not include the interaction intent index. Then, each interaction information can be processed based on the rules of positive and negative sample identification and the interaction intent index to obtain the first identification result of each interaction information.

[0056] In some embodiments, each interaction message is subjected to rule-based positive and negative sample identification processing according to positive and negative sample identification rules and interaction intent indicators to obtain a first identification result for each interaction message. This includes: determining whether each interaction message contains keywords from the interaction intent indicators according to the positive sample identification rules; if so, identifying the interaction message as a positive sample; and determining whether each interaction message does not contain keywords from the interaction intent indicators according to the negative sample identification rules; if not, identifying the interaction message as a negative sample. Distinguishing between positive and negative sample identification rules, and identifying each interaction message based on both rules to determine its category, can improve the accuracy of identifying the interaction message category.

[0057] Optionally, based on the positive sample identification rules, it is determined whether each interactive message contains keywords in the interactive intent indicator. This includes: determining keywords in the interactive intent indicator based on the positive sample identification rules; performing word segmentation on each interactive message to obtain multiple word segments contained in the interactive message; and determining whether the multiple word segments contain keywords in the interactive intent indicator under the guidance of the positive sample identification rules. By combining word segmentation with the positive sample identification rules, the accuracy of keyword identification is improved.

[0058] In an optional embodiment, determining keywords in the interaction intent metric based on positive sample identification rules includes: performing word segmentation on each interaction intent metric to obtain multiple word segments contained in the interaction intent metric; performing word segmentation on the positive sample identification rules to obtain multiple word segments contained in the positive sample identification rules; matching the multiple word segments contained in the interaction intent metric with the multiple word segments contained in the positive sample identification rules to obtain at least one pair of successfully matched word segments; extracting the semantic features of the at least one pair of successfully matched word segments and extracting the semantic features of the interaction intent metric; and selecting any one of the word segments whose semantic feature distance with the semantic features of the interaction intent metric is less than a preset distance as a keyword. By performing word segmentation on the positive sample identification rules and the interaction intent metric separately, and selecting the word segment whose semantic features are close to those of the interaction intent metric from the at least one pair of successfully matched word segments as a keyword in the interaction intent metric, instead of comparing the semantic features of all word segments with the semantic features of the interaction intent metric, this not only improves the accuracy of determining keywords in the interaction intent metric but also improves the efficiency of determining keywords in the interaction intent metric.

[0059] Among them, word segmentation is an important step in natural language processing (NLP), and its purpose is to split a continuous text string into a sequence of meaningful words. The word segmentation operation can be implemented by rule-based methods. Rule-based word segmentation methods include the forward maximum matching method (Maximum Matching, MM) and the reverse maximum matching method (Reverse Maximum Matching, RMM). Specifically, the implementation methods of the forward maximum matching method and the reverse maximum matching method include: pre-defining a dictionary that contains all possible words; further, scanning the text from left to right by the forward maximum matching method and using the longest matching word in the dictionary for word segmentation to obtain the word segmentation result; scanning the text from right to left by the reverse maximum matching method and using the longest matching word in the dictionary for word segmentation to obtain the word segmentation result. It should be noted that the forward maximum matching method and the reverse maximum matching method can be used alone or in combination, depending on the accuracy requirements of word segmentation and the application scenario. When the forward maximum matching method and the reverse maximum matching method are used alone, the specific implementation method of the forward maximum matching method includes: scanning the text of the interaction information from left to right and trying to match the longest word each time. Its advantages are simple implementation and high speed. The specific implementation method of the reverse maximum matching method includes: scanning the text of the interaction information from right to left and trying to match the longest word each time. Since there are many modifier-head structures in Chinese, the reverse maximum matching method is more accurate than the forward maximum matching method in some cases. For example, for "master postgraduate production", the reverse maximum matching method can be correctly segmented into "master / research / production", while the forward maximum matching method may be segmented into "master postgraduate / production". When the forward maximum matching method and the reverse maximum matching method are used in combination (bidirectional maximum matching method, Bi-MM), the bidirectional maximum matching method uses both the forward maximum matching method and the reverse maximum matching method at the same time, and then compares the results of the two methods. The specific selection rule is: if the number of words in the word segmentation results of the two methods is different, select the one with fewer words; if the number of words is the same, select the one with fewer single-character words; if it is still impossible to distinguish, choose either one. The bidirectional maximum matching method can reduce the error of single-direction matching and improve the accuracy of word segmentation.

[0060] In addition, word segmentation operations include statistical methods, deep learning methods, and other methods. Rule-based segmentation methods rely on predefined dictionaries and segmentation rules. Statistical segmentation methods utilize statistical information from large-scale corpora to determine word boundaries; common models include Hidden Markov Models (HMMs) and Conditional Random Fields (CRFs). Deep learning-based segmentation methods use neural network models such as Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and Transformers. The implementation of statistical segmentation methods includes: integrating a trained statistical model, such as an HMM or CRF, into the sample recognition model; using the trained statistical model to segment the interaction information contained in the sample set to obtain the segmentation results. The training process of the statistical model is as follows: preprocessing the interaction information; labeling the samples as segmentation results for model training; extracting features from the sample text, such as character context information; and training the statistical model, such as an HMM or CRF, using the labeled samples. The implementation process of deep learning-based word segmentation methods includes: integrating a trained neural network model, such as LSTM or Transformer, into the sample recognition model; using the neural network model to segment the interactive information to obtain the segmentation results. The training process of the neural network model includes: constructing a large-scale labeled sample; selecting a suitable neural network architecture; training the model using the labeled sample and optimizing the model parameters.

[0061] In another optional embodiment, the sample identification rule includes keywords from the interaction intent information indicator. The sample identification rule also has a preset data structure, and the keywords are located at a specified position in the data structure. Therefore, determining the keywords in the interaction intent indicator according to the positive sample identification rule includes: obtaining the keywords from the specified position in the data structure according to the data structure of the positive sample identification rule.

[0062] In another alternative embodiment, keywords can also be determined directly from the interaction intent metric. Specifically, the interaction intent metric has a preset data structure, and keywords are stored in a specified position within the data structure. Therefore, keywords can be directly retrieved from the specified position within the data structure corresponding to the interaction intent metric. For example, if the interaction intent metric is "few phone calls intent," and the data structure for "few phone calls intent" is "'few phone calls' (located in a specified position) + 'intent'," then the keyword "few phone calls" can be directly retrieved from the specified position.

[0063] Further optionally, guided by the positive sample identification rule, determining whether multiple word segments contain keywords in the interaction intent indicator includes: calculating the semantic feature distance between the semantic features of the keywords in the interaction intent indicator described according to the positive sample identification rule and the semantic feature distance between each semantic feature corresponding to the multiple word segments contained in each interaction intent information. The semantic feature distance represents semantic similarity; the smaller the semantic feature distance, the higher the semantic similarity, and vice versa. Word segments with a semantic feature distance less than a preset distance are considered as words similar to or the same as the keywords contained in the interaction intent indicator. If so, it is determined that multiple word segments contain keywords in the interaction intent indicator; otherwise, it is determined whether multiple word segments contain keywords in the interaction intent indicator.

[0064] Optionally, based on the negative sample identification rules, it is determined whether each interactive message does not contain keywords in the interactive intent indicator. This includes: determining keywords in the interactive intent indicator based on the negative sample identification rules; performing word segmentation on each interactive message to obtain multiple word segments contained in the interactive message; and, guided by the negative sample identification rules, determining whether the multiple word segments do not contain keywords in the interactive intent indicator. By combining word segmentation with the negative sample identification rules, the accuracy of keyword determination is improved. For the specific implementation of each step in this embodiment, please refer to the relevant description of determining whether each interactive message contains keywords in the interactive intent indicator based on the positive sample identification rules, which will not be repeated here.

[0065] Method 2: Positive and negative sample identification based on a sample recognition model

[0066] In this embodiment, the target prompt word, interaction intent index, and each interaction information can be input into the sample type recognition model. Guided by the target prompt word, and combined with the intent knowledge base, the model can identify whether each interaction information is compatible with the interaction intent index, so as to obtain the second recognition result of each interaction information.

[0067] In some embodiments, the sample type recognition model includes a knowledge learning network layer and a classification recognition network layer. The target prompt, interaction intent indicator, and each interaction message are input into the sample type recognition model. Guided by the target prompt and combined with an intent knowledge base, the model identifies whether each interaction message matches the interaction intent indicator to obtain a second classification recognition result for each interaction message. This includes: using the knowledge learning network layer to learn the knowledge meanings related to each intent indicator contained in the intent knowledge base; inputting the learned knowledge meanings related to each intent indicator, each interaction message, and the target prompt into the classification recognition network layer; and, guided by the knowledge meanings related to each intent indicator and the target prompt, identifying whether each interaction message matches the interaction intent indicator to obtain a second classification recognition result for each interaction message.

[0068] Optionally, the knowledge meaning related to each interaction intent indicator contained in the knowledge base is learned using the knowledge learning network layer. This includes extracting the semantic features of each piece of knowledge related to each interaction intent indicator, which can be in vector or matrix form. Further, optionally, each piece of knowledge related to each interaction intent indicator can be encoded to obtain its semantic features. For example, word embedding technology can be used to convert each piece of knowledge related to each interaction intent indicator into a vector representation, constructing a knowledge graph for each interaction intent indicator. This involves determining the entities, attributes, and relationships of each piece of knowledge related to each interaction intent indicator, and encoding the entities, attributes, and relationships into fact triples (v, r, v'), where v and v' are entities, and r is a relationship. Entities refer to things with independent existence meaning, such as people, places, and organizations. Entity encoding can use vector representation, matrix representation, or other methods. For example, in natural language processing, entities can be converted into word embedding vectors, and high-dimensional vectors can be generated to represent the entities. Attributes are data items used to describe the characteristics of an entity. Each entity has one or more attributes; for example, a student's attributes include student ID, name, and gender. Attribute encoding can convert attribute values ​​into numerical or vector forms. Gender can be encoded as 0 and 1 (0 for male, 1 for female), age can be directly represented numerically, and text attributes (such as name) can be converted into vectors using word embedding techniques. Relationships between entities refer to the various connections and interactions between different entities. For example, the "selection" relationship between "student" and "course." In an Entity-Relationship Diagram (ER), relationships are typically represented by diamonds connecting related entities. Relationship encoding can use matrix or graph representations. For example, an adjacency matrix can be used to represent relationships between entities, where elements indicate whether a relationship exists and its strength. In a graph representation, entities are nodes, relationships are edges, and edge weights represent the strength of the relationship. Knowledge graphs provide rich semantic features for models, helping them understand the meaning of knowledge related to intent metrics.

[0069] Optionally, the sample recognition model further includes a semantic feature extraction network layer, which is located between the knowledge learning network layer and the classification and recognition network layer. Each interaction information and the target prompt word are input into the semantic feature extraction network layer to encode the first prompt word (task prompt word), the second prompt word (model role), the third prompt word (positive sample recognition rule), the fourth prompt word (negative sample recognition rule), and each interaction information, thereby obtaining the semantic features (semantic feature vector) of each interaction information and each prompt word related to the knowledge of each intent indicator in the same vector space.

[0070] Optionally, the semantic feature extraction network layer can be a BERT (Bidirectional Encoder Representations from Transformers) model. Since the BERT model (BERT layer) can only process a limited amount of text at a time, to improve the semantic feature extraction efficiency of the semantic feature extraction layer, an exemplary sample recognition model structure is as follows: Figure 2a As shown, the sample recognition model contains multiple BERT layers. When each interaction information and each prompt word are input into the semantic feature extraction network layer, if the length of the text content corresponding to each interaction information and each prompt word is greater than the text length threshold supported by each BERT layer (the text length threshold supported by each BERT layer can be the same or different), each text content can be segmented into at least two text segments, and the length of each text segment is not greater than the text length threshold. Further, the at least two text segments are input into at least two semantic feature extraction layers for encoding, i.e., semantic feature extraction, to obtain at least two semantic features corresponding to each text content. The at least two semantic feature extraction layers are selected from multiple semantic feature extraction layers, with the same number of semantic feature extraction layers as the number of at least two text segments. Further, based on the at least two semantic features, target semantic features corresponding to each text content are generated to obtain the semantic features corresponding to each interaction information and the semantic features corresponding to each prompt word. In this embodiment, after obtaining at least two text segments, GPU memory is allocated to the same number of semantic feature extraction layers according to the number of text segments to support their operation, which can improve the extraction efficiency of semantic features and save GPU memory resources. It should be noted that when the text length of each text segment is different, it is necessary to adaptively select the semantic feature extraction layer that supports the text length of each text segment and allocate GPU memory to it in order to support its operation.

[0071] Optionally, each text segment contains multiple word segments; at least two text segments are input into at least two semantic feature extraction layers to extract semantic features, thereby obtaining at least two semantic features, including: performing a special word segmentation addition operation on each text segment to obtain at least two text segments containing special word segments; inputting each text segment into a semantic feature extraction layer with allocated GPU memory, and learning the semantic features of each word segment in the text segment and the semantic relationships between each word segment based on a self-attention learning mechanism, so as to obtain the target semantic features corresponding to the special characters as the semantic features of the text segment, thereby obtaining at least two semantic features.

[0072] Specifically, a special word segmentation and addition operation is performed on each text segment to obtain at least two text segments containing the special word segmentation. Optional implementation methods include: selecting special characters, the selection of which depends on the specific task and model architecture; common special characters include... <cls>(Classification) and <sep>(Separator), for example, in the BERT model, <cls>A marker is added to the beginning of the sentence. <sep>A token is added to the end of the sentence; tokenization: the text content is divided into multiple tokens, which can be words, subwords, or characters; the tokenizer converts the text content into a token sequence according to predefined rules, and the token sequence contains multiple tokens from each text content; adding special characters: special characters are added to the token sequence, for example, when processing a single sentence... <cls>It will be added to the beginning of the sequence. <sep>These will be added to the end of the sequence; encoding converts multiple word segments (including special characters) into numerical representations (numeric form) that the model can understand. Specifically, the numerical representation of each word can be found in the vocabulary (containing each word and its corresponding numerical mapping relationship), and the numerical representation of each word can be input into the model. The numerical representation of each word can represent the initial semantic information of that word. The numerical representation of each word can be implemented as a matrix containing the numerical representation of each word. In addition, in order for the model to understand the position of multiple words in the word segmentation sequence, positional encoding can be added to each word in the word segmentation sequence, which helps the model capture the order information in the sequence data.

[0073] In this process, each text segment containing special characters is input into a semantic feature extraction layer. Based on a self-attention learning mechanism, the semantic features of each word segment and the semantic relationships between each word segment are learned to obtain the semantic features of the text segment corresponding to each special character. Optional implementation methods include: performing vector mapping on each word segment in each text segment to obtain the query vector, key vector, and value vector corresponding to each word segment in the text segment; calculating the matching degree between the query vector of each word segment and the key vector of each word segment in the text segment to obtain the attention weight of each word segment in the text segment; and performing a fully connected computation operation on the value vector of each word segment in the text segment according to the attention weight of each word segment in the text segment to obtain at least two semantic features.

[0074] Optionally, the numerical representations corresponding to the word segmentation sequences containing special characters are input into the semantic feature extraction network layer. Vector mapping is performed on the numerical representations corresponding to each word in the target training samples to obtain the query vector, key vector, and value vector corresponding to each word. This includes mapping the numerical representations of each word in the word segmentation sequence to the query vector, key vector, and value vector respectively through different weight matrices, such as mapping to the query vector, key vector, and value vector respectively through three different weight matrices W^Q, W^K, and W^V.

[0075] Optionally, for each word segment, the matching degree is calculated using the query vector of that word and the key vectors of each word to obtain the attention weight of each word segment. This includes calculating the dot product between the query vector of the current word (each word) and the key vector of each word (including the current word) to obtain multiple score values. Each score value represents the attention weight of the word providing the key vector to the current word segment. The attention weight reflects the relative importance of one word to another in the sequence. The higher the weight, the more important the corresponding word is in the current context.

[0076] Optionally, the multiple scores may include both positive and negative scores. To facilitate subsequent fully connected computation, each score can be normalized to obtain multiple normalized scores, all of which are positive. This normalization process can be an exponential calculation operation (with the score as a parameter for the exponential calculation), but is not limited to this.

[0077] Optionally, after obtaining the attention weight of each word, before performing pooling, a weighted summation can be performed on the value vectors of each word based on its attention weight to obtain a weighted summation vector (updated representation). This weighted summation vector integrates word segments from all positions in the input sequence, but the contribution of each segment at a different position is determined by its attention relative to the query. This means that for each word in the segmentation sequence, its weighted summation vector is a weighted combination of information from other related words in the entire segmentation sequence, with the weights determined by the attention score (normalized). Through this process, the weighted summation vector of each word represents not only its own semantic information but also the comprehensive contextual semantic information of the entire segmentation sequence. This allows the model to capture long-distance dependencies within the sequence and consider the semantic information of the entire sequence when processing the current word.

[0078] Optionally, the sample recognition model further includes a fully connected layer. Located after the feature extraction network layer, after obtaining the weighted summation vector of each word and before pooling, a fully connected computation operation can be performed on the value vector of each word based on its attention weight to obtain the target semantic features corresponding to the special characters as the semantic features of the corresponding text. This includes: inputting the weighted summation vector into the fully connected layer, which includes two linear transformation functions and a non-linear activation function (such as ReLU); the first linear transformation function maps the value vector to an intermediate dimension, then applies the non-linear activation function for non-linear computation, and finally the second linear transformation function maps the non-linear computation result back to the original dimension or the target dimension to output the semantic information of the text content corresponding to the special characters. The semantic information of the text content integrates the contextual information of the entire input word segmentation sequence and can be used as the semantic information of the text content.

[0079] Optionally, the sample recognition model further includes a pooling layer located between the fully connected network layer and the classification recognition network layer. After obtaining at least two semantic features, the target semantic features corresponding to the text content are generated based on the at least two semantic features, including: using the pooling layer to perform pooling processing on the at least two semantic features to obtain the target semantic features corresponding to the text content.

[0080] In some embodiments, the pooling layer has a pooling window, the window size of which is equal to the sliding step size. Generating target semantic features corresponding to text content based on at least two semantic features includes: inputting at least two semantic features into the pooling layer; semantically concatenating the at least two semantic features to obtain concatenated semantic features, where the concatenated semantic features are multi-dimensional parameter vectors; based on the window size and sliding step size, selecting the maximum parameter value as the output within the area covered by each pooling window for the multi-dimensional parameter vector to obtain the target semantic features corresponding to the text content; or, based on the window size and sliding step size, calculating the average of all parameter values ​​as the output within the area covered by each pooling window for the multi-dimensional parameter vector to obtain the target semantic features corresponding to the text content; or, based on the window size and sliding step size, calculating the sum of all parameter values ​​as the output within the area covered by each pooling window for the multi-dimensional parameter vector to obtain the target semantic features corresponding to each text content.

[0081] It should be noted that the semantic feature extraction process in each embodiment of this application can refer to the above-described implementation method for extracting target semantic features corresponding to each text content.

[0082] Optionally, the learned knowledge meanings related to each intent indicator, each interactive information, and the target prompt word are input into the classification and recognition network layer. Guided by the knowledge meanings related to each intent indicator and the target prompt word, the network layer identifies whether each interactive information is compatible with the interactive intent indicator to obtain the second classification recognition result for each interactive information. This includes: inputting the learned knowledge meanings related to each intent indicator, the target semantic features corresponding to each interactive information and the target prompt word into the classification and recognition network layer; concatenating or fusing the target semantic features corresponding to each interactive information and the target prompt word into a comprehensive semantic feature (semantic feature vector), such as using concatenation or weighted average to obtain a comprehensive vector; and mapping the comprehensive vector to the number of output categories to obtain the second classification recognition result for each interactive information.

[0083] For example, the linear function `self.linear = nn.Linear(input_size, num_classes)` maps the comprehensive input vector to the number of output categories (vector form). `input_size` is the dimension of the input feature vector, `num_classes` is the number of output categories, and `nn.Linear` refers to linear transformation, whose mathematical expression is `y = Wx + b`, where W is the weight matrix and b is the bias vector. The softmax function is used to convert the number of output categories (vector form) into corresponding probability values. Each probability value represents the probability that each interaction information belongs to each result category. Taking the classification results as "yes (0)" and "no (1)" as an example, a probability threshold (50%) can be set. When the probability is greater than 50%, the classification result is "yes (0)", and when the probability is greater than 50%, the classification result is "no (1)". The probability results can be expressed as follows:

[0084] The first interactive information has a 70% probability of belonging to category 0 (yes) and a 30% probability of belonging to category 1 (no);

[0085] The second interactive information has a 20% probability of belonging to category 0 (yes) and an 80% probability of belonging to category 1 (no);

[0086] The third interactive information has a 60% probability of belonging to category 0 (yes) and a 40% probability of belonging to category 1 (no).

[0087] It should be noted that this embodiment does not limit the order of positive and negative sample identification based on rules and positive and negative sample identification based on sample identification models. For example, positive and negative sample identification can be performed first based on rules, and then based on sample identification models. Alternatively, positive and negative sample identification can be performed first based on sample identification models, and then based on rules. Furthermore, both rule-based and model-based positive and negative sample identification can be performed simultaneously; performing both processes concurrently can improve the efficiency of positive and negative sample identification.

[0088] Furthermore, after obtaining the first and second recognition results of multiple interactive information, the interactive information in which the first and second recognition results are consistent is taken as the first interactive information. The first interactive information is either a true sample or a true negative sample. A true sample means that the first and second recognition results of the same interactive information are both positive samples, and a true negative sample means that the first and second recognition results of the same interactive information are both negative samples.

[0089] In this embodiment, the first and second identification results of the same interactive information may be inconsistent. When the first and second identification results are inconsistent, the second identification result of the sample identification model can be used as the actual identification result. That is, the second identification result of the sample identification model can be understood as being used to verify the first identification result based on rules for sample identification. Furthermore, to avoid errors in the second identification result based on the sample identification model, the samples can be manually identified, and the manual identification result can be used as the actual identification result, thus serving as a manual verification and review of the sample identification result. Based on this, interactive information where the first and second identification results are inconsistent among multiple interactive information is considered as the second interactive information. For the second interactive information, manual classification and identification are performed to obtain the third identification result of the second interactive information. The third identification result indicates whether the second interactive information is a positive sample that matches the interactive intent indicator or a negative sample that does not match the interactive intent indicator. When the first and second recognition results are inconsistent, the interactive information is manually classified and recognized to obtain the actual recognition result. The manual recognition has a high accuracy rate, and the manual classification and recognition result is taken as the actual correct recognition result, which solves the problem of not being able to determine the correct recognition result corresponding to the interactive information when the first and second recognition results are inconsistent.

[0090] Furthermore, to improve the accuracy of positive and negative sample identification rules and model-based identification, the positive and negative sample identification rules or target prompts in the configuration information can be optimized and adjusted. Specifically, based on the first, second, and third identification results of the second interaction information, it can be determined whether the second interaction information is a false positive or false negative sample; based on the false positive or false negative sample, the positive and negative sample identification rules or target prompts in the configuration information can be optimized and adjusted. Here, a false positive sample refers to a sample whose second identification result from the sample category identification model is positive, while the third identification result from manual identification is negative; a false negative sample refers to a sample whose second identification result from the sample category identification model is negative, while the third identification result from manual identification is positive.

[0091] Optionally, determining whether the second interactive information is a false positive or false negative sample based on the first, second, and third identification results of the second interactive information includes: if the first identification result of the second interactive information is a positive sample, and the second and third identification results of the second interactive information are the same and both are negative samples, then the second interactive information is determined to be a false positive sample; if the first identification result of the second interactive information is a negative sample, and the second and third identification results of the second interactive information are different but the third identification result is a positive sample, then the second interactive information is determined to be a false negative sample. In other words, when the second and third identification results are the same, both the second and third identification results are the actual identification results corresponding to the sample; when the second and third identification results are different, the third identification result is the actual identification result corresponding to the sample, and the accuracy of the first identification result is determined based on the third identification result.

[0092] Optionally, based on false positive or false negative samples, the positive and negative sample identification rules or target prompts in the configuration information can be optimized and adjusted. This includes: calculating the accuracy, precision, recall, and the harmonic mean between precision and recall (i.e., the F1 score) for classifying the sample set based on the corresponding numbers of true samples, true negative samples, false positive samples, and false negative samples in the sample set; and optimizing and adjusting the positive and negative sample identification rules or target prompts in the configuration information based on the accuracy, precision, recall, and the harmonic mean between precision and recall for classifying the sample set. The F1 score ranges from [0,1]. A score closer to 1 indicates better performance for both sample-based and rule-based identification. Considering the balance between precision and recall for both sample-based and rule-based identification, the F1 score is crucial for binary classification problems, especially when a balance between precision and recall is desired. A high F1 score signifies a good balance between precision and recall achieved by both sample-based and rule-based identification methods.

[0093] Optionally, based on the respective numbers of true samples, true negative samples, false positive samples, and false negative samples contained in the sample set, the accuracy, precision, recall, and harmonic mean between precision and recall for classification and identification of the sample set are calculated, including: taking the sum of the number of true samples and the number of false positive samples as the numerator, and the sum of the number of true samples, true negative samples, false positive samples, and false negative samples as the numerator, and performing a quotient operation to obtain the accuracy; taking the number of true samples as the numerator, and the sum of the number of true samples and the number of false negative samples as the denominator, and performing a quotient operation to obtain the precision; taking the number of true samples as the numerator, and the sum of the number of true samples and the number of true negative samples as the denominator, and performing a quotient operation to obtain the recall; taking twice the product of precision and recall as the numerator, and the sum of precision and recall as the denominator, and performing a quotient operation to obtain the harmonic mean.

[0094] Understandably, optimizing the positive and negative sample identification rules or target prompts in the configuration information based on false positive or false negative samples involves optimizing these rules or prompts based on the accuracy, precision, recall, and the harmonic mean of precision and recall used for classifying the sample set. In other words, the accuracy, precision, recall, and the harmonic mean of precision and recall can be used as the loss function for classification. If the loss function does not meet the loss conditions, the positive and negative sample identification rules or target prompts in the configuration information are optimized until the loss function meets the loss conditions.

[0095] It should be noted that precision, accuracy, recall, and the harmonic mean of precision and recall are each treated as separate loss functions. Therefore, when optimizing the positive / negative sample identification rules or target prompts in the configuration information, each loss function must satisfy its corresponding loss condition. Alternatively, a comprehensive function of precision, accuracy, recall, and the harmonic mean of precision and recall can be calculated using weighted summation or similar methods. This comprehensive function can then be used as the loss function, ensuring that the loss condition of this loss function is met when optimizing the positive / negative sample identification rules or target prompts in the configuration information.

[0096] In this embodiment, through configuration operations, the interaction intent indicators adapted to the target service provided by the target application, the positive and negative sample recognition rules required for rule-based positive and negative sample recognition corresponding to the interaction intent indicators, and the target prompt words and intent knowledge base required for positive and negative sample recognition based on the sample recognition model can be configured. This configuration information guides the subsequent positive and negative sample type recognition process, thereby improving the accuracy of positive and negative sample recognition. Furthermore, guided by the configuration information, combining rule-based sample recognition processing and model-based recognition processing, sample recognition is performed on multiple interaction information generated in at least one interaction process within the sample set, yielding a first recognition result and a second recognition result. Based on the first and second recognition results, it is possible to determine whether each interaction information is a positive or negative sample. Determining the positive and negative sample type through the recognition results of both methods further improves the system's accuracy in sample type recognition, reduces manual intervention, and ensures a balance between performance and efficiency in positive and negative sample type recognition.

[0097] The execution process of each step in the above embodiments of this application can be controlled by the corresponding page. In order to facilitate the understanding of the above technical solution, the above technical solution will be described below with reference to the exemplary schematic diagram of each page.

[0098] First, combine Figure 2b and Figure 2c The overall functional structure and technical structure of this application are described. Figure 2b This is a schematic diagram illustrating the exemplary functional structure of this application, comprising an offline task layer, a presentation layer, a control layer, and a data layer. The offline tasks include: service source data (business data source (Hive table)), SQL process processing of the Hive table, dp Hive2es scheduled tasks, and Elasticsearch (ES) storage of real samples from multiple services. This involves extracting data from the service data source (Hive table), processing it with SQL, and then synchronizing the processed data to ES via scheduled tasks, ultimately for storing and querying real samples from multiple services (businesses). In the service (business) source data (Hive table), Hive is a Hadoop-based data warehouse tool used to store and process large-scale structured data. Business source data is typically stored in Hive as tables. The Hive table structure defines the data schema, including field names and data types. Data is stored as files in a distributed file system (Hadoop Distributed File System, HDFS), typically in columnar storage formats such as Parquet or ORC to optimize read performance. Business log data is loaded into the Hive table from the original data source (such as a log system or database) through an ETL (Extract, Transform, Load) process. The SQL processing of Hive tables involves manipulating data within them, typically including data cleaning, transformation, and aggregation. Specifically: Hive SQL: Using Hive SQL to query and process data. Hive SQL is similar to traditional relational database SQL, but optimized for large-scale data; Data cleaning: Removing invalid and duplicate data, correcting formatting errors, etc.; Data transformation: Converting data into a format suitable for subsequent processing, such as extracting specific fields or converting data types; Data aggregation: Aggregating data according to business needs, such as statistics by time, business type, etc.; Storing results: Storing the processed data in a new Hive table for use by subsequent tasks. Hive2es scheduled tasks refer to the data synchronization process triggered by scheduled tasks (such as Cron Jobs) to synchronize data from Hive tables to Elasticsearch. Specifically: Scheduled task scheduling: Using Cron expressions to define the execution frequency of tasks (e.g., hourly, daily); Data extraction: Extracting processed data from Hive tables. This can be done using Hive command-line tools (such as Hive-e) or through Hive's JDBC interface; Data transfer: Transferring data to Elasticsearch. Efficient data import can be achieved using Elasticsearch's Bulk API; script implementation: the entire synchronization process is typically implemented using shell scripts or Python scripts. Storing real-world samples from multiple services in Elasticsearch refers to storing processed data in Elasticsearch for real-time querying and analysis.Specifically, the Elasticsearch (ES) cluster is configured, including nodes, indexes, and shards. Index creation involves creating indexes based on business needs and defining index mappings, including field names and data types. Data import utilizes ES's Bulk API to import data in batches into specified indexes. Real-time querying leverages ES's query capabilities (such as the DSL query language) for real-time data querying and analysis. The presentation layer includes configuration information management (metric management), configuration information editing (metric editing), the evaluation homepage, task list, task details, and result analysis display. The control layer includes configuration information management (metric management), sample model identification (preliminary classification of large models), manual identification (manual annotation), and result analysis. The data layer includes service sample data (business sample data), multi-configuration information data (multi-metric data), sample identification model evaluation data (large model evaluation data), manually identified data (manually annotated data), and result data.

[0099] Figure 2c This is a schematic diagram illustrating the exemplary technical architecture of this application, providing technical architectural support for the various functions in section 2b. The front-end page corresponds to the presentation layer, which uses technologies such as Vue, ElementUI, JavaScript, and CSS to display content. JavaWeb Service and RPC Service provide technical support for the execution process of the control layer. RPCService includes, but is not limited to, Base DAO, EntityInterface, and EntityService. Basic components provide data services for the data layer, including, but not limited to, MySQL, JSON, multi-threaded management, sample recognition models, and MAVEN.

[0100] In this embodiment, an exemplary schematic diagram of the process of configuring at least one configuration information adapted to the target service provided by the target application in response to a configuration operation is shown below. Figure 2d-2e As shown, Figure 2d The target service selection page includes at least some of the services offered by the target application, along with an "Edit Metrics" control for each service. These services may include, for example, intelligent customer service, intelligent customer service tagging, and enterprise WeChat order recognition services. Additionally, the target service selection page includes the interface name for each service's data acquisition interface, the service type, and the service's update time. For each service, responding to a trigger operation on the "Edit Metrics" control displays the service's configuration page. For example, if the target service is intelligent customer service (chat service), responding to a trigger operation on the "Edit Metrics" control leads to the intelligent customer service's configuration page, where the service's configuration information can be configured. An example configuration page is shown below. Figure 2e As shown, the configuration page includes interactive intent metrics, "positive and negative sample recognition rules" and "sample set collection" items required for rule-based positive and negative sample recognition. These are used to configure the interactive intent metrics, positive and negative sample recognition rules, and sample set collection rules, respectively. The configuration page may also include "Add Intent Metric" controls, "Related Information for Adding Intent Metric" controls, and "Configuration Save" controls. New intent metrics can be added using the "Add Intent Metric" control, more information items related to the intent metrics can be added using the "Related Information for Adding Intent Metric" control, and the configured configuration information can be saved using the "Configuration Save" control, so that sample recognition can be performed based on the configured configuration information during the sample type recognition process.

[0101] In this embodiment, the sample set can be collected through the sample evaluation homepage. This page allows users to query and collect multiple samples related to the interaction intent metric to be evaluated. In other words, the sample set of any interaction intent of the target service can be collected through its sample evaluation homepage. An exemplary sample evaluation homepage is shown below. Figure 2f As shown, this page includes the following items: "Evaluation Service," "Data Environment," "Start Date," "Start Time," "End Time," "Sample Filtering (also known as Extended Query Filtering)," "Query" control, and "Start Evaluation" control. Specifically, the "Evaluation Service" item is used to configure the target service to be evaluated; the "Data Environment" item is used to configure the source of the sample data (e.g., from ES-online data); the "Date" item is used to configure the evaluation date; the "Start Time" item is used to configure the evaluation start time; the "End Time" item is used to configure the evaluation end time; the "Sample Filtering" item is used to filter multiple samples associated with the interaction intent metric to be evaluated from the database; the "Sample Filtering" item corresponds to a sample query command line, such as the JSONPath expression in the above embodiment; this sample query command line contains keywords related to the sample domain corresponding to the interaction intent metric, in order to query multiple samples containing the keyword, so as to identify positive and negative samples from multiple samples. The diagram uses the interaction intent "few phone calls" and the target service "intelligent customer service" as an example. The keyword related to the sample domain corresponding to this interaction intent metric is "phone". After configuring the evaluation service item, data environment item, start date item, start time item, end time item, and sample filtering item, in response to the triggering operation of the "query" control, a sample set related to the intent metric to be evaluated can be queried based on the above configuration information. The sample set related to "phone" contains 2214 samples. Among them, this sample set includes samples with the keyword "few phone calls / no phone calls" and samples with the keyword "phone" but not with the keyword "few phone calls / no phone calls".

[0102] Furthermore, after collecting the sample set corresponding to the interaction intent metrics, the "Start Evaluation" page is displayed in response to the triggering operation of the "Start Evaluation" control. An example "Start Evaluation" page is shown below. Figure 2g As shown in the diagram, the system includes: "Evaluation Task Name," "Intent Metric Selection" (also known as "Select Evaluation Metric"), "Requirement Association," and "OK." The "Evaluation Task Name" field configures the name of the evaluation task, the "Intent Metric Selection" field configures the interaction intent metric to be evaluated, and the "Requirement Association" field configures whether the evaluation task is associated with other requirements. The configuration information for the "Evaluation Task Name," "Intent Metric Selection," and "Requirement Association" fields can be pre-set automatically or manually set by the user in real time. Furthermore, in response to the user's "OK" button, the system begins rule-based positive and negative sample identification, sample identification model-based positive and negative sample identification, and evaluation of the sample set's identification results. After the evaluation is completed, the evaluation task list page is displayed. An exemplary diagram of the evaluation list page is shown below. Figure 2h and Figure 2m As shown, this diagram includes at least the "Query Details" control, "Query Statement" control, "Result Analysis" control, "View Analysis Results" control, and information details corresponding to the sample set evaluation results. The information details include, but are not limited to: sample set ID, task name, business name, associated requirements, total number of evaluations, number of completed tasks, and number of anomalies.

[0103] Furthermore, in response to a triggering action on the "View Details" control, the task details page is displayed. An example task details page is as follows: Figure 2i As shown in the diagram, this figure includes information such as the ID of each sample, the automatic evaluation status, the annotation status, and a "View Evaluation Results" control for each sample. The automatic evaluation status includes at least two states: "Partial Interaction Intent Metrics Verification Passed" and "All Metrics Verification Failed." Samples with "Partial Interaction Intent Metrics Verification Passed" and "All Metrics Verification Failed" indicate that the positive and negative sample identification rule results required for rule-based positive and negative sample identification are inconsistent with the positive and negative sample identification results based on the sample identification model. Furthermore, in response to triggering the "View Evaluation Results" control for any sample, the evaluation result for that sample is displayed. An example sample evaluation result is shown below. Figure 2j As shown in the figure, the first recognition result of the rule-based positive and negative sample identification corresponding to the sample in the figure is a positive sample, and the second recognition result of the sample-based positive and negative sample identification corresponding to the sample is a negative sample.

[0104] Furthermore, such as Figure 2j It also includes a "manual annotation" control, in such Figure 2j In cases where the first recognition result obtained from rule-based positive and negative sample identification and the second recognition result obtained from sample identification model identification are both negative samples, manual annotation can be performed. In response to the triggering operation of the "Manual Annotation" control, the manual annotation page is displayed. An exemplary manual annotation diagram is shown below. Figure 2k As shown, the manual annotation page includes an interaction intent indicator and a "manual annotation" item. In response to the selection operation based on the "manual annotation" item, the manual recognition result is determined, which is the third recognition result.

[0105] Furthermore, such as Figure 2h and Figure 2m As shown, in response to Figure 2h The "Results Analysis" control in the code allows for the analysis of evaluation results. After the analysis is complete, responding to a trigger on the "View Analysis Results" control displays the analysis results page. An example results analysis page is shown below. Figure 2l As shown, the results analysis page includes the interaction intent metric name, number of valid samples, number of invalid samples, number of true positives, number of true negatives, number of false positives, number of false negatives, accuracy, precision, recall, and F1 score, so as to optimize and adjust the positive and negative sample identification rules or target prompts in the configuration information based on these data.

[0106] Figure 3 A schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. For example... Figure 3 As shown, it includes: a memory 30a and a processor 30b; the memory 30a is used to store computer programs; the processor 30b is coupled to the memory 30a and is used to execute the computer programs to implement the steps in the sample type identification method described above.

[0107] Furthermore, such as Figure 3 As shown, the electronic device also includes other components such as a communication component 30c, a display 30d, a power supply component 30e, and an audio component 30f. Figure 3 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 3 The components shown.

[0108] The detailed implementation methods and beneficial effects provided in the embodiments of this application have been described in detail in the foregoing embodiments, and will not be elaborated here.

[0109] Exemplary embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps in the above-described method embodiments.

[0110] An exemplary embodiment of this application also provides a computer program product comprising a computer program / instructions that, when executed by a processor, enable the processor to perform the steps described in the above method embodiments.

[0111] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0112] The aforementioned communication components are configured to facilitate wired or wireless communication between the device containing the communication components and other devices. The device containing the communication components can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication components receive broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication components also include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0113] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0114] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0115] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0116] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program code.

[0117] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0120] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interfaces, network interfaces, and memory.

[0121] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0122] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0123] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0124] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.< / sep> < / cls> < / sep> < / cls> < / sep> < / cls>

Claims

1. A sample type identification method, characterized in that, include: In response to the configuration operation, configure at least one configuration information that is compatible with the target service provided by the target application. Each configuration information includes an interaction intent indicator, which corresponds to the positive and negative sample recognition rules required for rule-based positive and negative sample recognition, as well as the target prompt words and intent knowledge base required for positive and negative sample recognition based on the sample recognition model. Collect a sample set, which contains multiple interaction information generated during at least one interaction process between the user and the target application, wherein the at least one interaction process is initiated by the user for the target service; According to the positive and negative sample identification rules and the interaction intent index, each interaction information is processed for rule-based positive and negative sample identification to obtain a first identification result for each interaction information. The first identification result indicates that the corresponding interaction information is a positive sample that matches the interaction intent index or a negative sample that does not match the interaction intent index. The target prompt word, the interaction intent index, and each interaction information are input into the sample type recognition model. Under the guidance of the target prompt word, and in conjunction with the intent knowledge base, the model identifies whether each interaction information is compatible with the interaction intent index, so as to obtain a second recognition result for each interaction information. The second recognition result indicates whether the corresponding interaction information is a positive sample that is compatible with the interaction intent index or a negative sample that is not compatible with the interaction intent index. For each piece of interactive information, if the first identification result and the second identification result of the interactive information are consistent, the interactive information is determined to be a positive sample or a negative sample represented by either the first identification result or the second identification result.

2. The method according to claim 1, characterized in that, In response to configuration operations, configure at least one configuration piece of information adapted to the target service provided by the target application, including: For each configuration information, a configuration interface is displayed. The configuration interface includes: the interaction intent index corresponding to the configuration information, and the information configuration items corresponding to the interaction intent index. The information configuration items include positive and negative sample recognition rule configuration items, sample recognition model configuration items, prompt word configuration items, and intent knowledge base configuration items. In response to the editing operation of the positive and negative sample recognition rule configuration item, the positive sample recognition rule and negative sample recognition rule required for rule-based positive and negative sample recognition are obtained; In response to the selection operation of the sample recognition model configuration item, the sample recognition model corresponding to the interaction intent indicator is obtained; In response to the editing operation of the prompt word configuration item, the target prompt word required for positive and negative sample recognition based on the sample recognition model is obtained; In response to the editing operation of the intent knowledge base configuration item, the intent knowledge base required for positive and negative sample recognition based on the sample recognition model is obtained.

3. The method according to claim 1, characterized in that, The positive sample identification rule in the positive and negative sample identification rule is used to describe keywords in the interaction intent index, and the negative sample identification rule in the positive and negative sample identification rule is used to describe keywords that do not include the interaction intent index. Based on the positive and negative sample identification rules and the interaction intent index, each interaction information undergoes rule-based positive and negative sample identification processing to obtain a first identification result for each interaction information, including: According to the positive sample identification rule, determine whether each interaction message contains the keywords in the interaction intent index. If it does, determine that the interaction message is a positive sample. According to the negative sample identification rule, it is determined whether each interaction message does not contain the keywords in the interaction intent index. If it does not contain them, the interaction message is determined to be a negative sample.

4. The method according to claim 3, characterized in that, Based on the positive sample identification rules, determine whether each interaction message contains keywords from the interaction intent indicator, including: Based on the positive sample identification rule, the keywords in the interaction intent index are determined; each interaction message is segmented to obtain multiple segments contained in the interaction message; under the guidance of the positive sample identification rule, it is determined whether the multiple segments contain the keywords in the interaction intent index. Based on the negative sample identification rules, determine whether each interaction message does not contain the keywords in the interaction intent indicator, including: Based on the negative sample identification rule, the keywords in the interaction intent index are determined; each interaction message is segmented to obtain multiple segments contained in the interaction message; guided by the negative sample identification rule, it is determined whether the multiple segments do not contain the keywords in the interaction intent index.

5. The method according to claim 1, characterized in that, The sample type recognition model includes a knowledge learning network layer and a classification recognition network layer. The target prompt word, the interaction intent indicator, and each interaction piece of information are input into the sample type recognition model. Guided by the target prompt word and combined with the intent knowledge base, the model identifies whether each interaction piece of information matches the interaction intent indicator to obtain a second classification recognition result for each interaction piece of information, including: The knowledge learning network layer is used to learn the knowledge meaning related to each intent indicator contained in the intent knowledge base; The learned knowledge meanings related to each intent indicator, each interaction information, and the target prompt word are input into the classification and recognition network layer. Guided by the knowledge meanings related to each intent indicator and the target prompt word, the network layer identifies whether each interaction information is compatible with the interaction intent indicator, so as to obtain the second classification and recognition result of each interaction information.

6. The method according to any one of claims 1-5, characterized in that, Also includes: The interaction information in which the first identification result and the second identification result are consistent among the multiple interaction information is taken as the first interaction information, and the first interaction information is a true sample or a true negative sample. Interaction information in which the first identification result and the second identification result are inconsistent among the multiple interaction information is taken as the second interaction information. For the second interaction information, the second interaction information is manually classified and identified to obtain the third identification result of the second interaction information. The third identification result indicates that the second interaction information is a positive sample that matches the interaction intent index or a negative sample that does not match the interaction intent index. Based on the first recognition result, the second recognition result, and the third recognition result of the second interaction information, the second interaction information is determined to be a false positive sample or a false negative sample. Based on the false positive or false negative samples, the positive and negative sample identification rules or target prompt words in the configuration information are optimized and adjusted.

7. The method according to claim 6, characterized in that, Based on the first recognition result, the second recognition result, and the third recognition result of the second interaction information, the second interaction information is determined to be a false positive sample or a false negative sample, including: If the first recognition result of the second interactive information is a positive sample, and the second recognition result and the third recognition result of the second interactive information are the same and both are negative samples, then the second interactive information is determined to be a false positive sample. If the first recognition result of the second interactive information is a negative sample, and the second recognition result of the second interactive information is different from the third recognition result and the third recognition result is a positive sample, then the second interactive information is determined to be a false negative sample.

8. The method according to claim 6, characterized in that, Based on the false positive or false negative samples, the positive and negative sample identification rules or target prompts in the configuration information are optimized and adjusted, including: Based on the number of each of the true samples, true negative samples, false positive samples, and false negative samples contained in the sample set, calculate the accuracy, precision, recall, and harmonic mean between precision and recall for the classification and recognition of the sample set. Based on the accuracy, precision, recall, and harmonic mean between precision and recall for the classification and recognition of the sample set, the positive and negative sample recognition rules or target prompt words in the configuration information are optimized and adjusted.

9. The method according to claim 8, characterized in that, Based on the number of each of the true samples, true negative samples, false positive samples, and false negative samples in the sample set, calculate the accuracy, precision, recall, and the harmonic mean between precision and recall for classification and recognition of the sample set, including: The accuracy rate is obtained by performing a quotient operation on the sum of the true number of samples and the false positive number of samples, and the sum of the true number of samples, the true number of negative samples, the false positive number of samples, and the false negative number of samples. The precision rate is obtained by performing a quotient operation using the true sample count as the numerator and the sum of the true sample count and the false negative sample count as the denominator. The recall rate is obtained by performing a quotient operation using the true sample count as the numerator and the sum of the true sample count and the true negative sample count as the denominator. The harmonic mean is obtained by taking twice the product of precision and recall as the numerator and the sum of precision and recall as the denominator.

10. An electronic device, characterized in that, include: Memory and processor; The memory is used to store a computer program; the processor, coupled to the memory, is used to execute the computer program to implement the steps of the method according to any one of claims 1-9.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method according to any one of claims 1-9.

12. A computer program product, characterized in that, The computer program product includes a computer program / instruction that, when executed by a processor, causes the processor to perform the steps of any one of the methods of claims 1-9.

Images