Keyword definition method and apparatus, computer device, and storage medium

Abstract

The application relates to a keyword definition method and device, computer equipment and a storage medium. The application relates to the fields of prompt learning and artificial intelligence technology. The method comprises the following steps: acquiring a plurality of text information that needs to be classified and defined, and extracting keywords of each text information; for each text information, analyzing the distribution probability of the keywords of each text information, thereby screening target keywords of the text information; for each text information, generating a feature vector of the text information based on the target keywords of the text information and the distribution probability thereof, and respectively calculating the similarity between each feature vector, thereby determining text representation information, and calculating label representation information of a text category for each text category; screening text representation information with the maximum distribution probability from the label representation information as keyword definition information of the text category. The method can improve the keyword definition accuracy of text classification in a professional field.

Description

Technical Field

[0001] This application relates to the fields of prompting learning and artificial intelligence technology, and in particular to a keyword definition method, apparatus, computer device, and storage medium. Background Technology

[0002] Traditional cue learning models are relatively simple to define verbalizers for text classification in general domains. However, text classification in specialized domains (such as finance) generally requires more domain knowledge, making verbalizer definition for specialized text classification very difficult. Therefore, improving the accuracy of verbalizer definition for specialized text classification is a current research focus.

[0003] Traditional technical solutions involve manually defining keywords for text classification in specialized fields. However, this method is not only labor-intensive, but the defined keywords may not accurately reflect the data characteristics of the text categories, resulting in low accuracy in keyword definition for text classification in specialized fields. Summary of the Invention

[0004] Therefore, it is necessary to provide a keyword definition method, apparatus, computer device, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0005] Firstly, this application provides a keyword definition method. The method includes:

[0006] Obtain multiple text messages that need to be categorized and defined, and extract keywords from each text message;

[0007] For each piece of text information, analyze the distribution probability of keywords in the text information, and based on the distribution probability, filter the target keywords of the text information;

[0008] For each piece of text information, a feature vector is generated based on the target keywords of the text information and the probability distribution of each target keyword, and the average similarity between each feature vector and the other feature vectors is calculated respectively.

[0009] The feature vector corresponding to the maximum average similarity is used as the text representation information of the text information, and the text category corresponding to each text information is identified;

[0010] For each text category, based on the text representation information of each text information corresponding to the text category, the label representation information of the text category is calculated, and the distribution probability between each text representation information and the label representation information is calculated respectively;

[0011] The text representation information with the highest probability distribution is selected as the keyword definition information for the text category.

[0012] Optionally, the extraction of keywords for each text message includes:

[0013] For each piece of text information, the text information is divided according to the segments contained in the text information to obtain multiple text segments, and the semantic features of each text segment are identified through a text feature recognition network;

[0014] Calculate the similarity between each word in each text segment and the semantic features of each text segment, and select the word information corresponding to the maximum similarity in each text segment as the sub-keywords of the text segment;

[0015] All sub-keywords of the text segments are used as keywords for the text information.

[0016] Optionally, the analysis of the distribution probability of keywords in each text message includes:

[0017] For each keyword, identify the number of times the keyword appears in the text information, and identify the total number of words in the text information;

[0018] The ratio of the number of keywords to the total number of words in the text is calculated to obtain the probability distribution of the keywords in the text information.

[0019] Optionally, calculating the average similarity between each feature vector and all other feature vectors includes:

[0020] Calculate the cosine similarity between each pair of feature vectors and the loss function between each pair of feature vectors respectively, and calculate the similarity between each pair of feature vectors based on the cosine similarity between each pair of feature vectors and the loss function between each pair of feature vectors.

[0021] For each feature vector, the average similarity between each feature vector and the other feature vectors is calculated based on the similarity between the feature vector and the other feature vectors.

[0022] Optionally, identifying the text category corresponding to each text piece of information includes:

[0023] Obtain the sample text categories and extract the category feature information for each sample text category;

[0024] Identify the text content corresponding to each category feature information, and calculate the similarity distance between the target keywords of each text information and the text content corresponding to each category feature information;

[0025] For each piece of text information, the sample text category to which the category feature information corresponding to the minimum similarity distance belongs is selected as the text category of the text information.

[0026] Optionally, calculating the probability distribution between each text representation and the tag representation includes:

[0027] Identify the position information of each text representation information in the text information corresponding to each text representation information, and calculate the similarity score between each text representation information and each tag representation information based on the position information corresponding to each text representation information and the tag representation information through a scoring function;

[0028] Based on the similarity score between each text representation information, the label representation information of the text category corresponding to each text representation information, and the similarity score between each text representation information and each of the aforementioned label representation information, the distribution probability between each text representation information and the label representation information of the text category corresponding to each text representation information is calculated using a distribution probability algorithm.

[0029] Optionally, after filtering the text representation information with the highest distribution probability as the keyword definition information of the text category, the method further includes:

[0030] In response to the user's keyword definition information upload operation, the actual keyword definition information of each text category is obtained, and for each text category, the keyword definition information of the text category and the deviation information between the keyword definition information of the text category and the actual keyword definition information of the text category are identified;

[0031] Calculate the ratio between the deviation information and the keyword definition information, and if the ratio is greater than a preset ratio threshold, adjust the scoring parameters of the scoring function for the text category based on the deviation information to obtain a new scoring function.

[0032] Secondly, this application also provides a keyword definition device. The device includes:

[0033] The acquisition module is used to acquire multiple text messages that need to be categorized and defined, and to extract keywords from each text message.

[0034] The analysis module is used to analyze the distribution probability of keywords in each text message for each text message, and to filter target keywords in the text message based on the distribution probability;

[0035] The generation module is used to generate a feature vector for each text information based on the target keywords of the text information and the probability distribution of each target keyword, and to calculate the average similarity between each feature vector and other feature vectors respectively.

[0036] The recognition module is used to take the feature vector corresponding to the maximum average similarity as the text representation information of the text information, and to identify the text category corresponding to each text information.

[0037] The calculation module is used to calculate the tag representation information of each text category based on the text representation information of each text information corresponding to the text category, and to calculate the distribution probability between each text representation information and the tag representation information respectively;

[0038] The filtering module is used to filter text representation information with the highest probability distribution as keyword definition information for the text category.

[0039] Optionally, the acquisition module is specifically used for:

[0040] For each piece of text information, the text information is divided according to the segments contained in the text information to obtain multiple text segments, and the semantic features of each text segment are identified through a text feature recognition network;

[0041] Calculate the similarity between each word in each text segment and the semantic features of each text segment, and select the word information corresponding to the maximum similarity in each text segment as the sub-keywords of the text segment;

[0042] All sub-keywords of the text segments are used as keywords for the text information.

[0043] Optionally, the analysis module is specifically used for:

[0044] For each keyword, identify the number of times the keyword appears in the text information, and identify the total number of words in the text information;

[0045] The ratio of the number of keywords to the total number of words in the text is calculated to obtain the probability distribution of the keywords in the text information.

[0046] Optionally, the generation module is specifically used for:

[0047] Calculate the cosine similarity between each pair of feature vectors and the loss function between each pair of feature vectors respectively, and calculate the similarity between each pair of feature vectors based on the cosine similarity between each pair of feature vectors and the loss function between each pair of feature vectors.

[0048] For each feature vector, the average similarity between each feature vector and the other feature vectors is calculated based on the similarity between the feature vector and the other feature vectors.

[0049] Optionally, the identification module is specifically used for:

[0050] Obtain the sample text categories and extract the category feature information for each sample text category;

[0051] Identify the text content corresponding to each category feature information, and calculate the similarity distance between the target keywords of each text information and the text content corresponding to each category feature information;

[0052] For each piece of text information, the sample text category to which the category feature information corresponding to the minimum similarity distance belongs is selected as the text category of the text information.

[0053] Optionally, the computing module is specifically used for:

[0054] Identify the position information of each text representation information in the text information corresponding to each text representation information, and calculate the similarity score between each text representation information and each tag representation information based on the position information corresponding to each text representation information and the tag representation information through a scoring function;

[0055] Based on the similarity score between each text representation information, the label representation information of the text category corresponding to each text representation information, and the similarity score between each text representation information and each of the aforementioned label representation information, the distribution probability between each text representation information and the label representation information of the text category corresponding to each text representation information is calculated using a distribution probability algorithm.

[0056] Optionally, the device further includes:

[0057] The response module is used to respond to the user's keyword definition information upload operation, obtain the actual keyword definition information of each text category, and for each text category, identify the keyword definition information of the text category and the deviation information between the keyword definition information of the text category and the actual keyword definition information of the text category;

[0058] An adjustment module is used to calculate the ratio between the deviation information and the keyword definition information, and if the ratio is greater than a preset ratio threshold, adjust the scoring parameters of the scoring function for the text category based on the deviation information to obtain a new scoring function.

[0059] Thirdly, this application provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in any one of the first aspects.

[0060] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0061] Fifthly, this application provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0062] The aforementioned keyword definition method, apparatus, computer device, storage medium, and computer program product acquire multiple text information pieces that need to be classified and defined, and extract keywords from each text information piece; for each text information piece, analyze the distribution probability of the keywords in the text information piece, and based on the distribution probability, filter the target keywords of the text information piece; for each text information piece, generate a feature vector of the text information piece based on the target keywords of the text information piece and the distribution probability of each target keyword, and calculate the average similarity between each feature vector and other feature vectors respectively; use the feature vector corresponding to the maximum average similarity as the text representation information of the text information piece, and identify the text category corresponding to each text information piece; for each text category, calculate the tag representation information of the text category based on the text representation information of each text information piece corresponding to the text category, and calculate the distribution probability between each text representation information piece and the tag representation information respectively; filter the text representation information piece with the maximum distribution probability as the keyword definition information of the text category. This scheme calculates the text representation information of each text by filtering the similarity between the feature vectors of each text by selecting the target keywords of each text. Then, by identifying the text category of each text, it calculates the tag representation information corresponding to each text in the same text category. Finally, it selects the text representation information with the highest probability distribution with the tag representation information as the keyword definition information of the text category. This allows the identification of the similarity between representation information to replace the low-precision problem of manual definition. Furthermore, by selecting the keyword definition information of the text category from the tag representation information corresponding to the semantically refined text representation information of each text, the comprehensiveness of the keyword definition information in summarizing the text features of each text in the text category is improved, thereby improving the accuracy of keyword definition for text classification in professional fields. Attached Figure Description

[0063] Figure 1 This is a flowchart illustrating a keyword definition method in one embodiment;

[0064] Figure 2 A flowchart illustrating an example of keyword definition in one embodiment;

[0065] Figure 3 This is a structural block diagram of a keyword definition device in one embodiment;

[0066] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] The keyword definition method provided in this application can be applied to a prompt text learning environment. This method can be applied to a terminal, a server, or a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc. The terminal calculates the text representation information of each text by filtering the similarity between the feature vectors of each text by selecting target keywords. Then, by identifying the text category of each text, it calculates the tag representation information corresponding to each text in the same category. The text representation information with the highest probability distribution with the tag representation information is then selected as the keyword definition information for the text category. This allows for similarity recognition between representation information, replacing the low-precision problem of manual definition. Furthermore, by selecting the keyword definition information for the text category from the tag representation information corresponding to the semantically refined text representation information of each text, the comprehensiveness of the keyword definition information in summarizing the text features of each text in the text category is improved, thereby enhancing the accuracy of keyword definition for text classification in professional fields.

[0069] In one embodiment, such as Figure 1 As shown, a keyword definition method is provided. Taking the application of this method to a terminal as an example, the method includes the following steps:

[0070] Step S101: Obtain multiple text information that needs to be classified and defined, and extract keywords from each text information.

[0071] In this embodiment, the terminal responds to the user's information upload operation and obtains multiple text messages that need to be categorized. These text messages differ in text type, including the department to which they belong and their application function. Then, the terminal extracts keywords from each text message. The specific extraction process will be explained in detail later. These keywords are words that characterize the semantics of the text message and are important for understanding it. A single text message may contain multiple words.

[0072] Step S102: For each piece of text information, analyze the distribution probability of keywords in the text information, and based on the distribution probability, filter the target keywords of the text information.

[0073] In this embodiment, the terminal analyzes the probability distribution of keywords in each text message and filters target keywords based on these probabilities. Specifically, the terminal presets a probability distribution threshold, and then filters keywords with probabilities greater than the threshold from each keyword in each text message as target keywords for that text message.

[0074] Step S103: For each piece of text information, generate a feature vector based on the target keywords of the text information and the distribution probability of each target keyword, and calculate the average similarity between each feature vector and other feature vectors.

[0075] In this embodiment, the terminal generates a feature vector for each piece of text information based on the target keywords and the probability distribution of each target keyword. The formula corresponding to this feature vector is:

[0076]

[0077] In the above formula, It is a K-dimensional vector, where each dimension represents the importance of the target keyword (i.e., the probability distribution of that keyword), and together they can form the input X. i A feature vector in the latent space, where i is the virtual number of the keyword.

[0078] Then, the terminal calculates the average similarity between each feature vector and all other feature vectors. The specific calculation process for the average similarity will be explained in detail later.

[0079] Step S104: Use the feature vector corresponding to the maximum average similarity as the text representation information of the text information, and identify the text category corresponding to each text information.

[0080] In this embodiment, the terminal uses the feature vector corresponding to the maximum average similarity as the text representation information of the text information, and identifies the text category corresponding to each text information. The text category is a preset category of text information on the terminal, which is obtained by staff classifying financial-related text information according to business needs.

[0081] Step S105: For each text category, based on the text representation information of each text information corresponding to the text category, calculate the label representation information of the text category, and calculate the distribution probability between each text representation information and the label representation information respectively.

[0082] In this embodiment, the terminal calculates the tag representation information of each text category based on the text representation information of each text information corresponding to the text category.

[0083] The specific formula for calculating the label representation information is as follows:

[0084]

[0085] In the above formula, I l C represents the number of all text messages in text category l. l For label characterization information, For each text information, i is the virtual number of the target keyword in the text representation information, and K is the dimension of each target keyword in a single text information.

[0086] Then, the terminal calculates the probability distribution between each text representation and the tag representation. The specific calculation process will be explained in detail later.

[0087] Step S106: Filter the text representation information with the highest distribution probability as the keyword definition information for the text category.

[0088] In this embodiment, the terminal filters the text representation information with the highest probability distribution as the keyword definition information for the text category.

[0089] Based on the above scheme, the similarity between the feature vectors of each text information obtained by filtering the target keywords of each text information is used to calculate the text representation information of each text information. Then, by identifying the text category of each text information, the label representation information corresponding to each text information of the same text category is calculated. The text representation information with the highest distribution probability of the label representation information is then selected as the keyword definition information of the text category. This allows the identification through the similarity between representation information to replace the low-precision problem of manual definition. Furthermore, by selecting the keyword definition information of the text category from the label representation information corresponding to the semantically refined text representation information of each text information, the comprehensiveness of the keyword definition information in summarizing the text features of each text information in the text category is improved, thereby improving the accuracy of keyword definition for text classification in professional fields.

[0090] Optionally, keywords for each text information are extracted, including: for each text information, dividing the text information into multiple text segments according to the segments contained in the text information, and identifying the semantic features of each text segment through a text feature recognition network; calculating the similarity between each word information of each text segment and the semantic features of each text segment, and selecting the word information corresponding to the maximum similarity in each text segment as the sub-keywords of the text segment; and using the sub-keywords of all text segments as the keywords of the text information.

[0091] In this embodiment, the terminal divides each text information into multiple text segments according to the segments it contains, and then identifies the semantic features of each text segment using a text feature recognition network. The text feature recognition network is a fully convolutional neural network based on an attention mechanism. Next, the terminal calculates the similarity between each word in each text segment and the semantic features of that text segment. The similarity calculation method involves using a similarity distance algorithm to calculate the similarity distance between word information and semantic features. Then, the similarity distances between each word and semantic feature are sorted to obtain a similarity distance sequence. Finally, for each word, the terminal calculates the similarity distance between the word and the semantic features by subtracting the similarity distance sequence position from the total number of words, and then dividing by the total number of words. The similarity distance algorithm can be, but is not limited to, Euclidean distance, Mahalanobis distance, etc.

[0092] The terminal filters the words with the highest similarity in each text segment and uses them as sub-keywords for that text segment. Finally, the terminal uses the sub-keywords of all text segments as the keywords for the text information.

[0093] Based on the above scheme, by identifying the semantic features of each text segment, keywords are selected, thereby improving the accuracy of the selected keywords in representing the textual meaning of the text information.

[0094] Optionally, the distribution probability of keywords in each text message can be analyzed, including: for each keyword, identifying the number of keywords in the text message and identifying the total number of words in the text message; calculating the ratio of the number of keywords to the total number of words to obtain the distribution probability of keywords in the text message.

[0095] In this embodiment, the terminal identifies the number of keywords in the text information for each keyword, and also identifies the total number of words in the text information. Then, the terminal calculates the ratio of the number of keywords to the total number of words to obtain the probability distribution of the keywords in the text information.

[0096] Based on the above scheme, by identifying the number of keywords in the text information, the distribution probability of keywords in the text information is calculated, thereby improving the accuracy of the calculated distribution probability.

[0097] Optionally, the average similarity between each feature vector and all other feature vectors can be calculated separately, including: calculating the cosine similarity between each pair of feature vectors and the loss function between each pair of feature vectors, and calculating the similarity between each pair of feature vectors based on the cosine similarity between each pair of feature vectors and the loss function between each pair of feature vectors; for each feature vector, calculating the average similarity between each feature vector and all other feature vectors based on the similarity between the feature vector and all other feature vectors.

[0098] In this embodiment, the terminal calculates the cosine similarity between each pair of feature vectors and the loss function between each pair of feature vectors, and calculates the similarity between each pair of feature vectors based on the cosine similarity between each pair of feature vectors and the loss function between each pair of feature vectors.

[0099] The formula for calculating cosine similarity is as follows:

[0100]

[0101] In the above formula, Let i be the i-th feature vector in the text information. Let i be the j-th feature vector in the text information, where i ≠ j, and i and j are both virtual numbers of the feature vector.

[0102] The loss function is based on the InfoNCE loss definition for contrastive learning, and its calculation formula is as follows:

[0103]

[0104] In the above formula, τ is a temperature coefficient used to adjust the shape of the similarity distribution. The larger τ is, the smoother the distribution; the smaller τ is, the more undulating the distribution. Let i be the i-th feature vector in the text information. Let i be the j-th feature vector in the text information, where i ≠ j, and i and j are both virtual numbers of the feature vector.

[0105] Then, for each feature vector, the terminal sums the similarities between the feature vector and all other feature vectors and divides the sum by the number of all feature vectors to obtain the average similarity between each feature vector and all other feature vectors.

[0106] Based on the above scheme, the similarity between two feature vectors is obtained by calculating cosine similarity and loss function, which improves the accuracy of similarity calculation.

[0107] Optionally, identifying the text category corresponding to each text information includes: obtaining sample text categories and extracting category feature information for each sample text category; identifying the text content corresponding to each category feature information and calculating the similarity distance between the target keywords of each text information and the text content corresponding to each category feature information; for each text information, selecting the sample text category of the category feature information corresponding to the minimum similarity distance as the text category of the text information.

[0108] In this embodiment, the terminal acquires the sample text categories and extracts the category feature information for each sample text category. The extraction of category feature information involves recognizing the text information of the sample text category, and then using a text feature recognition network to identify the feature information of the text information of that sample text category, thus obtaining the category feature information.

[0109] The terminal identifies the text content corresponding to each category of feature information and calculates the similarity distance between the target keywords of each text information and the text content corresponding to each category of feature information. The algorithm for calculating this similarity distance can be Euclidean distance or Mahalanobis distance, etc.

[0110] For each piece of text information, the terminal selects the sample text category corresponding to the category feature information with the minimum similarity distance, and uses it as the text category of the text information.

[0111] Based on the above scheme, by extracting the category feature information of the sample text categories, the similarity distance between each text information and the text category is identified, thereby filtering the text category corresponding to the text information and improving the accuracy of the identified text category.

[0112] Optionally, the distribution probability between each text representation and the label representation is calculated separately, including: identifying the position information of each text representation in the text information corresponding to each text representation, and calculating the similarity score between each text representation and each label representation using a scoring function based on the position information corresponding to each text representation and each label representation; and calculating the distribution probability between each text representation and the label representation of the text category of each text representation using a distribution probability algorithm based on the similarity score between each text representation, the label representation of the text category of the text information corresponding to each text representation, and the similarity score between each text representation and each label representation.

[0113] In this embodiment, the terminal identifies the location information within the text information corresponding to each text representation information. This location information is denoted as T. Then, based on the location information corresponding to each text representation information and each tag representation information, the terminal calculates a similarity score between each text representation information and each tag representation information using a scoring function. The formula for calculating the similarity score between each text representation information and each tag representation information obtained by the scoring function S is as follows:

[0114]

[0115] In the above formula, T represents the location information of each text representation information, and C represents the location information of each text representation information. l Let S be the label representation information, S be the scoring function, and l be the set of all label representation information.

[0116] The terminal calculates the probability distribution between each text representation and the tag representation of the text category corresponding to each text representation based on the similarity score between each text representation and each tag representation, using a probability distribution algorithm. The formula for calculating the probability distribution is as follows:

[0117]

[0118] In the above formula, L represents the number of all text categories. The similarity score between each text representation and the label representation of the text category corresponding to each text representation. The similarity score between each text representation and each label representation.

[0119] Based on the above scheme, by calculating the similarity score, the distribution probability between each text representation information and the label representation information is calculated, thereby improving the accuracy of the calculated distribution probability between each text representation information and the label representation information.

[0120] Optionally, after filtering the text representation information with the highest probability distribution as the keyword definition information for the text category, the method further includes: in response to the user's keyword definition information upload operation, obtaining the actual keyword definition information for each text category, and for each text category, identifying the keyword definition information of the text category and the deviation information between the actual keyword definition information of the text category; calculating the ratio between the deviation information and the keyword definition information, and if the ratio is greater than a preset ratio threshold, adjusting the scoring parameters of the scoring function for the text category based on the deviation information to obtain a new scoring function.

[0121] In this embodiment, the terminal responds to the user's keyword definition information upload operation by obtaining the actual keyword definition information for each text category. Specifically, the actual keyword definition information for each text category is the target text category selected by the staff from all text categories, and then the staff manually defines the keywords for each text category. Subsequently, the terminal receives the actual keyword definition information for each target text category uploaded to the terminal by the staff.

[0122] For each text category, the terminal identifies the keyword definition information for that text category and the deviation information between it and the actual keyword definition information for that text category. Then, the terminal calculates the ratio between the deviation information and the keyword definition information. The algorithm for this ratio may include, but is not limited to, the ratio between the number of text fields corresponding to the deviation information and the number of text fields corresponding to all keyword definitions. Next, the terminal presets a ratio threshold. If this ratio exceeds the preset threshold, the terminal adjusts the scoring function's scoring parameters for the text category based on the deviation information, resulting in a new scoring function. Specifically, the terminal identifies the ratio between the parameter change value corresponding to each scoring parameter of the scoring function and the number of deviation text fields in the training database of the scoring function. Then, based on the number of text fields in the deviation information, the terminal identifies the corresponding scoring parameter change value. Finally, the terminal adds this scoring parameter change value to the scoring function's scoring parameters for that text category, resulting in new scoring parameters.

[0123] Based on the above scheme, the scoring parameters are adjusted by using the actual keyword information uploaded by users, thereby improving the accuracy of the similarity score between each text representation information and each tag representation information calculated by the scoring parameters.

[0124] This application also provides a keyword definition example, such as Figure 2 As shown, the specific processing procedure includes the following steps:

[0125] Step S201: Obtain multiple text information that need to be classified and defined.

[0126] Step S202: For each piece of text information, the text information is divided according to the segments contained in the text information to obtain multiple text segments, and the semantic features of each text segment are identified through a text feature recognition network.

[0127] Step S203: Calculate the similarity between each word information in each text segment and the semantic features of each text segment, and select the word information corresponding to the maximum similarity in each text segment as the sub-keywords of the text segment.

[0128] Step S204: Use the sub-keywords of all text segments as keywords for the text information.

[0129] Step S205: For each piece of text information, for each keyword, identify the number of keywords in the text information, and identify the total number of words in the text information.

[0130] Step S206: Calculate the ratio of the number of keywords to the total number of words in the text information to obtain the distribution probability of keywords in the text information.

[0131] Step S207: Based on the probability distribution, filter the target keywords of the text information.

[0132] Step S208: For each piece of text information, generate a feature vector based on the target keywords of the text information and the probability distribution of each target keyword.

[0133] Step S209: Calculate the cosine similarity between each pair of feature vectors and the loss function between each pair of feature vectors, and calculate the similarity between each pair of feature vectors based on the cosine similarity between each pair of feature vectors and the loss function between each pair of feature vectors.

[0134] Step S210: For each feature vector, calculate the average similarity between each feature vector and other feature vectors based on the similarity between the feature vector and other feature vectors.

[0135] Step S211: The feature vector corresponding to the maximum average similarity is used as the text representation information of the text information.

[0136] Step S212: Obtain the sample text categories and extract the category feature information for each sample text category.

[0137] Step S213: Identify the text content corresponding to each category feature information, and calculate the similarity distance between the target keywords of each text information and the text content corresponding to each category feature information.

[0138] Step S214: For each piece of text information, select the sample text category of the category feature information corresponding to the minimum similarity distance, and use it as the text category of the text information.

[0139] Step S215: For each text category, calculate the label representation information of the text category based on the text representation information of each text information corresponding to the text category.

[0140] Step S216: Identify the position information of each text representation information in the text information corresponding to each text representation information, and calculate the similarity score between each text representation information and each label representation information based on the position information corresponding to each text representation information and each label representation information through a scoring function.

[0141] Step S217: Based on the similarity score between each text representation information, the label representation information of the text category of the text information corresponding to each text representation information, and the similarity score between each text representation information and each label representation information, the distribution probability between each text representation information and the label representation information of the text category of the text information corresponding to each text representation information is calculated using a distribution probability algorithm.

[0142] Step S218: Filter the text representation information with the highest distribution probability as the keyword definition information for the text category.

[0143] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0144] Based on the same inventive concept, this application also provides a keyword definition apparatus for implementing the keyword definition method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more keyword definition apparatus embodiments provided below can be found in the limitations of the keyword definition method described above, and will not be repeated here.

[0145] In one embodiment, such as Figure 3 As shown, a keyword definition device is provided, including: an acquisition module 310, an analysis module 320, a generation module 330, an identification module 340, a calculation module 350, and a filtering module 360, wherein:

[0146] The acquisition module 310 is used to acquire multiple text information that need to be classified and defined, and to extract keywords from each text information;

[0147] Analysis module 320 is used to analyze the distribution probability of keywords in each text message for each text message, and to filter target keywords in the text message based on the distribution probability;

[0148] The generation module 330 is used to generate a feature vector for each text information based on the target keywords of the text information and the distribution probability of each target keyword, and to calculate the average similarity between each feature vector and other feature vectors respectively.

[0149] The recognition module 340 is used to take the feature vector corresponding to the maximum average similarity as the text representation information of the text information, and to identify the text category corresponding to each text information.

[0150] The calculation module 350 is used to calculate the tag representation information of the text category for each text category based on the text representation information of each text information corresponding to the text category, and to calculate the distribution probability between each text representation information and the tag representation information respectively;

[0151] The filtering module 360 ​​is used to filter text representation information with the highest probability distribution as keyword definition information for the text category.

[0152] Optionally, the acquisition module 310 is specifically used for:

[0153] For each piece of text information, the text information is divided according to the segments contained in the text information to obtain multiple text segments, and the semantic features of each text segment are identified through a text feature recognition network;

[0154] Calculate the similarity between each word in each text segment and the semantic features of each text segment, and select the word information corresponding to the maximum similarity in each text segment as the sub-keywords of the text segment;

[0155] All sub-keywords of the text segments are used as keywords for the text information.

[0156] Optionally, the analysis module 320 is specifically used for:

[0157] For each keyword, identify the number of times the keyword appears in the text information, and identify the total number of words in the text information;

[0158] The ratio of the number of keywords to the total number of words in the text is calculated to obtain the probability distribution of the keywords in the text information.

[0159] Optionally, the generation module 330 is specifically used for:

[0160] Calculate the cosine similarity between each pair of feature vectors and the loss function between each pair of feature vectors respectively, and calculate the similarity between each pair of feature vectors based on the cosine similarity between each pair of feature vectors and the loss function between each pair of feature vectors.

[0161] For each feature vector, the average similarity between each feature vector and the other feature vectors is calculated based on the similarity between the feature vector and the other feature vectors.

[0162] Optionally, the identification module 340 is specifically used for:

[0163] Obtain the sample text categories and extract the category feature information for each sample text category;

[0164] Identify the text content corresponding to each category feature information, and calculate the similarity distance between the target keywords of each text information and the text content corresponding to each category feature information;

[0165] For each piece of text information, the sample text category to which the category feature information corresponding to the minimum similarity distance belongs is selected as the text category of the text information.

[0166] Optionally, the computing module 350 is specifically used for:

[0167] Identify the position information of each text representation information in the text information corresponding to each text representation information, and calculate the similarity score between each text representation information and each tag representation information based on the position information corresponding to each text representation information and the tag representation information through a scoring function;

[0168] Based on the similarity score between each text representation information, the label representation information of the text category corresponding to each text representation information, and the similarity score between each text representation information and each of the aforementioned label representation information, the distribution probability between each text representation information and the label representation information of the text category corresponding to each text representation information is calculated using a distribution probability algorithm.

[0169] Optionally, the device further includes:

[0170] The response module is used to respond to the user's keyword definition information upload operation, obtain the actual keyword definition information of each text category, and for each text category, identify the keyword definition information of the text category and the deviation information between the keyword definition information of the text category and the actual keyword definition information of the text category;

[0171] An adjustment module is used to calculate the ratio between the deviation information and the keyword definition information, and if the ratio is greater than a preset ratio threshold, adjust the scoring parameters of the scoring function for the text category based on the deviation information to obtain a new scoring function.

[0172] The modules in the aforementioned keyword definition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0173] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a keyword definition method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0174] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0175] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program.

[0176] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0177] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0178] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0179] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0180] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0181] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A keyword definition method, characterized in that, The method includes: Obtain multiple text messages that need to be categorized and defined, and extract keywords from each text message; For each piece of text information, the distribution probability of keywords in the text information is analyzed, and target keywords of the text information are filtered based on the distribution probability, wherein the distribution probability is the ratio of the number of keywords to the total number of words in the text information; For each piece of text information, a feature vector is generated based on the target keywords of the text information and the probability distribution of each target keyword, and the average similarity between the feature vector of each piece of text information and the feature vectors of other text information is calculated respectively. The feature vector corresponding to the maximum average similarity is used as the text representation information of the text information, and the text category corresponding to each text information is identified; For each text category, based on the text representation information of each text information corresponding to the text category, the tag representation information of the text category is calculated, and the distribution probability between each text representation information and the tag representation information is calculated respectively. The distribution probability is calculated by using a scoring function based on the position information of each text representation information and each tag representation information, and is then calculated by using a distribution probability algorithm based on the similarity score. The text representation information with the highest probability distribution is selected as the keyword definition information for the text category.

2. The method according to claim 1, characterized in that, The keywords extracted from each text message include: For each piece of text information, the text information is divided according to the segments contained in the text information to obtain multiple text segments, and the semantic features of each text segment are identified through a text feature recognition network; Calculate the similarity between each word in each text segment and the semantic features of each text segment, and select the word information corresponding to the maximum similarity in each text segment as the sub-keywords of the text segment; All sub-keywords of the text segments are used as keywords for the text information.

3. The method according to claim 1, characterized in that, The analysis of the distribution probability of keywords in each text message includes: For each keyword, identify the number of times the keyword appears in the text information, and identify the total number of words in the text information; The ratio of the number of keywords to the total number of words in the text is calculated to obtain the probability distribution of the keywords in the text information.

4. The method according to claim 1, characterized in that, The step of calculating the average similarity between each feature vector and all other feature vectors includes: Calculate the cosine similarity between each pair of feature vectors and the loss function between each pair of feature vectors respectively, and calculate the similarity between each pair of feature vectors based on the cosine similarity between each pair of feature vectors and the loss function between each pair of feature vectors. For each feature vector, the average similarity between each feature vector and the other feature vectors is calculated based on the similarity between the feature vector and the other feature vectors.

5. The method according to claim 1, characterized in that, The identification of the text category corresponding to each text information includes: Obtain the sample text categories and extract the category feature information for each sample text category; Identify the text content corresponding to each category feature information, and calculate the similarity distance between the target keywords of each text information and the text content corresponding to each category feature information; For each piece of text information, the sample text category to which the category feature information corresponding to the minimum similarity distance belongs is selected as the text category of the text information.

6. The method according to claim 1, characterized in that, The step of calculating the probability distribution between each text representation and the tag representation includes: Identify the position information of each text representation information in the text information corresponding to each text representation information, and calculate the similarity score between each text representation information and each tag representation information based on the position information corresponding to each text representation information and the tag representation information through a scoring function; Based on the similarity score between each text representation information, the label representation information of the text category corresponding to each text representation information, and the similarity score between each text representation information and each of the aforementioned label representation information, the distribution probability between each text representation information and the label representation information of the text category corresponding to each text representation information is calculated using a distribution probability algorithm.

7. The method according to claim 6, characterized in that, After filtering the text representation information with the highest probability distribution as the keyword definition information for the text category, the method further includes: In response to the user's keyword definition information upload operation, the actual keyword definition information of each text category is obtained, and for each text category, the keyword definition information of the text category and the deviation information between the keyword definition information of the text category and the actual keyword definition information of the text category are identified; Calculate the ratio between the deviation information and the keyword definition information, and if the ratio is greater than a preset ratio threshold, adjust the scoring parameters of the scoring function for the text category based on the deviation information to obtain a new scoring function.

8. A keyword definition device, characterized in that, The device includes: The acquisition module is used to acquire multiple text messages that need to be categorized and defined, and to extract keywords from each text message. The analysis module is used to analyze the distribution probability of keywords in each text message for each text message, and to filter target keywords in the text message based on the distribution probability, wherein the distribution probability is the ratio of the number of keywords to the total number of words in the text message; The generation module is used to generate a feature vector for each text information based on the target keywords of the text information and the distribution probability of each target keyword, and to calculate the average similarity between the feature vector of each text information and the feature vectors of other text information respectively. The recognition module is used to take the feature vector corresponding to the maximum average similarity as the text representation information of the text information, and to identify the text category corresponding to each text information. The calculation module is used to calculate the tag representation information of the text category for each text category based on the text representation information of each text information corresponding to the text category, and to calculate the distribution probability between each text representation information and the tag representation information respectively. The distribution probability is calculated by a scoring function based on the position information of each text representation information and each tag representation information, and is calculated by a distribution probability algorithm based on the similarity score value. The filtering module is used to filter text representation information with the highest probability distribution as keyword definition information for the text category.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images