Keyword category identification method and apparatus, and electronic device

Abstract

The present invention provides a keyword category identification method and apparatus, and an electronic device. The method comprises: acquiring text information to be identified; and inputting the text information into a target model, and outputting a keyword category to which each element in the text information belongs, wherein the target model comprises a first processing layer used for extracting semantic features of the text information, a second processing layer used for extracting, from the text information, a target keyword existing in a service dictionary, and a third processing layer used for determining, on the basis of the semantic features and the target keyword, the keyword category to which each element in the text information belongs, the first processing layer and the second processing layer are arranged in parallel, and the service dictionary comprises a plurality of keywords identified in a service process.

Description

Method, device and electronic device for identifying keyword categories

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the application with CN application number 202310800741.7 and application date June 30, 2023, and claims its priority. The disclosed content of the CN application is hereby introduced as a whole into this application. Technical Field

[0003] The present disclosure relates to the field of communication technologies, and in particular to a method, device, and electronic device for identifying keyword categories. Background Art

[0004] Communication Assistant is a value-added service application developed for communication assistant business users in the mobile Internet era. It provides functions such as missed call inquiry, address book backup, SMS backup, schedule reminder, private space, blacklist and whitelist, etc.

[0005] At present, the communication assistant services of various operators have become increasingly mature, and market competition has become increasingly fierce. Therefore, communication assistants need to achieve semantic understanding in business, while helping users extract important information and provide reminders.

[0006] In the communication assistant service, extracting key information from a call is crucial. If the categories of each element in the text information corresponding to the call content can be determined before extracting the keywords from the call content, the keywords in the text information can be extracted more accurately.

[0007] Summary of the Invention

[0008] In a first aspect, an embodiment of the present disclosure provides a method for identifying keyword categories, the method comprising: obtaining text information to be identified; and inputting the text information into a target model, and outputting the keyword category to which each element in the text information belongs; wherein the target model comprises a first processing layer for extracting semantic features of the text information, a second processing layer for extracting target keywords present in a business dictionary from the text information, and a third processing layer for determining the keyword category to which each element in the text information belongs based on the semantic features and the target keywords, the first processing layer and the second processing layer being arranged in parallel; the business dictionary comprises a plurality of keywords that have been identified during a business process.

[0009] In a second aspect, an embodiment of the present disclosure provides a device for identifying keyword categories, the device comprising: a first acquisition module for acquiring text information to be identified; and an identification module for inputting the text information into a target model and outputting the keyword category to which each element in the text information belongs; wherein the target model comprises a first processing layer for extracting semantic features of the text information, a second processing layer for extracting target keywords existing in a business dictionary from the text information, and a third processing layer for determining the keyword category to which each element in the text information belongs based on the semantic features and the target keywords, the first processing layer and the second processing layer being arranged in parallel; the business dictionary includes multiple keywords that have been identified in the business process.

[0010] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising a memory, a transceiver, and a processor: the memory is used to store a computer program; the transceiver is used to send and receive data under the control of the processor; and the processor is used to read the computer program in the memory and execute the method for identifying keyword categories described in the first aspect above.

[0011] In a fourth aspect, an embodiment of the present disclosure provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, and the computer program is used to enable a processor to execute the method for identifying keyword categories described in the first aspect.

[0012] In a fifth aspect, an embodiment of the present disclosure provides a computer program, comprising: instructions, which, when executed by a processor, cause the processor to execute the method for identifying keyword categories described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0014] FIG1 is a flow chart of a method for identifying keyword categories provided by an embodiment of the present disclosure;

[0015] FIG2 is a schematic diagram of a mask operation during pre-training in the related art;

[0016] FIG3 is a schematic diagram of a MASK operation during pre-training in an embodiment of the present disclosure;

[0017] FIG4 is a schematic diagram of the architecture of a target model in a specific implementation of the method for identifying keyword categories according to an embodiment of the present disclosure;

[0018] FIG5 is a structural block diagram of an apparatus for identifying keyword categories provided by an embodiment of the present disclosure;

[0019] FIG6 is a structural block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] In the embodiments of the present disclosure, the term "and / or" describes the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0021] In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0022] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] The inventors of the present disclosure discovered that in related technologies, identifying the category of each element based solely on the semantic features of text information may lead to misidentification in different business scenarios of communication assistants, resulting in incorrect extraction of keywords and thus reducing the business experience.

[0024] In view of this, an embodiment of the present disclosure provides a method for identifying keyword categories to improve the accuracy of extracting keywords from text information.

[0025] In a first aspect, an embodiment of the present disclosure provides a method for identifying keyword categories. As shown in FIG1 , the method includes the following steps 101 to 102 .

[0026] Step 101: Acquire text information to be recognized.

[0027] In some embodiments, the text information may be text information corresponding to the call content, such as text content converted from the call content; or, the text information may be text information extracted from a video file or other audio files other than the call content; or, the text information may be directly obtained text content, such as content in a document.

[0028] Step 102: Input the text information into a target model, and output the keyword category to which each element in the text information belongs.

[0029] Among them, the target model includes a first processing layer for extracting the semantic features of the text information, a second processing layer for extracting target keywords existing in the business dictionary from the text information, and a third processing layer for determining the keyword category to which each element in the text information belongs based on the semantic features and the target keywords. The first processing layer and the second processing layer are arranged in parallel; the business dictionary includes multiple keywords that have been identified in the business process.

[0030] That is, the text information is input into the target model, the semantic features of the text information are obtained through the first processing layer, the target keywords existing in the business dictionary are extracted from the text information through the second processing layer, and then the obtained semantic features and target keywords are respectively input into the third processing layer, so that the category of the keyword to which each element in the text information belongs is output through the third processing layer.

[0031] As can be seen, in the embodiments of the present disclosure, keywords identified during business processes can be stored in a business dictionary, allowing the target model to extract keywords from text information based on the business dictionary. This, combined with the semantic features of the text information, can more accurately identify the keyword category to which each element in the text information belongs.

[0032] In some embodiments, the keyword category includes at least one of time keywords, location keywords, person keywords, and event keywords. It is understandable that the keyword category may also include other categories besides the categories listed here.

[0033] In addition, after obtaining the keyword category to which each element in the text information belongs in step 102, keywords can be extracted from the text information according to the keyword category to which each element belongs, thereby making the extracted keywords more accurate.

[0034] In addition, the above-mentioned business dictionary can be provided by the business party, or it can be composed of keywords with relatively high confidence and support collected by data backflow after the target model is actually deployed and launched.

[0035] It can be seen from the above steps 101 to 102 that in the embodiment of the present disclosure, the text information to be identified can be obtained, so that the text information is input into the target model, and the keyword category to which each element in the text information belongs is output, wherein the target model includes a first processing layer for extracting the semantic features of the text information, a second processing layer for extracting target keywords existing in the business dictionary in the text information, and a third processing layer for determining the keyword category to which each element in the text information belongs based on the semantic features and the target keywords, and the first processing layer and the second processing layer are set in parallel; the business dictionary includes multiple keywords that have been identified in the business process.

[0036] It can be seen that in the embodiment of the present disclosure, the above-mentioned target model adds a second processing layer for extracting keywords in the business dictionary on the basis of the first processing layer for extracting the semantic features of text information. In this way, the target model can combine the semantic features and the keywords that have been identified in the business process to determine the keyword category to which each element in the text information belongs, making full use of the existing keyword business knowledge, thereby improving the accuracy of determining the keyword category to which each element in the text information belongs, and further improving the accuracy of extracting keywords from the text information.

[0037] In some embodiments, the first processing layer includes a word representation layer, a segment representation layer, a position representation layer and a multi-attention network layer, the word representation layer, the segment representation layer and the position representation layer are arranged in parallel, and the outputs of the word representation layer, the segment representation layer and the position representation layer are respectively input into the multi-attention network layer.

[0038] From this, we can see that after the text information is input into the first processing layer, it is processed by the word representation layer, segment representation layer and position representation layer respectively to obtain word vectors, segment vectors and position vectors respectively. Then the word vectors, segment vectors and position vectors are input into the multi-attention network layer for processing and the text semantic vector is output.

[0039] Here, the token embedding layer converts the elements of the input text information into fixed-dimensional word vectors. For example, for English, a word is broken down into roots and affixes, and then each root and affix is ​​converted into a fixed-dimensional vector. For Chinese, each character is converted into a fixed-dimensional vector. The vector obtained here is the word vector.

[0040] The segment embedding layer is used to distinguish the order of sentences in the text and convert them into corresponding vectors. The vector obtained here is the segment vector.

[0041] The Position Embedding layer is used to encode the position information of elements in text information into a feature vector. The vector obtained by conversion here is the position vector.

[0042] The multi-attention network layer can fuse multiple features. In the embodiment of the present disclosure, the three semantic features extracted by the word representation layer, the segment representation layer and the position representation layer can be fused to obtain a text semantic vector that can more accurately represent the semantic features of the text information.

[0043] In addition, the dimension of the second processing layer can be the same as that of the Token Embedding layer, and the initialization parameters are also the same as those of the Token Embedding layer; and the parameters of the Keyword Embedding layer can also be set to be trainable.

[0044] In some embodiments, the method further includes the following steps A1 to A3.

[0045] Step A1: Obtain pre-training text information.

[0046] Step A2: replacing the keywords in the business dictionary in the pre-trained text information with masks (ie, performing a MASK operation) to obtain pre-trained input text.

[0047] Step A3: Pre-training the first processing layer according to the pre-training input text.

[0048] For example, before the above target model is used, the above pre-training input text can be used to pre-train the first processing layer.

[0049] For example, if the pre-trained text message is "Hello, I am AB takeaway, please come down to pick up your delivery", and the keyword in the business dictionary is "AB takeaway", then "AB takeaway" can be replaced with the mask <mask>, that is, "Hello, I am <mask> <mask> <mask> <mask>, please come down to pick up your express", then "Hello, I am <mask> <mask> <mask> <mask>, please come down to pick up your delivery" is the pre-training input text.

[0050] In the related art MLM task (i.e., the task of predicting a central word by specifying surrounding words), the mask matrix of the self-attention layer of the multi-attention setting based on the Bidirectional Encoder Representation Transformers (BERT) is randomly generated, as shown in Figure 2. That is, during MLM pre-training, the BERT model in the related art randomly masks some words (replacing them with a unified marker [MASK]) and then trains the bidirectional language model by predicting these masked words. Furthermore, the representation of each word refers to contextual information. This makes the model unable to improve the targeted keyword sequence prediction task.

[0051] The pre-training method in the embodiment of the present disclosure makes special restrictions on the MASK matrix of the first processing layer, and only performs MASK operations on the keyword part of the pre-training input text that hits the business dictionary, as shown in Figure 3, so that the first processing layer can make predictions for the keyword sequence.

[0052] It can be seen from this that compared with the related art, in the pre-training process in the embodiment of the present disclosure, the content of the MASK operation is targeted, and the time required to obtain the pre-training input text is shorter. In this way, within the same time, the pre-training method in the embodiment of the present disclosure can be used to process more pre-training text information, thereby obtaining more pre-training input text, thereby further improving the accuracy of the first processing layer.

[0053] Optionally, step A3 "pre-training the first processing layer according to the pre-training input text" includes the following steps B1 to B3.

[0054] Step B1: Input the pre-trained input text into the first processing layer, and output the prediction result of the mask.

[0055] Step B2: Calculate the cross entropy between the prediction results corresponding to the same pre-trained input text and the replaced keywords.

[0056] Step B3: Adjust the parameters of the first processing layer according to the cross entropy.

[0057] After obtaining multiple pre-trained input texts according to steps A1 to A2 above, the pre-trained texts can be input into the first processing layer to obtain the output of the first processing layer (i.e., the content after the mask in the pre-trained input text is replaced by the prediction result). In this way, the parameters of the first processing layer can be adjusted based on the output of the first processing layer and the pre-trained text information before the MASK operation. For example, the cross entropy between the two can be calculated, and the parameters of the first processing layer can be adjusted based on the cross entropy.

[0058] However, the output of the first processing layer here (that is, the content after the mask in the pre-trained input text is replaced by the prediction result) has repeated content with the pre-trained text information before the MASK operation (for example, the aforementioned example: Hello, I am, please come down to pick up your express delivery). These repeated contents do not need to be calculated again for cross entropy. Therefore, in order to simplify the calculation process, only the cross entropy between the prediction result of the mask output by the first processing layer and the keyword replaced by the mask can be calculated.

[0059] In addition, the cross entropy between the prediction result of the mask output by the first processing layer and the keyword replaced by the mask is calculated, that is, the loss function value of the two is calculated and the gradient is returned, so that the output of the first processing layer at the position of the mask can predict the "keyword replaced by the mask", such as in the above example: "Hello, I am <mask> <mask> <mask> <mask>, please come down to pick up your express", can be in " <mask> <mask> <mask> <mask>” is predicted at the position of “AB Takeaway”.

[0060] Optionally, the above step 102 of “inputting the text information into the target model and outputting the keyword category to which each element in the text information belongs” includes the following steps C1 to C3.

[0061] Step C1: input the text information into the first processing layer, and output a text semantic vector for representing the semantic features of the text information.

[0062] Step C2: Input the text information into the second processing layer, extract the target keywords through the second processing layer, concatenate the target keywords into a target length to obtain target information, and convert the target information into an embedding vector, wherein the target length is the length of the text information.

[0063] Step C3: Input the text semantic vector and the embedding vector into the third processing layer respectively, and output the keyword category to which each element in the text information belongs.

[0064] In step C2, for example, the text message is "Hello, I am AB takeout, please come down to pick up your express.", and the target keyword that hits the business dictionary is "AB takeout", then the target keyword can be padded to the same length as the text message, that is: AB takeout [PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD]; and then the target information obtained after padding is converted into an Embeding vector.

[0065] It can be seen that through steps C1 to C3, the target keywords in the text information that hit the business dictionary are combined with the text semantic vector that represents the semantic features of the text information, so that the keyword category to which each element in the text information belongs can be obtained more accurately.

[0066] In some embodiments, the third processing layer includes at least one standard fully connected layer and a normalization processing layer;

[0067] Step C3, "inputting the text semantic vector and the embedding vector into the third processing layer respectively, and outputting the keyword category to which each element in the text information belongs", includes the following steps D1 to D4.

[0068] Step D1: Add the text semantic vector to the embedding vector to obtain a first vector;

[0069] Step D2: inputting the first vector into the at least one standard fully connected layer and outputting a second vector;

[0070] Step D3: inputting the second vector into the normalization processing layer, and outputting the probability (probs) distribution of each element in the text information belonging to a plurality of predetermined keyword categories;

[0071] Step D4: Outputting the keyword category to which each element in the text information belongs according to the probability distribution.

[0072] Among them, in step D4, the argmax operation can be performed on the probs distribution of each element in the text information belonging to a predetermined plurality of keyword categories (that is, in the probability distribution, the maximum probability value corresponding to each element is determined), so that the keyword category to which each element in the text information belongs can be obtained.

[0073] Here, argmax is a function that finds the parameter (set) of a function. When there is a function y=f(x), if the result x0=argmax(f(x)), it means that when the function f(x) takes x=x0, the maximum value of the value range of f(x) is obtained; if there are multiple points that make f(x) obtain the same maximum value, then the result of argmax(f(x)) is a point set. That is, argmax(f(x)) is the variable point x (or set of x) corresponding to the maximum value of f(x). Therefore, in the embodiment of the present disclosure, the argmax operation is performed on the probs distribution of each element in the text information belonging to a plurality of predetermined keyword categories, that is, to determine the maximum probability value corresponding to each element.

[0074] In addition, the standard fully connected layer plays the role of a classifier here, that is, after the first vector after the addition of the text semantic vector and the embedding vector is input into the standard fully connected layer, the various elements of the text information are divided into multiple keyword categories through the processing of the standard fully connected layer, and then through the processing of the normalization layer, the probability distribution of each element in the text information belonging to a predetermined multiple keyword categories can be obtained. In this way, in the probability distribution, the keyword category to which the maximum probability value corresponding to each element belongs is the keyword category to which the element belongs.

[0075] In addition, the at least one standard fully connected layer may, for example, include two first standard fully connected layers and one second standard fully connected layer, wherein the output dimension (outdim) of the first standard fully connected layer may be the same as the hidden layer size (hidden size) in the word representation layer described above, and the remaining parameters may be defaulted; the input dimension of the second standard fully connected layer may be the same as the hidden layer size (hidden size) in the word representation layer described above, and the output dimension is k*2+1, where k represents the number of predetermined keyword categories. Here, the "*2" in "k*2+1" is because a BIO structure is used, and each category has a distinction between begin and I; and the "+1" is because non-keyword text is marked as O.

[0076] In some embodiments, before inputting the text information into the target model, the method further includes the following steps E1 to E2.

[0077] Step E1: Acquire the domain information to which the text information belongs.

[0078] Step E2: Add the domain information to the text information.

[0079] Among them, any method in related technologies can be used to obtain the domain information to which the text information belongs.

[0080] In addition, the domain information can be added to the front end of the text information, and the two can be connected using a preset character (such as a period ".").

[0081] Furthermore, in scenarios like communication assistants, where the domain is highly segmented, keyword slot extraction involves multiple different domains, the number of categories is large, and long-tail data is heavily distributed, sequence annotation technology in related technologies is prone to misjudgment between domains due to the high semantic similarity of slot types within different scenarios. However, in the embodiments of the present disclosure, domain information can be integrated into text information. This information can then be combined with the keyword category to identify elements in the text information, thereby reducing the probability of misjudgment and improving keyword extraction accuracy.

[0082] It is understandable that domain information may also be incorporated into the pre-training process described above. For example, after step A1 and before step A2, domain information of the pre-trained text information is obtained and added to the pre-trained text information.

[0083] In some embodiments, the method further includes the following steps F1 to F2.

[0084] Step F1: using a sequence labeling method to label the elements in the training text information, and obtaining a label for each of the training text information.

[0085] Step F2: training the target model according to the training text information and the label of the training text information.

[0086] It should be noted that the target model training process here is different from the pre-training process described above. Pre-training is performed on the first processing layer. After pre-training the first processing layer, the second and third processing layers are added to the first processing layer to construct the target model, which is then trained.

[0087] Sequence labeling involves marking each element in a given sequence, or assigning a label to each element. Labeling uses the standard sequence labeling BIO structure (B-begin, I-inside, O-outside). For example, if the training text is "Hello, I'm AB Takeout. Please come down to pick up your delivery," the label would be: OOOOO B-Takeout Name I-Takeout Name I-Takeout Name OOOOOOOOOOOO.

[0088] In addition, the domain information of the training text information can also be added to the above training text information. For example, the above training text information is "Hello, I am AB takeout, please come down to pick up your express." After adding the domain information "takeout", the corresponding label is: OOOOOOOO B-takeout name I-takeout name I-takeout name OOOOOOOOOO.

[0089] It can be seen that in the training stage of the target model, a training sample includes training text information, domain information of the training text information, and labels.

[0090] In the above step F2, "training the target model according to the training text information and the label of the training text information" may specifically be:

[0091] The training text information is input into the target model to obtain the corresponding output. By comparing the output of the model with the label corresponding to the training text, it can be determined whether the keyword category to which each element in the training text information predicted by the target model belongs is accurate. Based on this, the parameters of the target model can be adjusted until the accuracy of the result output by the target model meets certain requirements, and the training of the target model is completed.

[0092] In summary, the specific implementation of the method for identifying keyword categories in the embodiment of the present disclosure can be described as follows:

[0093] (1) Pre-training stage:

[0094] Use the basic BERT intermediate layer architecture (i.e., Token Embedding layer, Segment Embedding layer, Position Embedding layer, and Multi-attention layer) as the first processing layer for pre-training, as described in steps 1.1 to 1.5 below.

[0095] 1.1 Incorporate domain information into one of the pre-trained text messages, for example: "Takeout. Hello, I'm AB Takeout. Please come down to pick up your delivery."

[0096] 1.2 Identify the location of the keywords in the business dictionary in the pre-trained text information after integrating domain information. For example, "Takeout. Hello, I am AB Takeout. Please come down to pick up your delivery." The keyword in the business dictionary is AB Takeout, and the index position is: [8,12].

[0097] 1.3 Change the position of keywords that hit the business dictionary to <mask>, get the pre-training input text, that is, get: "Takeaway. Hello, I am <mask> <mask> <mask> <mask>, please come down to pick up your delivery."

[0098] 1.4 Input the pre-trained input text into the first processing layer (i.e., input into the Token Embedding layer, Segment Embedding layer, Position Embedding layer respectively, and then input the output of these three layers into the Multi-attention layer), and output the <mask> <mask> <mask> <mask>The predicted result Y(y1, y2, y3, y4).

[0099] 1.5 Calculate the loss function value of Y(y1, y2, y3, y4) and the original text (A, B, takeout), and perform gradient backpropagation, so that the first processing layer is at <mask> <mask> <mask> <mask>The output can predict the four words "A, B, outside, takeout".

[0100] 1.6 Repeat the above steps 1.1 to 1.5 for all pre-trained text information to complete the pre-training process.

[0101] (2) Target model construction and training phase:

[0102] That is, after pre-training the first processing layer, on the basis of the first processing layer (i.e., Token Embedding layer, Segment Embedding layer, Position Embedding layer and Multi-attention layer), a Keyword Embedding layer (i.e., the second processing layer), two first standard fully connected layers, one second standard fully connected layer, a normalization processing layer and a result output layer are further added to obtain the target model shown in Figure 4.

[0103] After the target model is constructed, the constructed target model is trained. The training process is described in steps 2.1 to 2.3 below.

[0104] 2.1 Obtain training text information and integrate domain information into the training text information, for example: "Takeout. Hello, I am AB Takeout, please come down to pick up your delivery."

[0105] 2.2 Use the sequence labeling method to label the elements in the training text information and obtain the label of each training text information. For example, "Takeout. Hello, I am AB takeout, please come down to pick up your delivery." The obtained label is: OOOOO B-Takeout name I-Takeout name I-Takeout name OOOOOOOOOO.

[0106] 2.3 Input the training text information into the target model to obtain the corresponding output, and then compare the model output with the label corresponding to the training text to determine whether the keyword category to which each element in the training text information predicted by the target model belongs is accurate. Based on this, the parameters of the target model can be adjusted until the accuracy of the target model output result meets certain requirements, and the training of the target model is completed.

[0107] (3) Target model usage phase:

[0108] After obtaining the text information to be identified, the keyword category to which each element in the text information belongs is obtained through the following process of steps 3.1 to 3.5.

[0109] 31. Integrate domain information into text messages. For example, if the text message is "Hello, I am AB takeout, please come down to pick up your delivery.", after integrating the domain information, it will become: "Takeout. Hello, I am AB takeout, please come down to pick up your delivery."

[0110] 3.1: As shown in Figure 4, the text information after incorporating domain information is input into the target model's Token Embedding layer, Segment Embedding layer, Position Embedding layer, and Keyword Embedding layer respectively to obtain word vectors, segment vectors, and position vectors, respectively. These three vectors are then input into the Multi-attention layer to output the text semantic vector.

[0111] Among them, the Keyword Embedding layer extracts keywords that hit the business dictionary from the text information, then concatenates them to the same length as the text information, and then converts them into embedding vectors.

[0112] 3.2 After adding the text semantic vector and the embedding vector, the first vector is obtained.

[0113] 3.3 The first vector input is sequentially input into two first standard fully connected layers and one second standard fully connected layer, and the second vector is output.

[0114] 3.4 The second vector is input to the normalization processing layer, which outputs the probability distribution of each element in the text information belonging to multiple predetermined keyword categories.

[0115] 3.5. The result output layer outputs the keyword category to which each element in the text information belongs according to the probability distribution.

[0116] In summary, the embodiments of the present disclosure have the following advantages:

[0117] 1. Integrate the domain information of the text information to be identified into the text information and input it into the target model together, so as to facilitate the knowledge guidance of the subdivided domain in the target model and solve the problem of misjudgment of similar entities between domains as much as possible.

[0118] 2. In the target model, each input token (i.e., word embedding) is represented by adding its corresponding word vector, segment vector, position vector, and embedding vector of the matching business dictionary. Furthermore, a calculation method for fusing the vectors output by the Keyword Embedding and Multi-Attention layers is presented, integrating the Keyword Embedding layer with the first processing layer.

[0119] Among them, due to the particularity of the business, there are many repetitive keywords in calls, but in related technologies, there is no effective way to utilize the semantics of these organized and identified keywords.

[0120] As can be seen from the above, in order to solve this problem, in the embodiment of the present disclosure, a Keyword Embedding layer is set in the input layer, which can make full use of the existing keyword business knowledge, so that the existing knowledge can be maximized in training and subsequent operations.

[0121] 3. In the embodiments of the present disclosure, during the pre-training of the first processing layer, MAS operations are performed only on keywords that match the business dictionary. This allows semantic knowledge embedding training of specific keywords to be performed on large amounts of data, assigning these keywords specific embedding parameter values ​​to help improve model accuracy. Specifically, in the embodiments of the present disclosure, through the unique MASK method, certain specific keyword information can be incorporated into the conversational corpus, thereby improving model generalization and enhancing model prediction accuracy.

[0122] The above describes the method for identifying keyword categories provided by the embodiment of the present disclosure. The following describes the device for identifying keyword categories provided by the embodiment of the present disclosure with reference to the accompanying drawings.

[0123] 5 , an embodiment of the present disclosure further provides an apparatus for identifying keyword categories. As shown in FIG5 , the apparatus includes: a first acquisition module 501 and an identification module 502 .

[0124] The first acquisition module 501 is used to acquire text information to be recognized.

[0125] Identification module 502, configured to input the text information into a target model and output the keyword category to which each element in the text information belongs;

[0126] Among them, the target model includes a first processing layer for extracting the semantic features of the text information, a second processing layer for extracting target keywords existing in the business dictionary in the text information, and a third processing layer for determining the keyword category to which each element in the text information belongs based on the semantic features and the target keywords. The first processing layer and the second processing layer are set in parallel.

[0127] The business dictionary includes a plurality of keywords that have been identified in the business process.

[0128] In some embodiments, the first processing layer includes a word representation layer, a segment representation layer, a position representation layer and a multi-attention network layer, the word representation layer, the segment representation layer and the position representation layer are arranged in parallel, and the outputs of the word representation layer, the segment representation layer and the position representation layer are respectively input into the multi-attention network layer.

[0129] In some embodiments, the device also includes: a second acquisition module for acquiring pre-trained text information; a mask processing module for replacing keywords in the business dictionary in the pre-trained text information with masks to obtain pre-trained input text; and a pre-training module for pre-training the first processing layer based on the pre-trained input text.

[0130] In some embodiments, the pre-training module is specifically used to: input the pre-training input text into the first processing layer and output the prediction result of the mask; calculate the cross entropy between the prediction result corresponding to the same pre-training input text and the replaced keyword; and adjust the parameters of the first processing layer according to the cross entropy.

[0131] In some embodiments, the recognition module 502 includes: a first processing sub-module, used to input the text information into the first processing layer, and output a text semantic vector for characterizing the semantic features of the text information; a second processing sub-module, used to input the text information into the second processing layer, extract the target keywords through the second processing layer, splice the target keywords into a target length to obtain target information, and convert the target information into an embedding vector, wherein the target length is the length of the text information; and a third processing sub-module, used to input the text semantic vector and the embedding vector into the third processing layer respectively, and output the keyword category to which each element in the text information belongs.

[0132] In some embodiments, the third processing layer includes at least one standard fully connected layer and a normalization processing layer; the third processing submodule is specifically used to: add the text semantic vector to the embedding vector to obtain a first vector; input the first vector to the at least one standard fully connected layer and output a second vector; input the second vector to the normalization processing layer and output the probability distribution of each element in the text information belonging to a predetermined plurality of keyword categories; and output the keyword category to which each element in the text information belongs based on the probability distribution.

[0133] In some embodiments, the apparatus further includes: a third acquisition module, configured to acquire domain information to which the text information belongs; and a domain fusion module, configured to add the domain information to the text information.

[0134] In some embodiments, the apparatus further comprises: a labeling module configured to label elements in the training text information using a sequence labeling method to obtain a label for each of the training text information; and

[0135] A training module is used to train the target model based on the training text information and the label of the training text information.

[0136] In some embodiments, the keyword category includes at least one of a time keyword, a location keyword, a person keyword, and an event keyword.

[0137] Among them, the method and the device are based on the same application concept. Since the principles of solving problems by the method and the device are similar, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated.

[0138] It should be noted that the division of units in the embodiments of the present application is schematic and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0139] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0140] It should be noted here that the above-mentioned device provided in the embodiment of the present disclosure can implement all the method steps implemented in the above-mentioned method embodiment and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0141] An embodiment of the present disclosure further provides an electronic device, as shown in FIG6 , which includes a memory 620 , a transceiver 610 , and a processor 600 .

[0142] The memory 620 is used to store computer programs.

[0143] The transceiver 610 is configured to receive and send data under the control of the processor 600 .

[0144] The processor 600 is configured to read the computer program in the memory 620 and execute the method for identifying keyword categories described in the first aspect.

[0145] In FIG6 , the bus architecture may include any number of interconnected buses and bridges, specifically various circuits linked together by one or more processors represented by processor 600 and memory represented by memory 620. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 610 may be a plurality of components, i.e., a transmitter and a receiver, providing a unit for communicating with various other devices over a transmission medium, such as a wireless channel, a wired channel, an optical cable, and the like. The processor 600 is responsible for managing the bus architecture and general processing, and the memory 620 may store data used by the processor 600 when performing operations.

[0146] The processor 600 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD). The processor 600 may also adopt a multi-core architecture.

[0147] It should be noted here that the above-mentioned device provided in the embodiment of the present disclosure can implement all the method steps implemented in the above-mentioned method embodiment and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0148] In some embodiments of the present disclosure, a computer program is further provided, comprising: instructions, which, when executed by a processor, cause the processor to perform the aforementioned method for identifying keyword categories.

[0149] An embodiment of the present disclosure further provides a processor-readable storage medium storing a computer program configured to cause the processor to execute the above-described method for identifying keyword categories. For example, the processor-readable storage medium is a non-transitory processor-readable storage medium.

[0150] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO)), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROMs, EPROMs, EEPROMs, non-volatile memories (NAND FLASH), solid-state drives (SSDs)), etc.

[0151] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) that contain computer-usable program code.

[0152] The present application is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer-executable instructions. These computer-executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0153] These processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the processor-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0154] These processor-executable instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0155] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.< / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask>

Claims

1. A method for identifying keyword categories, comprising: Obtain text information to be recognized; and Inputting the text information into a target model, and outputting the keyword category to which each element in the text information belongs; The target model includes a first processing layer for extracting semantic features of the text information, a second processing layer for extracting target keywords in the text information that exist in a business dictionary, and a third processing layer for determining the keyword category to which each element in the text information belongs based on the semantic features and the target keywords, and the first processing layer and the second processing layer are arranged in parallel; The business dictionary includes a plurality of keywords that have been identified in the business process.

2. The method according to claim 1, wherein: The first processing layer includes a word representation layer, a segment representation layer, a position representation layer and a multi-attention network layer. The word representation layer, the segment representation layer and the position representation layer are arranged in parallel, and the outputs of the word representation layer, the segment representation layer and the position representation layer are respectively input into the multi-attention network layer.

3. The method according to claim 1 or 2, further comprising: Get pre-trained text information; Replacing the keywords in the business dictionary in the pre-trained text information with masks to obtain pre-trained input text; and The first processing layer is pre-trained according to the pre-training input text.

4. The method according to claim 3, wherein: The pre-training of the first processing layer according to the pre-training input text comprises: Inputting the pre-trained input text into the first processing layer, and outputting a prediction result of the mask; Calculating the cross entropy between the prediction results corresponding to the same pre-trained input text and the replaced keywords; and According to the cross entropy, parameters of the first processing layer are adjusted.

5. The method according to any one of claims 1 to 4, wherein: The step of inputting the text information into a target model and outputting the keyword category to which each element in the text information belongs includes: Inputting the text information into the first processing layer, and outputting a text semantic vector for representing the semantic features of the text information; Inputting the text information into the second processing layer, extracting the target keywords through the second processing layer, concatenating the target keywords into a target length to obtain target information, and converting the target information into an embedding vector, wherein the target length is the length of the text information; and The text semantic vector and the embedding vector are respectively input into the third processing layer, and the keyword category to which each element in the text information belongs is output.

6. The method according to claim 5, wherein: The third processing layer includes at least one standard fully connected layer and a normalization processing layer; The step of inputting the text semantic vector and the embedding vector into the third processing layer respectively, and outputting the keyword category to which each element in the text information belongs, comprises: Adding the text semantic vector to the embedding vector to obtain a first vector; Input the first vector to the at least one standard fully connected layer and output a second vector; Inputting the second vector to the normalization processing layer, and outputting the probability distribution of each element in the text information belonging to a plurality of predetermined keyword categories; and According to the probability distribution, the keyword category to which each element in the text information belongs is output.

7. The method according to any one of claims 1 to 6, further comprising: Before inputting the text information into the target model, obtaining the domain information to which the text information belongs; and The domain information is added to the text information.

8. The method according to any one of claims 1 to 7, further comprising: Using a sequence labeling method, labeling the elements in the training text information to obtain a label for each of the training text information; and The target model is trained according to the training text information and the label of the training text information.

9. The method according to any one of claims 1 to 8, wherein: The keyword category includes at least one of a time keyword, a location keyword, a person keyword, and an event keyword.

10. A device for identifying keyword categories, comprising: A first acquisition module, used to acquire text information to be recognized; and A recognition module, used for inputting the text information into a target model and outputting the keyword category to which each element in the text information belongs; The target model includes a first processing layer for extracting semantic features of the text information, a second processing layer for extracting target keywords in the text information that exist in a business dictionary, and a third processing layer for determining the keyword category to which each element in the text information belongs based on the semantic features and the target keywords, and the first processing layer and the second processing layer are arranged in parallel; The business dictionary includes a plurality of keywords that have been identified in the business process.

11. An electronic device comprising a memory, a transceiver and a processor: The memory is used to store computer programs; the transceiver is used to send and receive data under the control of the processor; the processor is used to read the computer program in the memory and execute the method for identifying keyword categories as described in any one of claims 1 to 9. 12 . A processor-readable storage medium storing a computer program, wherein the computer program is used to cause a processor to execute the method for identifying keyword categories according to claim 1 .

13. A computer program comprising: The instructions, when executed by a processor, cause the processor to perform the method for identifying keyword categories as claimed in any one of claims 1 to 9.

Images