Model training, semantic dependency analysis method, device, equipment and storage medium

Abstract

The embodiments of the present invention provide a model training, semantic dependency analysis method, device, equipment and storage medium. The model training method includes: adding a concept type label to the concept words in the training sample sentences to obtain a concatenated sentence; performing word embedding processing on the concatenated sentence through a transformer coding layer, so that the non-concept words in the training sample sentences are not affected by the concept type label; performing label masking processing on the concatenated sentence after the word embedding processing and passing it through a semantic dependency analysis layer, so that the semantic analysis layer learns the semantics of the concept type label, and obtains a trained semantic dependency analysis model including the transformer coding layer and the semantic dependency analysis layer. The embodiments of the present invention can improve the scope of application of the semantic dependency analysis model to concept types.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a model training, semantic dependency analysis method, device, equipment and storage medium. Background Art

[0002] Semantic Dependency Parsing (SDP) is used to analyze the semantic associations between the various language units of a sentence and present the semantic associations in a dependency structure. The advantage of using semantic dependencies to describe the semantics of a sentence is that there is no need to abstract the words themselves, but to describe the words through the semantic framework that the words bear, and the number of arguments is much smaller than that of the words. Analyzing the semantic dependency of natural language plays an important role in tasks such as intent recognition, intelligent question answering, and reasoning.

[0003] Entity concepts are an important feature in semantic dependency analysis. The concept types vary greatly in different service scenarios. Current semantic dependency analysis technology can only handle a very limited number of concept types, so there is still room for improvement. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a model training, semantic dependency analysis method, device, equipment and storage medium to solve or alleviate the above problems.

[0005] According to a first aspect of an embodiment of the present invention, a model training method is provided, comprising: adding a concept type label to concept words in a training sample sentence to obtain a concatenated sentence; performing word embedding processing on the concatenated sentence through a transformer coding layer, so that non-concept words in the training sample sentence are not affected by the concept type label; performing label masking processing on the concatenated sentence after the word embedding processing and passing it through a semantic dependency analysis layer, so that the semantic analysis layer learns the semantics of the concept type label, and obtains a trained semantic dependency analysis model including the transformer coding layer and the semantic dependency analysis layer.

[0006] According to a second aspect of an embodiment of the present invention, a semantic dependency analysis method is provided, comprising: generating a concatenated sentence by concatenating concept type labels of concept words in a sentence to be analyzed into the sentence to be analyzed; inputting the concatenated sentence into a semantic dependency analysis model to obtain a semantic dependency relationship of the concatenated sentence, wherein the semantic dependency analysis model is trained by the model training method according to the first aspect; in the semantic dependency relationship of the concatenated sentence, deleting dependency relationship data related to the concept type label to obtain the semantic dependency relationship of the sentence to be analyzed.

[0007] According to a third aspect of an embodiment of the present invention, a model training device is provided, comprising: a concatenation module, for adding a concept type label to the concept words in a training sample sentence to obtain a concatenated sentence; a word embedding module, for performing word embedding processing on the concatenated sentence through a transformer coding layer, so that non-concept words in the training sample sentence are not affected by the concept type label; a training module, for performing label masking processing on the concatenated sentence after the word embedding processing and then passing it through a semantic dependency analysis layer, so that the semantic analysis layer learns the semantics of the concept type label, and obtains a trained semantic dependency analysis model including the transformer coding layer and the semantic dependency analysis layer.

[0008] According to a fourth aspect of an embodiment of the present invention, a semantic dependency analysis device is provided, comprising: a splicing module, which generates a spliced ​​sentence by splicing concept type labels of concept words in a sentence to be analyzed into the sentence to be analyzed; an analysis module, which inputs the spliced ​​sentence into a semantic dependency analysis model to obtain an initial semantic dependency relationship of the sentence to be analyzed, wherein the semantic dependency analysis model is trained by the model training method according to the first aspect; and a post-processing module, which deletes dependency data related to the concept type label in the semantic dependency relationship of the spliced ​​sentence to obtain the semantic dependency relationship of the sentence to be analyzed.

[0009] According to a fifth aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect or the second aspect.

[0010] According to a sixth aspect of an embodiment of the present invention, there is provided a storage medium on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect or the second aspect is implemented.

[0011] In the solution of the embodiment of the present invention, since the concept type label is masked for non-concept words other than the concept words in the training sample sentence, the non-concept words are not affected by the concept type label, thereby increasing the influence of the concept type label on the concept words in the transformer encoding layer. In addition, after the concatenated sentence after the word embedding process is subjected to label masking process, it passes through the semantic dependency analysis layer, so that the semantic analysis layer learns the semantics of the concept type label, avoiding the influence of the concept type label on the semantic analysis of the concept word, thereby improving the concept type application range of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0013] Figure 1 A schematic diagram of a model training method according to an embodiment of the present invention;

[0014] Figure 2 A schematic diagram of a semantic dependency analysis model of a model training method according to another embodiment of the present invention;

[0015] Figure 3A A schematic diagram of a mask matrix for concatenating sentences according to another embodiment of the present invention;

[0016] Figure 3B is a schematic diagram of a mathematical representation of a mask matrix according to another embodiment of the present invention;

[0017] Figure 4A A schematic diagram of a multi-head attention mechanism in a transformer according to another embodiment of the present invention;

[0018] Figure 4B A schematic diagram of a method for processing dot product attention in a transformer according to another embodiment of the present invention;

[0019] Figure 5 is a schematic flow chart of a speech dependency analysis method according to another embodiment of the present invention;

[0020] Figure 6 is a schematic block diagram of a model training device according to another embodiment of the present invention;

[0021] Figure 7 is a schematic block diagram of a speech dependency analysis device according to another embodiment of the present invention;

[0022] Figure 8 The present invention is another embodiment of the hardware structure of an electronic device. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in the field based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0024] The specific implementation of the embodiment of the present invention is further described below in conjunction with the accompanying drawings of the embodiment of the present invention. When performing semantic dependency analysis, the sentence to be analyzed can be input into a pre-trained semantic dependency analysis model to obtain the corresponding semantic dependency relationship. In the training process of the above-mentioned semantic dependency analysis model, the training sample sentences can be vector embedded (embedding) processed. In one example, each concept word in the training sample sentence can be word vector embedded. In addition, the embedding vectors of the part of speech or concept type corresponding to each concept word are respectively connected to the word vector of each concept word, and the vector embedding processing of the concept type is performed to obtain a concept label embedding vector, thereby integrating the information of the concept type of the concept word into the concept label embedding vector. Then, the obtained concept label embedding vector can be used to train the semantic dependency analysis model.

[0025] In the above scheme, since the embedding vectors of the part of speech or concept type corresponding to each concept word are respectively connected to the word vector of each concept word, and the vector embedding processing of the concept type is performed to obtain the concept label embedding vector, there is a corresponding relationship between each concept word and its concept type. If, when using the semantic dependency analysis model for semantic dependency analysis, if a concept type that was not used in the training process appears, the concept label embedding vector of the sentence to be analyzed is greatly different from the concept label embedding vector of the training sample sentence, the accuracy of the analysis result of the semantic dependency analysis will be very poor, and the semantic dependency analysis model can only be applied to the concept types in the training sample.

[0026] Figure 1 A schematic diagram of a model training method according to an embodiment of the present invention. Figure 1 The model training method can be performed by any appropriate electronic device with data processing capabilities, including but not limited to devices such as servers and PCs. The model training method includes:

[0027] 110: Add concept type labels to concept words in training sample sentences to obtain concatenated sentences.

[0028] It should be understood that when splicing the concept type label of the concept word into the training sample sentence, the concept type label can be added before the concept word, the concept type label can be added after the concept word, or the concept type label can be added at other positions in the training sample sentence, and the position association between the concept word and the concept type label is established. In addition, in the present application, the splicing of the concept type label refers to adding a concept type label vector based on the position of the concept word vector in the sentence, rather than connecting the concept type label vector to the concept word vector (for example, to become a longer vector).

[0029] It should also be understood that the concept type in the current training sample sentence may not have appeared in the previous training sentence, or may have appeared in the previous training sentence. In addition, in the case where the training sample sentence includes multiple concept words, concept type tags may be added to some of the concept words. Concept type tags may also be added to all concept words. The embodiment of the present invention does not limit this.

[0030] 120: Through the transformer encoding layer, the concatenated sentences are processed by word embedding, so that the non-concept words in the training sample sentences are not affected by the concept type label.

[0031] It should be understood that the transformer coding layer and the semantic dependency analysis layer constitute a semantic dependency analysis model. The transformer coding layer can be a structure based on a transformer encoder, for example, it can be a structure based on a bidirectional transformer encoder (Bidirectional Encoder Representation from Transformer, BERT). The semantic dependency analysis layer can be a transformer-based structure.

[0032] It should also be understood that when applying the semantic dependency analysis model for analysis, the sentences to be analyzed are also correspondingly subjected to concept type masking processing, and each sentence to be analyzed may correspond to a mask matrix, or is called a visualization matrix.

[0033] 130: After the word embedding processing, the concatenated sentences are subjected to label masking processing and then passed through the semantic dependency analysis layer, so that the semantic analysis layer learns the semantics of the concept type label, and a trained semantic dependency analysis model including a transformer encoding layer and a semantic dependency analysis layer is obtained.

[0034] In the solution of the embodiment of the present invention, since the concept type label is masked for non-concept words other than the concept words in the training sample sentence, the non-concept words are not affected by the concept type label, thereby increasing the influence of the concept type label on the concept words in the transformer encoding layer. In addition, after the concatenated sentence after the word embedding process is subjected to label masking process, it passes through the semantic dependency analysis layer, so that the semantic analysis layer learns the semantics of the concept type label, avoiding the influence of the concept type label on the semantic analysis of the concept word, thereby improving the concept type application range of the model.

[0035] As an example, the concept type labels of the concept words in the training sample sentences are concatenated into the training sample sentences to obtain the concatenated sentences. Based on the concatenated sentences as the input of the transformer coding layer, and based on the semantic dependency relationship of the concatenated sentences as the output of the semantic dependency analysis layer, the semantic dependency analysis model composed of the transformer coding layer and the transformer coding layer is trained. The output of the transformer coding layer is connected to the input of the semantic dependency analysis layer. The transformer coding layer is used to adopt the attention mechanism to perform label embedding processing on the concatenated sentences, and the attention weight of the concept type label for the concept words in the training sample sentences is higher than the attention weight for other words.

[0036] It should be understood that when a training sample sentence includes multiple concept words, for the attention mechanism in this article, preferably, the target concept word is completely invisible to the concept type labels of all other concept words, that is, the attention weight of the target concept word to the concept type label of any other concept word is set to 0. In addition, the target concept word is visible to the concept type label corresponding to itself, that is, the attention weight of the target concept word to its concept type label is set to 1.

[0037] It should also be understood that, generally, the attention weight of the target concept word to the concept type label of any other concept word can also be set to be smaller than the attention weight of the target concept word to its own corresponding concept type label.

[0038] In this example, the transformer encoding layer is used to adopt the attention mechanism to perform label embedding processing on the concatenated sentences. The attention weight of the concept type label for the concept words in the training sample sentences is higher than the attention weights for other words. Therefore, for the concatenated sentences, the attention weight of the concept type label for the concept words is greater than the attention weights for other words, so that in the sentences after label embedding processing, the concept type label has a greater impact on the concept words than on other words. Therefore, even if a new concept type appears, the impact of the concept type on other words in the sentence is small, thereby ensuring the effect of using the semantic dependency analysis model for semantic dependency analysis and improving the applicability of the semantic dependency analysis model to concept types.

[0039] The Transformer and the Bidirectional Encoder Representation from Transformer (BERT) are described below.

[0040] Specifically, the Transformer model uses a multi-head attention mechanism. Transformer is an encoder-decoder architecture. But its structure is more complex than the attention mechanism. Usually, the encoder layer can be composed of multiple (for example, 6) encoders stacked together. Similarly, the decoder layer can also use the same number of encoders. In addition, the encoder can contain two layers, a self-attention layer and a feedforward neural network layer. The self-attention layer can help the current node not only focus on the current word, but also obtain the semantics of the context. The decoder can also contain the two-layer network mentioned by the encoder, but there is also an attention layer in the middle of the two layers to help the current node obtain the key content that needs to be paid attention to.

[0041] In addition, BERT uses the Transformer encoder structure, which is a pre-training model with strong generalization ability. The advantage of pre-training is that it does not require a large amount of corpus for training when used in specific scenarios, saving time and efficiency. Usually, the input of BERT can be a single sentence or a sentence pair, and the actual input value is the sum of segment embedding and position embedding. Therefore, usually, the input word vector of BERT is the sum of Token Embedding, Segment Embedding and Position Embedding.

[0042] In another implementation of the present invention, word embedding processing is performed on the concatenated sentences through a transformer coding layer, including: utilizing the transformer coding layer to perform label embedding processing on the concatenated sentences through a mask matrix of the concatenated sentences, wherein the elements in the mask matrix indicate the attention weights of the concept labels for each word in the training sample sentences.

[0043] In another implementation of the present invention, the concept type label has a first attention weight for the concept word, and the concept type label has a second attention weight for the non-concept word, wherein the first attention weight indicates that the concept type label is visible to the concept word, and the second attention weight indicates that the concept type label is not visible to the non-concept word.

[0044] In another implementation of the present invention, the training sample sentences belong to the target task training samples, and the transformer coding layer is also trained by the source task training samples.

[0045] Since the training sample sentences belong to the target task training samples, and the transformer encoding layer is trained through the source task training samples, the transformer encoding layer learns the common implicit contextual relationship through the multi-task training process based on the target task and the source task, thereby further improving the analysis efficiency of the trained semantic dependency analysis model. In other words, a model with strong generalization ability is trained using multi-task learning data. After training based on the source task, a reliable semantic dependency analysis training model can be obtained by only slightly adjusting the weight parameters based on the target task.

[0046] In another implementation of the present invention, the concatenated sentences after word embedding processing are subjected to label masking processing and then passed through a semantic dependency analysis layer, including: setting the information related to the concept type label in the semantic dependency structure of the concatenated sentences to default, and obtaining the semantic dependency analysis label of the training sample sentence; using the result of label masking processing on the sentence after word embedding processing as input, and using the semantic dependency analysis label as output, to train the semantic dependency analysis layer.

[0047] Specifically, Figure 2 A schematic diagram of a semantic dependency analysis model of a model training method according to another embodiment of the present invention. As shown in the figure, the input on the left is a source task, and the input on the right is a target task. For example, the source task and the target task can be data from the same field or different fields, or data from the same scene or different scenes.

[0048] In another implementation of the present invention, the transformer encoding layer performs label embedding processing on the concatenated sentences through a mask matrix of the concatenated sentences, wherein the elements in the mask matrix indicate the attention weights of the concept labels for each word in the training sample sentences.

[0049] Since the mask matrix of concatenated sentences can improve the efficiency of data processing, the mask matrix is ​​used in the transformer encoding layer, which can improve the compatibility with the transformer model framework while ensuring the data processing efficiency.

[0050] Specifically, Figure 3A Schematic diagram of a mask matrix for concatenating sentences according to another embodiment of the present invention. Figure 3B FIG. 1 is a schematic diagram of a mathematical representation of a mask matrix of another embodiment of the present invention. As shown in the figure, Figure 3A The dark blocks in the image represent visible relationships, and the light blocks represent invisible relationships. Reflected in the mask matrix, in the attention mechanism, visible relationships can be represented by matrix element 1, and invisible relationships can be represented by matrix element 0. It should be understood that this is only exemplary, and the matrix element value corresponding to the visible relationship can be set to be greater than the matrix element value corresponding to the invisible relationship. Figure 3AAs shown in the figure, for the data in the target task, the concept type is concatenated after the word with the concept, for example: Saturday and Sunday are the "date" concept, and provident fund is the "service" concept. In the BERT encoding stage, it is ensured that the encoding of other words in the sentence is not affected by the concept type word. In other words, the concept type word only affects the words it directly modifies.

[0051] Specifically, in the training phase, at the transformer encoding layer, the mask transformer is used to ensure that other words in the sentence are invisible to irrelevant entity type words. At the semantic dependency analysis layer, label masking is added to the entity type words so that it does not affect the training process of the semantic dependency analysis layer. At the analysis phase, entity types such as "date", "location", and "service target" are removed from the parsed semantic dependency tree (an example of a semantic dependency structure), and the semantic dependency structure of the concatenated sentence is restored to the format of the semantic dependency structure of the sentence to be analyzed.

[0052] As an example, a concept type label has a first attention weight for a concept word, and a concept type label has a second attention weight for other words. The first attention weight indicates that the concept type label is visible to the concept word. The second attention weight indicates that the concept type label is not visible to other words.

[0053] Since the first attention weight indicates that the concept type label is visible to the concept word, and the second attention weight indicates that the concept type label is invisible to other words, the efficiency of mask processing is greatly improved, which not only ensures the semantic dependency relationship between concept words, but also increases the applicable scope of concept types, thereby improving the analysis efficiency of the trained semantic dependency analysis model.

[0054] As another example, a training sample sentence includes multiple concept words, and the multiple concept words include a first concept word and a second concept word. The concept type label of the first concept word has a first attention weight for the first concept word, the concept type label of the second concept word has a second attention weight for the first concept word, and the second concept word has a third attention weight corresponding to the first concept word. Among them, the first attention weight is greater than the second attention weight, and the first attention weight is greater than the third attention weight.

[0055] Therefore, by learning context information, the context association between the target concept word and its concept type label is greater than the context association between the target concept word and other words, which increases the applicability of the concept type and ensures that more accurate semantic dependency relationships are learned, thereby improving the analysis efficiency of the trained semantic dependency analysis model.

[0056] In addition, in one example, the third attention weight is greater than the second attention weight. Thus, by learning context information, the context association between the target concept word and other concept words is greater than the context association between the concept types of the target concept word and other concept words, ensuring that more accurate semantic dependency relationships are learned, thereby improving the analysis efficiency of the trained semantic dependency analysis model.

[0057] Specifically, Figure 4A This is a schematic diagram of a multi-head attention mechanism in a converter of another embodiment of the present invention. As shown in the figure, the multi-head attention mechanism can be briefly described as follows: the query (Query, Q), key (Key, K) and value (Value, V) are first subjected to a linear process, and then input into the scaled dot multiplication attention (here it can be performed h times, corresponding to the so-called multi-head, each time is counted as a head, and the parameter W for each linear transformation of Q, K, V can be different), and then the h times of scaled dot multiplication attention results are spliced ​​(for example, array connection), and then the value obtained by linear processing is used as the result of the multi-head attention.

[0058] Figure 4B Schematic diagram of a method for processing dot product attention in a transformer of another embodiment of the present invention. The query (Query, Q) and the key (Key, K) are matrix multiplied (linearly processed) and then normalized. The result of the normalization is processed using a mask matrix. The result of the mask matrix processing is normalized and then matrix multiplied (linearly processed) again with the value (Value, V) to obtain the result of the dot product attention.

[0059] It should be understood that the mask matrix can be implemented based on the padding mask matrix to achieve compatibility with the encoder mechanism of the transformer. The mask matrix can also be used to align the sequence length of the input training sample sentences.

[0060] In another implementation of the present invention, the concept type labels of the concept words in the training sample sentences are spliced ​​into the training sample sentences, including: determining the position information of each word in the training sample sentences; and splicing the concept type labels to the adjacent positions of the concept words based on the position information of each word.

[0061] Since in the training sample sentences, the concept type labels are spliced ​​to the adjacent positions of the concept words to obtain the spliced ​​sentences, the efficiency of the splicing process is improved, and the marking efficiency of the concept type labels is further improved.

[0062] In another implementation of the present invention, the semantic dependency analysis model composed of the transformer coding layer and the transformer coding layer is trained, including: training the semantic dependency analysis model composed of the transformer coding layer and the transformer coding layer through a target loss function, wherein the target loss function is used to perform label masking on the sentence processed by the label embedding based on the position of the concept type label. In other words, the parameters of the transformer coding layer and the semantic dependency analysis layer are adjusted through the target loss function to obtain the semantic dependency analysis model.

[0063] Since the target loss function is used to perform label masking on sentences that have been processed with label embedding based on the position of the concept type label, the influence of the concept type label on words other than the concept words is greatly reduced in the parameters of the obtained semantic dependency analysis model, so that the obtained semantic dependency analysis model can perform more accurate semantic dependency analysis.

[0064] Specifically, for source task training sentences, the first transformer can be used for training. For target task training sentences, the second transformer can be used for training. The first transformer corresponds to the first loss function, and the second transformer corresponds to the second loss function (the above-mentioned target loss function). The first loss function can be used to perform label masking on sentences processed by label embedding based on the position of the concept type label.

[0065] In addition, for the source task training sample sentences, the embedding vectors of the part of speech or concept type corresponding to each concept word can be connected to the word vector of each concept word, and the concept type vector embedding processing is performed to obtain the concept label embedding vector, so as to integrate the information about the concept type of the concept word into the concept label embedding vector. Then, the obtained concept label embedding vector can be used to train the semantic dependency analysis model. Among them, the first loss function does not perform label masking processing.

[0066] Figure 5 A schematic flowchart of a speech dependency analysis method according to another embodiment of the present invention. Figure 5 The semantic dependency analysis methods include:

[0067] 510: Generate a concatenated sentence by concatenating the concept type labels of the concept words in the sentence to be analyzed into the sentence to be analyzed.

[0068] 520: Input the concatenated sentences into a semantic dependency analysis model to obtain semantic dependency relations of the concatenated sentences, wherein the semantic dependency analysis model is trained by a model training method.

[0069] 530: In the semantic dependency relationship of the concatenated sentence, the dependency relationship data related to the concept type label is deleted to obtain the semantic dependency relationship of the sentence to be analyzed.

[0070] In the scheme of the embodiment of the present invention, the transformer encoding layer is used to use the attention mechanism to perform label embedding processing on the spliced ​​sentences. The attention weight of the concept type label for the concept words in the training sample sentences is higher than the attention weight for other words. Therefore, for the spliced ​​sentences, the attention weight of the concept type label for the concept words is greater than the attention weight for other words, so that in the sentence after the label embedding processing, the concept type label has a greater impact on the concept words than on other words, so that even if a new concept type appears, the concept type has a smaller impact on other words in the sentence, thereby ensuring the effect of using the semantic dependency analysis model for semantic dependency analysis and improving the scope of application of the semantic dependency analysis model to concept types. In addition, in the semantic dependency relationship of the spliced ​​sentence, the dependency relationship data related to the concept type label is deleted to obtain the semantic dependency relationship of the sentence to be analyzed, thereby improving the correspondence between the semantic dependency analysis result and the sentence to be analyzed.

[0071] It should be understood that the model training method and semantic dependency analysis method of the embodiments of the present invention can be applied to intent recognition in dialogue scenarios, question-and-answer scenarios, reasoning scenarios, etc.

[0072] For example, a concatenated sentence can be generated by concatenating the concept type labels of the concept words in the sentence to be analyzed for intent recognition. The concatenated sentence is input into the semantic dependency analysis model to obtain the semantic dependency relationship of the concatenated sentence. In the semantic dependency relationship of the concatenated sentence, the dependency relationship data related to the concept type label is deleted to obtain the semantic dependency relationship of the sentence to be analyzed for intent recognition, and the intent recognition result is obtained based on the semantic dependency relationship.

[0073] For example, a concatenated sentence can be generated by concatenating the concept type labels of the concept words in the question sentence into the sentence to be analyzed. The concatenated sentence is input into the semantic dependency analysis model to obtain the semantic dependency relationship of the concatenated sentence. In the semantic dependency relationship of the concatenated sentence, the dependency relationship data related to the concept type label is deleted to obtain the semantic dependency relationship of the question sentence, and the answer sentence is obtained based on the semantic dependency relationship.

[0074] For example, a concatenated sentence can be generated by concatenating the concept type labels of the concept words in the text to be inferred into the sentence to be analyzed. The concatenated sentence is input into the semantic dependency analysis model to obtain the semantic dependency relationship of the concatenated sentence. In the semantic dependency relationship of the concatenated sentence, the dependency relationship data related to the concept type label is deleted to obtain the semantic dependency relationship of the text to be inferred, and the inference result is obtained based on the semantic dependency relationship.

[0075] Specifically, refer again to Figure 2In the example of , the semantic dependency of the concatenated sentence or the sentence to be analyzed can be represented by a semantic dependency structure. In the parsed semantic dependency tree (an example of a semantic dependency structure), entity types such as "date", "place", and "service target" are removed, and the semantic dependency structure of the concatenated sentence is restored to the format of the semantic dependency structure of the sentence to be analyzed.

[0076] In another implementation of the present invention, in the semantic dependency relationships of the concatenated sentences, the dependency data related to the concept type tags are deleted to obtain the semantic dependency relationships of the sentences to be analyzed, including: in the semantic dependency relationships of the concatenated sentences, the dependency data related to the concept type tags are deleted to obtain the initial semantic dependency relationships of the sentences to be analyzed; and the initial semantic dependency relationships are normalized to obtain the semantic dependency relationships of the sentences to be analyzed.

[0077] Since the initial semantic dependency is normalized to obtain the semantic dependency of the sentence to be analyzed, the obtained semantic dependency analysis result has a better correspondence with the sentence to be analyzed, so as to facilitate subsequent natural language processing.

[0078] In another implementation of the present invention, the method further includes: in a dialogue scenario, obtaining the sentence to be analyzed; and replying to the sentence to be analyzed based on the semantic dependency relationship of the sentence to be analyzed.

[0079] It should be understood that the sentence to be analyzed may be a first dialogue sentence such as a question sentence or a reply sentence. A corresponding second dialogue sentence such as a reply sentence or a question sentence may be determined based on the semantic dependency of the sentence to be analyzed. The sentence to be analyzed may be answered using the second dialogue sentence.

[0080] It should also be understood that the dialogue scenario can be a reasoning dialogue scenario, a knowledge question and answer scenario, or an intent recognition scenario, etc.

[0081] Figure 6 The present invention is a schematic block diagram of a model training device according to another embodiment of the present invention. Figure 6 The model training device includes:

[0082] The concatenation module 610 adds concept type labels to concept words in the training sample sentences to obtain concatenated sentences.

[0083] The word embedding module 620 performs word embedding processing on the concatenated sentence through a transformer coding layer, so that the non-concept words in the training sample sentence are not affected by the concept type label.

[0084] The training module 630 performs label masking on the concatenated sentences after word embedding processing and passes them through the semantic dependency analysis layer, so that the semantic analysis layer learns the semantics of the concept type label, and obtains a trained semantic dependency analysis model including the transformer encoding layer and the semantic dependency analysis layer.

[0085] In the solution of the embodiment of the present invention, since the concept type label is masked for non-concept words other than the concept words in the training sample sentence, the non-concept words are not affected by the concept type label, thereby increasing the influence of the concept type label on the concept words in the transformer encoding layer. In addition, after the concatenated sentence after the word embedding process is subjected to label masking process, it passes through the semantic dependency analysis layer, so that the semantic analysis layer learns the semantics of the concept type label, avoiding the influence of the concept type label on the semantic analysis of the concept word, thereby improving the concept type application range of the model.

[0086] In another implementation of the present invention, the transformer encoding layer performs label embedding processing on the concatenated sentences through a mask matrix of the concatenated sentences, wherein the elements in the mask matrix indicate the attention weights of the concept labels for each word in the training sample sentences.

[0087] In another implementation of the present invention, the concept type label has a first attention weight for the concept word, and the concept type label has a second attention weight for other words, wherein the first attention weight indicates that the concept type label is visible to the concept word, and the second attention weight indicates that the concept type label is not visible to other words.

[0088] In another implementation of the present invention, the training sample sentences belong to the target task training samples, and the transformer coding layer is also trained by the source task training samples.

[0089] In another implementation of the present invention, the splicing module is specifically used to: determine the position information of each word in the training sample sentence; and splice the concept type label to the adjacent position of the concept word based on the position information of each word.

[0090] In another implementation of the present invention, the training module is specifically used to: train the semantic dependency analysis model composed of the transformer coding layer and the transformer coding layer through a target loss function, wherein the target loss function is used to perform label masking on sentences that have been processed by label embedding based on the position of the concept type label.

[0091] In another implementation of the present invention, the training module is specifically used to: set the information related to the concept type label in the semantic dependency structure of the concatenated sentence to default, and obtain the semantic dependency analysis label of the training sample sentence; use the result of label masking processing of the sentence processed by word embedding as input, and use the semantic dependency analysis label as output to train the semantic dependency analysis layer.

[0092] The device of this embodiment is used to implement the corresponding methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. In addition, the functional implementation of each module in the device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be repeated here.

[0093] Figure 7 It is a schematic block diagram of a speech dependency analysis device according to another embodiment of the present invention. Figure 7 The semantic dependency analysis device comprises:

[0094] The concatenation module 710 generates a concatenated sentence by concatenating the concept type labels of the concept words in the sentence to be analyzed into the sentence to be analyzed.

[0095] The analysis module 720 inputs the concatenated sentences into a semantic dependency analysis model to obtain an initial semantic dependency relationship of the sentence to be analyzed, wherein the semantic dependency analysis model is obtained by training according to a model training method.

[0096] The post-processing module 730 deletes the dependency data related to the concept type label in the semantic dependency of the concatenated sentence to obtain the semantic dependency of the sentence to be analyzed.

[0097] In the scheme of the embodiment of the present invention, the transformer encoding layer is used to use the attention mechanism to perform label embedding processing on the spliced ​​sentences. The attention weight of the concept type label for the concept words in the training sample sentences is higher than the attention weight for other words. Therefore, for the spliced ​​sentences, the attention weight of the concept type label for the concept words is greater than the attention weight for other words, so that in the sentence after the label embedding processing, the concept type label has a greater impact on the concept words than on other words, so that even if a new concept type appears, the concept type has a smaller impact on other words in the sentence, thereby ensuring the effect of using the semantic dependency analysis model for semantic dependency analysis and improving the scope of application of the semantic dependency analysis model to concept types. In addition, in the semantic dependency relationship of the spliced ​​sentence, the dependency relationship data related to the concept type label is deleted to obtain the semantic dependency relationship of the sentence to be analyzed, thereby improving the correspondence between the semantic dependency analysis result and the sentence to be analyzed.

[0098] In another implementation of the present invention, the post-processing module is specifically used to: delete the dependency data related to the concept type label in the semantic dependency relationship of the concatenated sentence to obtain the initial semantic dependency relationship of the sentence to be analyzed; normalize the initial semantic dependency relationship to obtain the semantic dependency relationship of the sentence to be analyzed.

[0099] In another implementation of the present invention, the device further comprises an acquisition module and a reply module. The acquisition module acquires the sentence to be analyzed in the dialogue scene. The reply module replies to the sentence to be analyzed based on the semantic dependency relationship of the sentence to be analyzed.

[0100] The device of this embodiment is used to implement the corresponding methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. In addition, the functional implementation of each module in the device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be repeated here.

[0101] Figure 8 The hardware structure of the electronic device of another embodiment of the present invention is as follows; Figure 8 As shown, the hardware structure of the electronic device may include: a processor 801, a communication interface 802, a storage medium 803 and a communication bus 804;

[0102] The processor 801, the communication interface 802, and the storage medium 803 communicate with each other via a communication bus 804; optionally, the communication interface 802 may be an interface of a communication module;

[0103] The processor can be specifically configured to: add a concept type label to the concept words in the training sample sentences to obtain a concatenated sentence; perform word embedding processing on the concatenated sentence through the transformer coding layer, so that the non-concept words in the training sample sentences are not affected by the concept type label; perform label masking processing on the concatenated sentence after the word embedding processing and pass it through the semantic dependency analysis layer, so that the semantic analysis layer learns the semantics of the concept type label, and obtains a trained semantic dependency analysis model including the transformer coding layer and the semantic dependency analysis layer;

[0104] Alternatively, the processor can be specifically configured to: generate a concatenated sentence by concatenating the concept type labels of the concept words in the sentence to be analyzed into the sentence to be analyzed; input the concatenated sentence into a semantic dependency analysis model to obtain the semantic dependency relationship of the concatenated sentence, wherein the semantic dependency analysis model is obtained by training according to a model training method; in the semantic dependency relationship of the concatenated sentence, delete the dependency relationship data related to the concept type label to obtain the semantic dependency relationship of the sentence to be analyzed.

[0105] The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0106] The storage medium may be, but is not limited to, a random access memory medium (RAM), a read-only memory medium (ROM), a programmable read-only memory medium (PROM), an erasable programmable read-only memory medium (EPROM), an electrically erasable read-only memory medium (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0107] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a storage medium, and the computer program includes a program code configured to execute the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present invention are executed. It should be noted that the storage medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, an apparatus or a device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program configured to be used by or in conjunction with an instruction execution system, an apparatus or a device. The program code contained on the storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0108] Computer program code configured to perform the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains one or more executable instructions configured to implement the specified logical function. There are specific sequential relationships in the above-mentioned specific embodiments, but these sequential relationships are only exemplary. When the specific implementation is implemented, these steps may be less, more or the execution order may be adjusted. That is, in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0110] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The names of these modules do not limit the modules themselves in some cases.

[0111] As another aspect, the present invention further provides a storage medium having a computer program stored thereon, which implements the method described in the above embodiment when executed by a processor.

[0112] As another aspect, the present invention also provides a storage medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above storage medium carries one or more programs, and when the above one or more programs are executed by the device, the device: adds concept type labels to concept words in the training sample sentences to obtain concatenated sentences; performs word embedding processing on the concatenated sentences through the transformer coding layer, so that non-concept words in the training sample sentences are not affected by the concept type labels; performs label masking processing on the concatenated sentences after word embedding processing and passes them through the semantic dependency analysis layer, so that the semantic analysis layer learns the semantics of the concept type labels, and obtains a trained semantic dependency analysis model including the transformer coding layer and the semantic dependency analysis layer;

[0113] Alternatively, the device is configured to: generate a concatenated sentence by concatenating the concept type labels of the concept words in the sentence to be analyzed into the sentence to be analyzed; input the concatenated sentence into a semantic dependency analysis model to obtain the semantic dependency relationship of the concatenated sentence, wherein the semantic dependency analysis model is obtained by training according to a model training method; and delete the dependency relationship data related to the concept type label in the semantic dependency relationship of the concatenated sentence to obtain the semantic dependency relationship of the sentence to be analyzed.

[0114] The expressions "first", "second", "the first" or "the second" used in various embodiments of the present disclosure may modify various components regardless of order and / or importance, but these expressions do not limit the corresponding components. The above expressions are only configured for the purpose of distinguishing an element from other elements. For example, a first user device and a second user device represent different user devices, although both are user devices. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the present disclosure.

[0115] When one element (e.g., a first element) is referred to as being "(operably or communicatively) coupled" or "(operably or communicatively) coupled to" or "connected to" another element (e.g., a second element), it is understood that the one element is directly connected to the other element or the one element is indirectly connected to the other element via yet another element (e.g., a third element). Conversely, it is understood that when an element (e.g., a first element) is referred to as being "directly connected" or "directly coupled" to another element (the second element), no element (e.g., a third element) is interposed between the two.

[0116] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.

Claims

1. A model training method, comprising: Add concept type labels to concept words in training sample sentences to obtain concatenated sentences; Through the transformer encoding layer, word embedding processing is performed on the concatenated sentence, so that the non-concept words in the training sample sentence are not affected by the concept type label; The concatenated sentences after word embedding processing are subjected to label masking processing and then passed through a semantic dependency analysis layer, so that the semantic analysis layer learns the semantics of the concept type label, and a trained semantic dependency analysis model including the transformer encoding layer and the semantic dependency analysis layer is obtained; wherein the concept type label has a first attention weight for the concept word, and the concept type label has a second attention weight for the non-concept word, wherein the first attention weight indicates that the concept type label is visible to the concept word, and the second attention weight indicates that the concept type label is not visible to the non-concept word.

2. The method according to claim 1, wherein: The word embedding process is performed on the concatenated sentence through the transformer encoding layer, including: The transformer encoding layer is used to perform label embedding processing on the concatenated sentence through the mask matrix of the concatenated sentence, wherein the elements in the mask matrix indicate the attention weights of the concept labels for each word in the training sample sentence.

3. The method according to claim 1, wherein: The training sample sentences belong to target task training samples, and the transformer encoding layer is also trained by source task training samples.

4. The method according to claim 1, wherein: The step of adding concept type labels to concept words in the training sample sentences to obtain concatenated sentences includes: Determining the position information of each word in the training sample sentence; Based on the position information of each word, the concept type label is spliced ​​to the adjacent position of the concept word to obtain the spliced ​​sentence.

5. The method according to claim 1, wherein: The concatenated sentences after word embedding are subjected to label masking and then passed through a semantic dependency analysis layer, including: Setting the information related to the concept type label in the semantic dependency structure of the concatenated sentence as default to obtain a semantic dependency analysis label of the training sample sentence; The result of label masking the sentence processed by word embedding is used as input, and the semantic dependency analysis label is used as output to train the semantic dependency analysis layer.

6. The method according to claim 1, wherein: The method further comprises: The parameters of the transformer encoding layer and the semantic dependency analysis layer are adjusted through the target loss function to obtain the semantic dependency analysis model, wherein the target loss function is used to perform label masking on the sentences that have been processed by label embedding based on the position of the concept type label.

7. A semantic dependency analysis method, comprising: Generate a concatenated sentence by concatenating the concept type labels of the concept words in the sentence to be analyzed into the sentence to be analyzed; Inputting the concatenated sentences into a semantic dependency analysis model to obtain semantic dependency relations of the concatenated sentences, wherein the semantic dependency analysis model is trained by a model training method according to any one of claims 1 to 6; In the semantic dependency relationship of the concatenated sentences, the dependency relationship data related to the concept type label is deleted to obtain the semantic dependency relationship of the sentence to be analyzed.

8. The method according to claim 7, wherein: In the semantic dependency relationship of the concatenated sentences, the dependency relationship data related to the concept type label is deleted to obtain the semantic dependency relationship of the sentence to be analyzed, including: In the semantic dependency relationship of the concatenated sentence, the dependency relationship data related to the concept type label is deleted to obtain the initial semantic dependency relationship of the sentence to be analyzed; The initial semantic dependency relationship is normalized to obtain the semantic dependency relationship of the sentence to be analyzed.

9. The method according to claim 7, wherein: The method further comprises: In a dialogue scenario, obtaining the sentence to be analyzed; Based on the semantic dependency relationship of the sentence to be analyzed, the sentence to be analyzed is replied.

10. A model training device, comprising: The concatenation module adds concept type labels to the concept words in the training sample sentences to obtain concatenated sentences; A word embedding module, which performs word embedding processing on the concatenated sentence through a transformer encoding layer, so that the non-concept words in the training sample sentence are not affected by the concept type label; A training module, which performs label masking on the concatenated sentences after word embedding processing and passes them through a semantic dependency analysis layer, so that the semantic analysis layer learns the semantics of the concept type label, and obtains a trained semantic dependency analysis model including the transformer encoding layer and the semantic dependency analysis layer; wherein the concept type label has a first attention weight for the concept word, and the concept type label has a second attention weight for the non-concept word, wherein the first attention weight indicates that the concept type label is visible to the concept word, and the second attention weight indicates that the concept type label is not visible to the non-concept word.

11. A semantic dependency analysis device, comprising: A splicing module, which generates a spliced ​​sentence by splicing the concept type labels of the concept words in the sentence to be analyzed into the sentence to be analyzed; An analysis module, inputting the concatenated sentence into a semantic dependency analysis model to obtain an initial semantic dependency relationship of the sentence to be analyzed, wherein the semantic dependency analysis model is trained by a model training method according to any one of claims 1 to 6; The post-processing module deletes the dependency data related to the concept type label in the semantic dependency of the concatenated sentence to obtain the semantic dependency of the sentence to be analyzed.

12. An electronic device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method according to any one of claims 1 to 9.

13. A storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images