A semantic integrity judgment method and system based on multi-head attention mechanism

Abstract

The present application discloses a method and system for judging semantic integrity based on a multi-head attention mechanism, the method comprising: based on a preset semantic truncation mechanism, performing sentence truncation processing on an input corpus to obtain a truncated input corpus; performing data preprocessing on the truncated input corpus through a Bert word segmenter to obtain a preprocessed input corpus; introducing a masked multi-head attention sublayer and a multi-head attention mechanism module to construct a semantic integrity judgment model; performing semantic integrity judgment processing on the preprocessed input corpus based on the semantic integrity judgment model to obtain an input corpus integrity judgment result. The embodiments of the present application can improve the user's interactive experience by improving the computational efficiency of the model and the semantic judgment accuracy. The present application can be widely used in the field of intelligent semantic judgment technology.

Description

Technical Field

[0001] The present application relates to the field of intelligent semantic judgment technology, and in particular to a semantic integrity judgment method and system based on a multi-head attention mechanism. Background Art

[0002] When the user interacts with the machine by voice, the machine will continuously receive the user's voice input, and then use ASR technology to translate the user's voice input into text, and then send the text to the subsequent NLP program for processing. However, NLP programs are often very sensitive to input. When the user speaks too slowly or likes to prolong the sound, the VAD model will misjudge the end of the voice in the middle of playing the voice, resulting in incomplete sentences recognized by the subsequent ASR engine, and then it will be impossible to know the user's intention, because the existing ASR technology often uses a method of using VAD delay detection to judge, that is, real-time detection of the audio stream. If a pronunciation similar to text is detected, it will continue to monitor until it stops when no text pronunciation is received for a period of time, and then start recognition. However, when the quiet time is too short, it is very easy to cause false interruptions, especially when the user uses prolongation. When the quiet time is too long, the overall interactive experience will become dull, and there is no guarantee that false interruptions will not occur, resulting in a poor user experience.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a semantic integrity judgment method and system based on a multi-head attention mechanism, which can improve the user's interactive experience by improving the computational efficiency of the model and the semantic judgment accuracy.

[0005] To achieve the above purpose, one aspect of an embodiment of the present application proposes a semantic integrity judgment method based on a multi-head attention mechanism, the method comprising:

[0006] Based on the preset semantic truncation mechanism, the input corpus is subjected to sentence truncation processing to obtain the truncated input corpus;

[0007] Performing data preprocessing on the truncated input corpus by using a Bert word segmenter to obtain preprocessed input corpus;

[0008] Introduce masked multi-head attention sublayer and multi-head attention mechanism module to build a semantic integrity judgment model;

[0009] The preprocessed input corpus is subjected to semantic integrity judgment processing based on the semantic integrity judgment model to obtain an input corpus integrity judgment result.

[0010] In some embodiments, the sentence truncation processing is performed on the input corpus based on a preset semantic truncation mechanism to obtain the truncated input corpus, including:

[0011] Acquire speech input data and perform scan conversion processing to obtain input corpus;

[0012] Constructing a preset semantic truncation mechanism, the preset semantic truncation mechanism includes a first truncation position and a second truncation position, the first truncation position is n bytes before the end of the sentence of the input corpus, and the second truncation position is n bytes before the punctuation mark of the input corpus;

[0013] According to the preset semantic truncation mechanism, the input corpus is randomly truncate-processed to obtain preliminary truncated input corpus;

[0014] The input corpus that does not meet the preset byte length in the preliminary truncated input corpus is eliminated to obtain the truncated input corpus.

[0015] In some embodiments, the step of performing data preprocessing on the truncated input corpus by using a Bert word segmenter to obtain the preprocessed input corpus includes:

[0016] Performing data regularization processing on the truncated input corpus to obtain regularized truncated input corpus;

[0017] Based on the Bert word segmenter, special character preprocessing is performed on the regularized truncated input corpus to obtain a truncated input corpus with space characters;

[0018] The truncated input corpus having space characters is divided and processed to construct a truncated input corpus token list;

[0019] Assign an id tag to the characters in the token list of the truncated input corpus to construct an input corpus vocabulary;

[0020] The input corpus vocabulary is subjected to sequence vectorization processing to obtain the preprocessed input corpus.

[0021] In some embodiments, the performing special character preprocessing on the regularized truncated input corpus based on the Bert word segmenter to obtain the truncated input corpus with space characters includes:

[0022] Based on the Bert word segmenter, special characters and control characters are deleted from the regularized truncated input corpus to obtain the deleted truncated input corpus;

[0023] Obtaining the tab character of the deleted truncated input corpus, the carriage return character of the deleted truncated input corpus, and the line feed character of the deleted truncated input corpus, and replacing them with spaces to obtain preliminary truncated input corpus with space characters;

[0024] The CJK characters of the deleted truncated input corpus are obtained, and a space is inserted after the CJK characters to obtain the truncated input corpus with the space character.

[0025] In some embodiments, the semantic integrity judgment model includes a position encoding layer, a masked multi-head attention sublayer, a normative layer, a multi-head attention mechanism module, a forward feedback layer, a linear layer and a loss function calculation layer, wherein:

[0026] The position coding layer includes an embedding layer and a coding layer;

[0027] The multi-head attention mechanism module includes a segmentation module, several attention head modules, a self-attention map calculation module and a mapping module.

[0028] In some embodiments, the performing semantic integrity judgment processing on the preprocessed input corpus based on the semantic integrity judgment model to obtain an input corpus integrity judgment result includes:

[0029] Inputting the preprocessed input corpus into the semantic integrity judgment model;

[0030] Based on the position encoding layer of the semantic integrity judgment model, the preprocessed input corpus is subjected to word embedding and position encoding processing to obtain an embedding vector of the input corpus;

[0031] Based on the masked multi-head attention sublayer of the semantic integrity judgment model, calculating the attention feature value of the embedding vector of the input corpus to obtain the feature value of the first input corpus vector;

[0032] Based on the standardization layer of the semantic integrity judgment model, the first input corpus vector feature value is standardized to obtain a standardized first input corpus vector feature value;

[0033] Based on the multi-head attention mechanism module of the semantic integrity judgment model, multi-head attention feature value calculation is performed on the embedding vector of the input corpus to obtain a second input corpus vector feature value;

[0034] Combining the standardized first input corpus vector feature value with the second input corpus vector feature value to obtain an input corpus vector feature value;

[0035] Based on the forward feedback layer of the semantic integrity judgment model, the input corpus vector feature value is subjected to nonlinear transformation processing to obtain the input corpus vector feature value after nonlinear transformation;

[0036] Based on the linear layer of the semantic integrity judgment model, feature scaling is performed on the input corpus vector feature value after the nonlinear transformation to obtain the scaled input corpus vector feature value;

[0037] Based on the loss function calculation layer of the semantic integrity judgment model, a weight distribution calculation is performed on the scaled input corpus vector feature value to obtain the input corpus integrity judgment result.

[0038] In some embodiments, the position encoding layer based on the semantic integrity judgment model performs word embedding and position encoding processing on the preprocessed input corpus to obtain an embedding vector of the input corpus, including:

[0039] Inputting the preprocessed input corpus into the position encoding layer of the semantic integrity judgment model;

[0040] Based on the embedding layer of the position encoding layer, performing word embedding processing on the preprocessed input corpus to obtain embedded input corpus;

[0041] Based on the encoding layer of the position encoding layer, position encoding processing is performed on the embedded input corpus to obtain an embedding vector of the input corpus.

[0042] In some embodiments, the multi-head attention mechanism module based on the semantic integrity judgment model performs multi-head attention feature value calculation on the embedding vector of the input corpus to obtain the feature value of the second input corpus vector, including:

[0043] Inputting the embedding vector of the input corpus into the multi-head attention mechanism module of the semantic integrity judgment model;

[0044] Based on the segmentation module of the multi-head attention mechanism module, the embedding vector of the input corpus is segmented to obtain several embedding sub-vectors of the input corpus;

[0045] Based on the multiple attention head modules and the self-attention graph calculation module of the multi-head attention mechanism module, multi-head attention calculation is performed on the embedding sub-vectors of the multiple input corpora to obtain the embedding sub-vector feature values ​​of the multiple input corpora;

[0046] Based on the mapping module of the multi-head attention mechanism module, a connection mapping process is performed on the embedding sub-vector feature values ​​of several of the input corpora to obtain the vector feature value of the second input corpus.

[0047] In some embodiments, the loss function calculation layer of the semantic integrity judgment model includes a cross entropy loss function and a triple loss function. The expression of the loss function calculation layer is specifically as follows:

[0048] L=β·L CE +λ·L TP

[0049]

[0050] L TP =max(0,m+||f(x a )-f(x p )|| 2 -||f(x a )-f(x n )|| 2 )

[0051] In the above formula, L represents the comprehensive loss function of the loss function calculation layer, L CE represents the cross entropy loss function, L TP represents the triplet loss function, β represents the weight parameter of the cross entropy loss function, λ represents the weight parameter of the triplet loss function, n represents the training batch size, w represents the weight of the fully connected layer, x represents the input feature data, y i represents the probability of the i-th category, i and j represent the sequence numbers of samples in each batch, a represents the anchor sample, p represents the positive sample, n represents the negative sample, and m represents the boundary parameter.

[0052] To achieve the above purpose, another aspect of the embodiment of the present application proposes a semantic integrity judgment system based on a multi-head attention mechanism, the system comprising:

[0053] A truncation module is used to perform sentence truncation processing on the input corpus based on a preset semantic truncation mechanism to obtain truncated input corpus;

[0054] A preprocessing module, used for performing data preprocessing on the truncated input corpus through a Bert word segmenter to obtain preprocessed input corpus;

[0055] The construction module is used to introduce the masked multi-head attention sub-layer and the multi-head attention mechanism module to build a semantic integrity judgment model;

[0056] The judgment module is used to perform semantic integrity judgment processing on the preprocessed input corpus based on the semantic integrity judgment model to obtain an input corpus integrity judgment result.

[0057] The embodiments of the present application include at least the following beneficial effects: The present application provides a method and system for judging semantic integrity based on a multi-head attention mechanism. The scheme performs sentence truncation processing on the input corpus through a preset semantic truncation mechanism, thereby increasing the number of negative samples of the input data and improving the recognition and judgment accuracy of the later model. The truncated input corpus is further preprocessed by a Bert word segmenter. By preprocessing the data with special characters, the computational efficiency of the later model can be greatly improved. A masked multi-head attention sublayer and a multi-head attention mechanism module are further introduced to construct a semantic integrity judgment model. Finally, semantic integrity judgment processing is performed based on the semantic integrity judgment model. By splitting the input features and interacting each attention head with different feature subsets, repeated calculations between attention heads can be effectively reduced, thereby improving computational efficiency and enhancing user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flowchart of a semantic integrity judgment method based on a multi-head attention mechanism provided in an embodiment of the present application;

[0059] Figure 2 It is a structural diagram of the semantic integrity judgment model provided in the embodiment of the present application;

[0060] Figure 3 It is a structural diagram of the multi-head attention mechanism module provided in the embodiment of the present application;

[0061] Figure 4 It is a structural diagram of a semantic integrity judgment system based on a multi-head attention mechanism provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0063] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0064] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] Reference Figure 1 , Figure 1 A flowchart of a semantic integrity judgment method based on a multi-head attention mechanism provided by an embodiment of the present invention, referring to Figure 1 , the method comprises the following steps:

[0067] S100, based on a preset semantic truncation mechanism, performing sentence truncation processing on the input corpus to obtain truncated input corpus;

[0068] It should be noted that, in some embodiments, step S100 may include: S110, obtaining speech input data and performing scan conversion processing to obtain input corpus; S120, constructing a preset semantic truncation mechanism, the preset semantic truncation mechanism including a first truncation position and a second truncation position, the first truncation position being the n-byte position before the end of the sentence of the input corpus, and the second truncation position being the n-byte position before the punctuation mark of the input corpus; S130, performing random truncation processing on the input corpus according to the preset semantic truncation mechanism to obtain preliminary truncated input corpus; S140, eliminating the input corpus that does not meet the preset byte length after the preliminary truncation to obtain the truncated input corpus.

[0069] In some specific embodiments, first, sufficient Chinese corpus in business scenarios is collected, with punctuation marks as much as possible and without containing other languages, and then the corpus is preprocessed, wherein the corpus is divided into two categories, the corpus collected by the crawler and the business voice corpus. For the crawler input corpus, first, for an entire paragraph, start from the paragraph and scan it word by word. When a Chinese period (".") and more than one line break are recognized (at this time it is represented as a paragraph separator, such as "\n\n", the main reason is that there are a large number of sentences on the Internet that do not end with a period), the scanned sentence is stored in the database. If the sentence ends with multiple line breaks, the line breaks at the end are removed and an extra period is added before storing it in the database (mainly to facilitate subsequent data processing operations). For the voice input corpus, the ASR engine is used in advance to convert the voice corpus into text corpus. Since the voice corpus is collected from telephone voice data, there are only two speakers, and the voices of these two people do not affect each other. Therefore, the speaker classification engine is used directly to divide the voice pieces of different people according to the different voiceprints of the two people, and then a single voice piece is directly injected into the ASR engine to finally obtain a single sentence.

[0070] Secondly, the sentence is truncated. The data is taken from the database by sentence. For the part of the data containing punctuation marks, it is randomly truncated from the beginning of the sentence to the position that is 3 words away from the end of the sentence and at least 1 word away from each punctuation mark. At the same time, the length of the remaining words after truncation is guaranteed to be greater than 4 words. The reason why the distance to the end is required to be 3 words and the length of the remaining words must be greater than 4 words is that the sentence meaning contained in short sentences is often weak. In actual business scenarios, users often continue with the following after saying a single short sentence. For example, in actual business usage, the following sample words are semantically complete, but they should not be truncated, such as: "Hello", "Well, OK". The corresponding original sentences are: "Hello, how can I help you?", "Well, OK, I will call you tomorrow." We do not want users to pass the semantic integrity check after only saying the beginning word. Setting the truncated word to a remaining length greater than four words can effectively avoid the generation of such training data. The reason why it is required to be at least one word away from the punctuation mark is that the random truncated data generated in this case is more semantically incomplete.

[0071] For example:

[0072] “Um ma’am who can hear me, this is a call center”;

[0073] "Well ma'am you can hear me";

[0074] "Hmm audible female";

[0075] Examples of incorrect truncation methods:

[0076] “Well audible lady”;

[0077] It is obvious here that the confusion of the first two sentences with commas and one word length is higher, while the last sentence has higher semantic completeness.

[0078] For the training samples generated by truncated data and original data, the ratio of complete semantics to truncated semantics in the training samples is approximately 1:2. This is actually a method that easily leads to overfitting and makes the classifier more inclined to identify it as a negative sample. However, by setting the truncated semantics higher than the complete semantics, the recall rate of the model for negative samples can be reduced. For voice scenarios, it is better to make a misjudgment than a misjudgment, because the consequences of a misjudgment are more serious. For example, "Please check for me... Well, express delivery", here, "..." indicates that the user has a certain pause. At this time, VAD interruption will obtain the sentence "Please check for me". The semantic completeness of this sentence is relatively high, but in fact, a careful analysis shows that it has a certain degree of incompleteness ("Please check for me what"). When the training corpus is not sufficient and it is difficult to distinguish such subtle incompleteness, we still hope that the model can still judge (or misjudge) it as incomplete, so the proportion of negative samples is artificially raised during training.

[0079] S200, performing data preprocessing on the truncated input corpus through a Bert word segmenter to obtain preprocessed input corpus;

[0080] It should be noted that, in some embodiments, step S200 may include: S210, performing data regularization processing on the truncated input corpus to obtain regularized truncated input corpus; S220, based on the Bert word segmenter, performing special character preprocessing on the regularized truncated input corpus to obtain a truncated input corpus with space characters; S230, dividing the truncated input corpus with space characters to construct a truncated input corpus token list; S240, assigning an id tag to the characters in the truncated input corpus token list to construct an input corpus vocabulary; S250, performing sequence vectorization processing on the input corpus vocabulary to obtain the preprocessed input corpus.

[0081] It should be noted that, in some embodiments, step S220 may include: S221, based on the Bert word segmenter, performing special character and control character deletion processing on the regularized truncated input corpus to obtain the deleted truncated input corpus; S222, obtaining the tab character of the deleted truncated input corpus, the carriage return character of the deleted truncated input corpus, and the line feed character of the deleted truncated input corpus, and replacing them with spaces to obtain a preliminary truncated input corpus with space characters; S223, obtaining the CJK characters of the deleted truncated input corpus, and inserting spaces after the CJK characters to obtain the truncated input corpus with space characters.

[0082] In some specific embodiments, all spaces in the sentence are removed. Spaces do not affect the semantics, which can compress the length of the tensor. All English characters in the sentence are removed (because a pure Chinese ASR engine is used). After these operations are completed, it can be passed into the Bert tokenizer later. The following operations are for the Bert tokenizer. First, special Unicode characters in the data are removed (Bert tokenizer). Among the control characters, the tab character ("\t"), the carriage return character ("\r"), and the line feed character ("\n") are replaced with corresponding space characters. Among the control characters, special characters that do not belong to the text, such as replacement characters, emoji characters, and null characters ("\0x00"), are all deleted. For normal characters, characters in the CJK Unicode block are all appended with a space at the end, and for the remaining characters in other normal characters, they are output as they are.

[0083] Further, for the truncated input corpus with space characters, the characters are all divided into a token list using spaces. Prepare a vocabulary that contains the following categories: special characters, words that do not exist in the data and belong to special definitions used to control the behavior of the model, such as the layer "[CLS]", "[PAD]"; suffix characters, mainly used for languages like English to represent subwords. For example, "wave" can be split into "wav" and "##e", and there are no suffix characters in Chinese. After completion, an id is assigned to each character. For example, the id of "你" is 1, and the id of "好" is 2. Finally, the sequence is vectorized. For example, if the id of "你" is 1 and the id of "好" is 2, then "你好" is converted into a tensor as [1, 2].

[0084] S300. Introduce a masked multi-head attention sublayer and a multi-head attention mechanism module to construct a semantic integrity judgment model;

[0085] It should be noted that, as Figure 2 shown, the semantic integrity judgment model includes a position encoding layer, a masked multi-head attention sublayer, a normalization layer, a multi-head attention mechanism module, a forward feedback layer, a linear layer, and a loss function calculation layer. Among them, the position encoding layer includes an embedding layer and an encoding layer, and the multi-head attention mechanism module includes a segmentation module, several attention head modules, a self-attention graph calculation module, and a mapping module.

[0086] In this embodiment, as Figure 3As shown in the figure, the multi-head attention mechanism module splits the input features and interacts each attention head with different feature subsets. This method can effectively reduce the repeated calculations between attention heads, thereby improving the computational efficiency, distributing the calculations of the attention heads on different feature subsets, avoiding computational redundancy, and thus optimizing the performance of the model. Secondly, the attention map of each attention head is calculated in a cascade manner, and the output of each head is gradually added to the input of the subsequent head to gradually optimize the feature representation. The application of this method can increase the depth of the network, thereby further improving the capacity of the model without introducing additional parameters.

[0087] S400, performing semantic integrity judgment processing on the preprocessed input corpus based on the semantic integrity judgment model to obtain an input corpus integrity judgment result;

[0088] It should be noted that, in some embodiments, step S400 may include: S410, inputting the preprocessed input corpus into the semantic integrity judgment model; S420, based on the position coding layer of the semantic integrity judgment model, performing word embedding and position coding processing on the preprocessed input corpus to obtain the embedding vector of the input corpus; S430, based on the masked multi-head attention sublayer of the semantic integrity judgment model, performing attention feature value calculation on the embedding vector of the input corpus to obtain the feature value of the first input corpus vector; S440, based on the standardization layer of the semantic integrity judgment model, performing standardization processing on the feature value of the first input corpus vector to obtain the standardized feature value of the first input corpus vector; S450, based on the multi-head attention mechanism module of the semantic integrity judgment model, performing attention feature value calculation on the embedding vector of the input corpus S460, the first input corpus vector eigenvalue is calculated by multi-head attention to obtain the eigenvalue of the second input corpus vector; S460, the standardized eigenvalue of the first input corpus vector is combined with the eigenvalue of the second input corpus vector to obtain the eigenvalue of the input corpus vector; S470, based on the forward feedback layer of the semantic integrity judgment model, the input corpus vector eigenvalue is processed by nonlinear transformation to obtain the eigenvalue of the input corpus vector after nonlinear transformation; S480, based on the linear layer of the semantic integrity judgment model, the input corpus vector eigenvalue after nonlinear transformation is processed by feature scaling to obtain the eigenvalue of the scaled input corpus vector; S490, based on the loss function calculation layer of the semantic integrity judgment model, the scaled input corpus vector eigenvalue is calculated by weight distribution to obtain the input corpus integrity judgment result.

[0089] In some specific embodiments, in some embodiments, step S420 may include: S421, inputting the preprocessed input corpus into the position coding layer of the semantic integrity judgment model; S422, based on the embedding layer of the position coding layer, performing word embedding processing on the preprocessed input corpus to obtain the embedded input corpus; S423, based on the encoding layer of the position coding layer, performing position coding processing on the embedded input corpus to obtain the embedding vector of the input corpus.

[0090] In some specific embodiments, in some embodiments, step S450 may include: S451, inputting the embedding vector of the input corpus into the multi-head attention mechanism module of the semantic integrity judgment model; S452, based on the segmentation module of the multi-head attention mechanism module, performing segmentation processing on the embedding vector of the input corpus to obtain several embedding sub-vectors of the input corpus; S453, based on several attention head modules and self-attention map calculation modules of the multi-head attention mechanism module, performing multi-head attention calculation on the embedding sub-vectors of several input corpora to obtain several embedding sub-vector eigenvalues ​​of the input corpora; S454, based on the mapping module of the multi-head attention mechanism module, performing connection mapping processing on the embedding sub-vector eigenvalues ​​of several input corpora to obtain the vector eigenvalues ​​of the second input corpus.

[0091] In a specific embodiment of the present invention, the position encoding layer performs the following two operations: converting the tensorized sequence into a word embedding, and fusing the word embedding tensor with the position encoding vector, converting the tensorized sequence into a word embedding, and using the skip-diam model, taking out the corresponding embedding tensor d from the embedding matrix according to the tensorized sequence value as an index. model The skip-diam model needs to be pre-trained with large-scale corpus, and its expression is:

[0092]

[0093]

[0094] In the above formula, PE i represents the position encoding vector, i represents the data batch, d model Represents an embedding tensor.

[0095] The role of position encoding is that the closer the two words are, the higher the cosine product of their position encoding tensors is, indicating that the two words are more related. Then the tensor embedding vector and the position encoding vector are directly added to obtain the tensor value of the next layer input.

[0096] Furthermore, based on the masked multi-head attention sublayer, a masked dot product attention algorithm is adopted, and its expression is:

[0097]

[0098] In the above formula, the M matrix is ​​a mask matrix, which is mainly used to set the penalty weight so that QK T Some values ​​of become infinitesimal, so the attention generated after softmax processing is 0, which makes the model not pay attention to these words.

[0099] Furthermore, based on the norm layer, residual connection is used. The role of residual connection is to connect layers with residuals. Since the computer processor has an upper limit on the storage of numbers, and the back propagation algorithm needs to multiply the gradients, a gradient less than 1 will cause the floating point number to lose precision and become 0 after multiple consecutive multiplications because the number is too small. A gradient greater than 1 will cause the number to overflow after multiple consecutive multiplications because it is too large. Therefore, this layer is mainly to prevent the gradient from disappearing and exploding due to the depth of the layer. The role of the norm layer is to scale the output of the previous layer. Since the output of each layer will cause a certain degree of change in the data distribution, layer normalization is required to correct the data distribution so that its mean is 0 and the variance is 1. The corrected model after layer normalization can converge faster.

[0100] Furthermore, based on the forward feedback layer, a certain nonlinear capability is provided to the model, because the simple attention layer mechanism is a linear function, but for matrix multiplication, multiple linear multiplications are essentially still linear functions and cannot play a predictive role. Therefore, the FFN layer is used as an activation layer to provide nonlinearity, based on the linear layer. The linear layer processing here is relatively special. After the layer normalization of the last layer, the model will output the following tensors (hidden layer, attention cache, hidden layer cache, self-attention and cross attention). In the model configuration, the number of hidden layer features is 768, so the output shape of the hidden layer tensor is (batch_size, 768). This tensor of length 768 describes the 768 features of the input sentence. Some of its features have a certain linear correlation with the binary features, so a linear layer is used to scale the 768 features to 2 features.

[0101] Finally, the loss function used by the model of the embodiment of the present invention is described, which includes a cross entropy loss function and a triple loss function. The expression of the loss function calculation layer is specifically as follows:

[0102] L=β·L CE +λ·L TP

[0103]

[0104] L TP =max(0,m+||f(x a )-f(x p )|| 2-||f(x a )-f(x n )|| 2 )

[0105] In the above formula, L represents the comprehensive loss function of the loss function calculation layer, L CE represents the cross entropy loss function, L TP represents the triplet loss function, β represents the weight parameter of the cross entropy loss function, λ represents the weight parameter of the triplet loss function, n represents the training batch size, w represents the weight of the fully connected layer, x represents the input feature data, y i represents the probability of the i-th category, i and j represent the sequence numbers of samples in each batch, a represents the anchor sample, p represents the positive sample, n represents the negative sample, and m represents the boundary parameter.

[0106] By integrating these two loss functions, the model takes into account the needs of similarity optimization and global classification performance during training, thereby improving the feature representation and generalization ability of the deep metric learning model, while effectively addressing the convergence problem in training.

[0107] In summary, the user side uses a microphone to continuously collect the user's original voice data. The server first pre-processes the original voice data, including resampling, noise reduction, segmentation and other operations. After pre-processing, the segmented voice data is tensorized and imported into the ASR engine to recognize the user's text. Then a VAD quiet waiting delay is set that is much shorter than the previous one, for example, 1s (originally 3s). When the VAD wait times out, the recognized sentence is sent to the judgment model trained above. If the semantics are judged to be complete, stop monitoring the user data and send it for subsequent processing. Otherwise, continue to wait for user input until a final tolerable time is reached, which will be relatively long (for example, 4s).

[0108] See also Figure 4 The embodiment of the present application also provides a semantic integrity judgment system based on a multi-head attention mechanism, which can implement the above-mentioned semantic integrity judgment method based on a multi-head attention mechanism. The system includes:

[0109] A truncation module is used to perform sentence truncation processing on the input corpus based on a preset semantic truncation mechanism to obtain truncated input corpus;

[0110] A preprocessing module is used to perform data preprocessing on the truncated input corpus through a Bert word segmenter to obtain preprocessed input corpus;

[0111] The construction module is used to introduce the masked multi-head attention sub-layer and the multi-head attention mechanism module to build a semantic integrity judgment model;

[0112] The judgment module is used to perform semantic integrity judgment processing on the preprocessed input corpus based on the semantic integrity judgment model to obtain the input corpus integrity judgment result.

[0113] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0114] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A semantic integrity judgment method based on a multi-head attention mechanism, characterized in that: The method comprises the following steps: Based on the preset semantic truncation mechanism, the input corpus is subjected to sentence truncation processing to obtain the truncated input corpus; Performing data preprocessing on the truncated input corpus by using a Bert word segmenter to obtain preprocessed input corpus; Introduce masked multi-head attention sublayer and multi-head attention mechanism module to build a semantic integrity judgment model; The semantic integrity judgment model includes a position encoding layer, a masked multi-head attention sublayer, a normative layer, a multi-head attention mechanism module, a forward feedback layer, a linear layer and a loss function calculation layer, wherein: The position coding layer includes an embedding layer and a coding layer; The multi-head attention mechanism module includes a segmentation module, several attention head modules, a self-attention map calculation module and a mapping module; Inputting the preprocessed input corpus into the semantic integrity judgment model; Based on the position encoding layer of the semantic integrity judgment model, the preprocessed input corpus is subjected to word embedding and position encoding processing to obtain an embedding vector of the input corpus; Based on the masked multi-head attention sublayer of the semantic integrity judgment model, calculating the attention feature value of the embedding vector of the input corpus to obtain the feature value of the first input corpus vector; Based on the standardization layer of the semantic integrity judgment model, the first input corpus vector feature value is standardized to obtain a standardized first input corpus vector feature value; Inputting the embedding vector of the input corpus into the multi-head attention mechanism module of the semantic integrity judgment model; Based on the segmentation module of the multi-head attention mechanism module, the embedding vector of the input corpus is segmented to obtain several embedding sub-vectors of the input corpus; Based on the multiple attention head modules and the self-attention graph calculation module of the multi-head attention mechanism module, multi-head attention calculation is performed on the embedding sub-vectors of the multiple input corpora to obtain the embedding sub-vector feature values ​​of the multiple input corpora; Based on the mapping module of the multi-head attention mechanism module, a connection mapping process is performed on the embedding sub-vector feature values ​​of the plurality of input corpora to obtain a second input corpus vector feature value; Combining the standardized first input corpus vector feature value with the second input corpus vector feature value to obtain an input corpus vector feature value; Based on the forward feedback layer of the semantic integrity judgment model, the input corpus vector feature value is subjected to nonlinear transformation processing to obtain the input corpus vector feature value after nonlinear transformation; Based on the linear layer of the semantic integrity judgment model, feature scaling is performed on the input corpus vector feature value after the nonlinear transformation to obtain the scaled input corpus vector feature value; Based on the loss function calculation layer of the semantic integrity judgment model, a weight distribution calculation is performed on the scaled input corpus vector feature value to obtain the input corpus integrity judgment result.

2. The method according to claim 1, characterized in that The sentence truncation process is performed on the input corpus based on the preset semantic truncation mechanism to obtain the truncated input corpus, including: Acquire speech input data and perform scan conversion processing to obtain input corpus; Constructing a preset semantic truncation mechanism, the preset semantic truncation mechanism includes a first truncation position and a second truncation position, the first truncation position is n bytes before the end of the sentence of the input corpus, and the second truncation position is n bytes before the punctuation mark of the input corpus; According to the preset semantic truncation mechanism, the input corpus is randomly truncate-processed to obtain preliminary truncated input corpus; The input corpus that does not meet the preset byte length in the preliminary truncated input corpus is eliminated to obtain the truncated input corpus.

3. The method according to claim 1, characterized in that The step of performing data preprocessing on the truncated input corpus by using a Bert word segmenter to obtain the preprocessed input corpus includes: Performing data regularization processing on the truncated input corpus to obtain regularized truncated input corpus; Based on the Bert word segmenter, special character preprocessing is performed on the regularized truncated input corpus to obtain a truncated input corpus with space characters; The truncated input corpus having space characters is divided and processed to construct a truncated input corpus token list; Assign an id tag to the characters in the token list of the truncated input corpus to construct an input corpus vocabulary; The input corpus vocabulary is subjected to sequence vectorization processing to obtain the preprocessed input corpus.

4. The method according to claim 3, characterized in that The method of performing special character preprocessing on the regularized truncated input corpus based on the Bert word segmenter to obtain the truncated input corpus with space characters includes: Based on the Bert word segmenter, special characters and control characters are deleted from the regularized truncated input corpus to obtain the deleted truncated input corpus; Obtaining the tab character of the deleted truncated input corpus, the carriage return character of the deleted truncated input corpus, and the line feed character of the deleted truncated input corpus, and replacing them with spaces to obtain preliminary truncated input corpus with space characters; The CJK characters of the deleted truncated input corpus are obtained, and a space is inserted after the CJK characters to obtain the truncated input corpus with the space character.

5. The method according to claim 1, characterized in that The position encoding layer based on the semantic integrity judgment model performs word embedding and position encoding processing on the preprocessed input corpus to obtain an embedding vector of the input corpus, including: Inputting the preprocessed input corpus into the position encoding layer of the semantic integrity judgment model; Based on the embedding layer of the position encoding layer, performing word embedding processing on the preprocessed input corpus to obtain embedded input corpus; Based on the encoding layer of the position encoding layer, position encoding processing is performed on the embedded input corpus to obtain an embedding vector of the input corpus.

6. The method according to claim 1, characterized in that The loss function calculation layer of the semantic integrity judgment model includes a cross entropy loss function and a triple loss function. The expression of the loss function calculation layer is specifically as follows: L=β·L CE +λ·L TP L TP =max(0,m+||f(x a )-f(x p )|| 2 -||f(x a )-f(x n )|| 2 ) In the above formula, L represents the comprehensive loss function of the loss function calculation layer, L CE represents the cross entropy loss function, L TP represents the triplet loss function, β represents the weight parameter of the cross entropy loss function, λ represents the weight parameter of the triplet loss function, N represents the training batch size, w represents the weight of the fully connected layer, x represents the input feature data, y i represents the probability of the i-th category, i and j represent the sequence numbers of samples in each batch, a represents the anchor sample, p represents the positive sample, n represents the negative sample, and m represents the boundary parameter.

7. A semantic integrity judgment system based on a multi-head attention mechanism, characterized in that: The system comprises: A truncation module is used to perform sentence truncation processing on the input corpus based on a preset semantic truncation mechanism to obtain truncated input corpus; A preprocessing module, used for performing data preprocessing on the truncated input corpus through a Bert word segmenter to obtain preprocessed input corpus; The construction module is used to introduce the masked multi-head attention sub-layer and the multi-head attention mechanism module to build a semantic integrity judgment model; The semantic integrity judgment model includes a position encoding layer, a masked multi-head attention sublayer, a normative layer, a multi-head attention mechanism module, a forward feedback layer, a linear layer and a loss function calculation layer, wherein: The position coding layer includes an embedding layer and a coding layer; The multi-head attention mechanism module includes a segmentation module, several attention head modules, a self-attention map calculation module and a mapping module; A judgment module, used for inputting the preprocessed input corpus into the semantic integrity judgment model; Based on the position encoding layer of the semantic integrity judgment model, the preprocessed input corpus is subjected to word embedding and position encoding processing to obtain an embedding vector of the input corpus; Based on the masked multi-head attention sublayer of the semantic integrity judgment model, calculating the attention feature value of the embedding vector of the input corpus to obtain the feature value of the first input corpus vector; Based on the standardization layer of the semantic integrity judgment model, the first input corpus vector feature value is standardized to obtain a standardized first input corpus vector feature value; Inputting the embedding vector of the input corpus into the multi-head attention mechanism module of the semantic integrity judgment model; Based on the segmentation module of the multi-head attention mechanism module, the embedding vector of the input corpus is segmented to obtain several embedding sub-vectors of the input corpus; Based on the multiple attention head modules and the self-attention graph calculation module of the multi-head attention mechanism module, multi-head attention calculation is performed on the embedding sub-vectors of the multiple input corpora to obtain the embedding sub-vector feature values ​​of the multiple input corpora; Based on the mapping module of the multi-head attention mechanism module, a connection mapping process is performed on the embedding sub-vector feature values ​​of the plurality of input corpora to obtain a second input corpus vector feature value; Combining the standardized first input corpus vector feature value with the second input corpus vector feature value to obtain an input corpus vector feature value; Based on the forward feedback layer of the semantic integrity judgment model, the input corpus vector feature value is subjected to nonlinear transformation processing to obtain the input corpus vector feature value after nonlinear transformation; Based on the linear layer of the semantic integrity judgment model, feature scaling is performed on the input corpus vector feature value after the nonlinear transformation to obtain the scaled input corpus vector feature value; Based on the loss function calculation layer of the semantic integrity judgment model, a weight distribution calculation is performed on the scaled input corpus vector feature value to obtain the input corpus integrity judgment result.

Images