DMPE-based semantic recognition method and device for improving self-attention mechanism
By introducing an improved self-attention mechanism with diagonal matrix position encoding, the problem of insufficient position information perception in the Transformer model during long sequence processing is solved, improving the model's performance and extrapolation ability, especially in the semantic recognition of long text sequences.
Patent Information
- Application Number
- CN202511631582.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-10
- Publication Date
- 2026-02-10
AI Technical Summary
Standard self-attention mechanisms cannot perceive the inherent order of input elements in a sequence or spatial arrangement, which limits the performance of Transformer models when processing long sequences, and existing positional encoding methods perform poorly in extrapolation.
An improved self-attention mechanism based on DMPE is introduced. The symmetry of positional interaction is restored by diagonal matrix positional encoding. Sine positional encoding vectors and preset query and key mapping matrices are used to calculate the attention scores between tokens and redesign the output sequence processing path.
It significantly improves the performance and extrapolation ability of the Transformer model on long sequences, ensures accurate modeling of relative position information, and enhances the model's generalization ability and hardware running speed.
Smart Images
Figure CN121503490A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence, and in particular to a semantic recognition method and apparatus based on an improved self-attention mechanism of DMPE. Background Technology
[0002] The Transformer architecture has recently achieved revolutionary success in numerous fields, including natural language processing and computer vision, and has become the cornerstone of many state-of-the-art models. Its core component—the self-attention mechanism—enables Transformer models to effectively capture long-range dependencies within data and is widely considered key to its superior performance. However, a fundamental challenge lies in the permutation invariance of standard self-attention mechanisms. That is, it treats all input elements equally, thus failing to perceive their inherent order in sequence or spatial arrangement. In most tasks, this "blindness" to positional information is detrimental because the order of elements carries crucial structural information. For example, in images, the spatial layout of pixels defines the shape of objects and the composition of scenes; in language, the sequence of words determines grammatical structure and semantics. Therefore, designing a robust and efficient improved self-attention mechanism based on Diagonal Matrix Positional Encoding (DMPE) is an indispensable part of the Transformer architecture.
[0003] To address this issue, the research community has proposed various positional encoding schemes, broadly categorized into two main types: absolute positional encoding and relative positional encoding. In its pioneering work, the initial Transformer employed a fixed sinusoidal absolute positional encoding, while subsequent research explored learnable absolute positional encoding. Despite their simplicity, these methods have significant limitations. Firstly, they struggle to generalize to inputs longer than the sequences seen during training, limiting the scalability of the Transformer model. Secondly, they fail to explicitly model the relative spatial relationships between elements, information that is often more critical than absolute coordinates in many visual and language tasks. To more directly incorporate relative positional information, much subsequent research has turned to relative positional encoding. These methods introduce a bias term dependent on the relative distance between elements into the self-attention computation, achieving significant performance improvements on numerous benchmarks. However, relative positional encoding methods in these techniques generally perform poorly in extrapolation, especially when dealing with long sequences as input to large language models. Therefore, designing a robust positional encoding paradigm with strong generalization capabilities for variable and longer input lengths remains an open and crucial research challenge. Summary of the Invention
[0004] The purpose of this application is to propose a semantic recognition method based on an improved self-attention mechanism using DMPE. This method mathematically restores the symmetry of positional interactions, making the position-to-position interaction terms in the attention score a pure function of relative distance. This significantly improves the performance and extrapolation ability of the Transformer model when processing ultra-long sequences. It overcomes the shortcomings of related technologies where unconstrained linear projection destroys the inherent structure of positional encoding, leading to distortion of relative distance signals and thus limiting the length extrapolation ability of the Transformer model. Furthermore, its application in computer hardware can significantly improve the operating speed of the hardware.
[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a semantic recognition method based on an improved self-attention mechanism using DMPE, comprising: The natural language is segmented into words to obtain m word elements, and a position index is assigned to each word element. Based on the position index, a sinusoidal position encoding vector is generated for each word element. ; The interaction matrix is determined based on the preset query mapping matrix and the preset key mapping matrix; Calculate the eigenvalues of the interaction matrix, and determine the diagonal matrix based on the eigenvalues and the preset linear layer weight matrix; For the i-th word, the attention score of the word to the j-th word is determined based on the sinusoidal position encoding vector of the word, the sinusoidal position encoding vector of the j-th word, and the diagonal matrix. ; ; Based on the attention score of each word and a preset value mapping matrix, the output sequence of the word is determined; Based on the output sequence of each word, a semantic representation vector is obtained, and semantic recognition is performed based on the semantic representation vector.
[0006] Secondly, this application provides a semantic recognition device based on an improved self-attention mechanism using DMPE, comprising: The sinusoidal positional encoding vector acquisition module is used to segment natural language into words, obtain m word elements, assign a position index to each word element, and generate a sinusoidal positional encoding vector for each word element based on the position index. ; The diagonal matrix acquisition module is used to determine the interaction matrix based on a preset query mapping matrix and a preset key mapping matrix; calculate the eigenvalues of the interaction matrix; and determine the diagonal matrix based on the eigenvalues and a preset linear layer weight matrix. The attention score acquisition module is used to determine the attention score of the i-th word based on the sinusoidal position encoding vector of the i-th word, the sinusoidal position encoding vector of the j-th word, and the diagonal matrix. ; ; The output sequence acquisition module is used to determine the output sequence of the word based on the attention score of each word and a preset value mapping matrix; The semantic recognition module is used to obtain a semantic representation vector based on the output sequence of each word, and to perform semantic recognition based on the semantic representation vector.
[0007] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a semantic recognition method and apparatus based on the improved self-attention mechanism of DMPE. By introducing a diagonal matrix into the sinusoidal positional encoding vector of two words, diagonal matrix positional encoding is performed. The diagonal matrix positional encoding adopts an unstructured interaction method and redesigns the processing path of the output sequence of words, enabling the Transformer model to exhibit stronger extrapolation performance in long context scenarios.
[0008] In addition, the diagonal matrix derived from the spectral properties of the preset query mapping matrix and the preset key mapping matrix maintains and restores mathematical symmetry, ensuring that the attention score becomes a function of the relative distance between elements. Through more accurate relative position modeling, the diagonal matrix position encoding significantly improves the Transformer model's ability to generalize to longer sequences. Attached Figure Description
[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0010] Figure 1 This is an application environment diagram of a semantic recognition method based on an improved self-attention mechanism of DMPE according to an embodiment of this application; Figure 2 A flowchart illustrating a semantic recognition method based on an improved self-attention mechanism using DMPE, provided as an embodiment of this application; Figure 3 A schematic diagram of the functional modules of a semantic recognition device based on an improved self-attention mechanism of DMPE provided in an embodiment of this application; Figure 4This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0012] The Transformer architecture uses scaled dot product attention to compute relationships between lexical units. Let the Transformer model dimension be represented as... Given a set of query vectors Key vector Sum value vector The output sequence for each word is determined using the following formula: .
[0013] in, Let be the dimension of the key vector. For transpose operation, is the activation function. Because this formula is permutation invariant, it is insensitive to the order of lexical units in natural language.
[0014] To address this limitation, a sinusoidal positional encoding vector is introduced. For a word located at position pos and with dimension n, its sinusoidal positional encoding vector... Defined by sine and cosine functions of different frequencies, specifically, the sine position encoding vector for each word is determined using the following formula: .
[0015] .
[0016] in, For position pos at the th Sine positional encoding vector in dimension For position pos at the th Sine positional encoding vector in dimension pos is the dimension index, and pos is the location index. This represents both the embedding dimension of the self-attention mechanism and the Transformer model dimension.
[0017] These fixed, non-learned encodings provide a unique representation for each position and allow the Transformer model to learn relative positional relationships, since the encoding at any position can be represented as a linear function of the encoding at any other position. In the standard Transformer model, these encodings are added to the word embedding vectors before being projected onto the query mapping matrix and key mapping matrix.
[0018] Specifically, for the i-th word element, we obtain the word embedding vector and the sinusoidal position encoding vector of the i-th word element. Based on the word embedding vector and the sinusoidal position encoding vector of the i-th word element, we obtain the final representation of the i-th word element: = + Similarly, the final representation of the j-th word is obtained: = + .in, ; ; ; m is the number of tokens. Based on the final representation of the i-th token, the final representation of the j-th token, and the preset query mapping matrix. and the preset key mapping matrix Determine the group query vector for the i-th term. Key vector Sum value vector ; the group query vector of the j-th word j Key vector K j Sum vector V j Therefore, the attention score of the i-th word to the j-th word is: .
[0019] in, For content items that overlap, For items where content and location intersect, For items where location and content intersect, For the intersection of positions.
[0020] In summary, the independent linear projections applied to the query and key vectors during the self-attention mechanism's processing produce an unstructured transformation, fundamentally disrupting the inherent triangular symmetry of the position vectors. The result is that the Transformer model receives a distorted signal of the distance between two positions, rather than a clear, consistent representation of their relative offsets. This subtle but significant distortion hinders the Transformer model's generalizable understanding of distances, a limitation particularly detrimental to length extrapolation.
[0021] To address this fundamental asymmetry problem, this application introduces Diagonal Matrix Positional Encoding (DMPE). Instead of employing an unstructured interaction method, DMPE redesigns the processing path of the word output sequence by introducing a carefully constrained diagonal matrix. This diagonal matrix is derived from the spectral properties of the original projection matrix, and its special structure aims to preserve and restore mathematical symmetry.
[0022] To make the objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0023] The semantic recognition method based on the improved self-attention mechanism of DMPE provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send the natural language to be processed to server 104. Server 104 performs word segmentation on the natural language, obtaining n tokens, and generates a sinusoidal positional encoding vector for each token; determines a diagonal matrix based on a preset query mapping matrix, a preset key mapping matrix, and a preset linear layer weight matrix; determines the attention score of the i-th token to the j-th token based on the diagonal matrix; determines the output sequence of the token based on the attention score of the i-th token to each token and a preset value mapping matrix; obtains a semantic representation vector based on the output sequence of each token, and performs semantic recognition based on the semantic representation vector.
[0024] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.
[0025] In one exemplary embodiment, such as Figure 2 As shown, a semantic recognition method based on an improved self-attention mechanism using DMPE is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 208. Wherein: Step 201: Segment the natural language to obtain m word elements, assign a position index to each word element, and generate a sinusoidal position encoding vector for each word element based on the position index; 1.
[0026] Step 202: Determine the interaction matrix based on the preset query mapping matrix and the preset key mapping matrix; Calculate the eigenvalues of the interaction matrix, and determine the diagonal matrix based on the eigenvalues and a preset linear layer weight matrix. Specifically, query the mapping matrix. Key mapping matrix The linear layer weight matrix F∈R (d / 2)×d DMPE diagonal matrix Λ∈R d×d Where d is the dimension of the Transformer model, d k This represents the dimension of the key vector. Based on the preset query mapping matrix... and the preset key mapping matrix Determine the interaction matrix M: M = .
[0027] Step 203: For the i-th word, based on the sinusoidal position encoding vector of the word, the sinusoidal position encoding vector of the j-th word, and the diagonal matrix, determine the attention score of the word to the j-th word. ; .
[0028] Step 204: Based on the attention score of each word and the preset value mapping matrix, determine the output sequence of the word.
[0029] Step 205: Based on the output sequence of each word, obtain the semantic representation vector, and perform semantic recognition based on the semantic representation vector.
[0030] Implementing steps 201 to 205 above enhances the extrapolation performance of the Transformer model in long-context scenarios. Specifically, this application uses a carefully constrained diagonal matrix when calculating the attention score. To replace the unstructured matrix in the position and position intersection terms Following existing research on relative position representation, such as Decoding-enhanced BERT with disentangled attention (DeBERTa) and Text-to-Text Transfer Transformer (T5), this approach omits the content-position interaction and position-content interaction terms. These terms create a complex dependency where the content of one lexical term is influenced by the absolute position of another. By removing these terms, the attention score becomes a clear sum of two independent components: a purely semantic relevance assessment of the content-content interaction and a purely geometric bias assessment of the position-position interaction. This prevents the Transformer model from learning spurious correlations between content and absolute position, allowing the position component to focus entirely on modeling relative distance. Through more accurate relative position modeling, DMPE significantly improves the Transformer model's ability to generalize to longer sequences. This results in stronger extrapolation performance in long-context scenarios.
[0031] The core assumption of this application is that it originates from... The key location information can be represented by a constrained diagonal matrix. Effective capture. This is a reasonable approximation based on the finding that weight matrices in large models are typically efficient low-rank matrices, and their updates during training occur in a low-dimensional subspace. This suggests that the transformation is primarily determined by its principal components, and the goal of this application is precisely to capture the diagonal elements. These principal components are captured in the process.
[0032] In another exemplary embodiment of this application, the advantage of the DMPE of this application lies in its versatility. At the same time, the DMPE framework itself opens up a new design space, providing a solid foundation for future research into alternative or even superior methods for determining diagonal weights (e.g., methods based on different matrix factorization or regularization techniques). Step 202 above is replaced by steps 301-304: Step 301: Sort the feature values by size to obtain sorted feature values.
[0033] Step 302: Determine the initial weight matrix based on the real parts of the sorted eigenvalues.
[0034] Step 303: Determine the second initial weight matrix based on the initial weight matrix and the preset linear layer weight matrix.
[0035] Step 304: Determine the diagonal matrix based on the second initial weight matrix. Specifically, for the second initial weight matrix A∈R... d / 2 The a-th element is used to determine the third initial weight matrix B∈R. d The 2a-1th element and the 2ath element. Based on the third initial weight matrix, a function for constructing a diagonal matrix is used. Determine the diagonal matrix.
[0036] The spectrum (i.e., the set of eigenvalues in a diagonal matrix) captures its fundamental properties such as scaling and rotation. Inspired by the success of spectral methods in graph neural networks and the insight in random matrix theory that learned information resides in the principal eigenvalues, it is assumed that... The spectrum provides a rich and standardized representation of position and position intersection terms.
[0037] In this model, the initial set of feature values is an unordered collection of numerical values. There is no inherent correspondence between specific feature values and specific frequency dimensions of positional encoding. A simple fixed mapping, such as direct pairing or averaging, is semantically unfounded. To bridge this gap, a small, learnable linear layer is introduced. Come on The sorting spectrum is mapped to the diagonal weights This enables the Transformer model to learn the optimal allocation between the core spectral properties of the transform and the frequency basis of the position encoding.
[0038] In this hybrid approach, eigenvalue decomposition provides interactive principal components. A stable and ordered representation. Then, from the linear layer weight matrix... Parameterized learnable linear layers It acts as an attention mechanism for the spectrum, learning how to weight and combine eigenvalues to produce the most efficient set of frequency weights. This spectral-guided approach combines a powerful, physics-inspired spectrum with the flexibility of learnable mappings, creating a robust and high-performance positional encoding method. A learnable layer is introduced. Only increased There are only a few parameters, which is a negligible number in modern large-scale models.
[0039] In another exemplary embodiment of this application, the method further includes, after assigning a position index to each lexical unit, mapping each lexical unit to a word embedding vector. Step 203 specifically includes: For the i-th word, based on the word embedding vector of the word, the preset query mapping matrix, the preset key mapping matrix, and the word embedding vector of the j-th word, the content-content cross vector is determined.
[0040] Based on the sinusoidal position encoding vector of the word, the sinusoidal position encoding vector of the j-th word, and the diagonal matrix, the diagonal matrix position encoding vector is determined.
[0041] The attention score of the word to the j-th word is determined based on the content-content cross vector and the diagonal matrix position encoding vector.
[0042] In another exemplary embodiment of this application, the attention score of the i-th word to the j-th word is determined using the following formula: .
[0043] in, Let be the attention score of the i-th word to the j-th word, where i and j are the indices of the word. Let be the word embedding vector of the i-th word element. This is a preset query mapping matrix. For the preset key mapping matrix, Let j be the word embedding vector of the j-th word element. For transpose operation, Let i be the sinusoidal position encoding vector of the i-th word. Let j be the sinusoidal position encoding vector of the j-th word. It is a diagonal matrix. For content items that overlap, For items where positions intersect, This is a positional encoding for a diagonal matrix.
[0044] In another exemplary embodiment of this application, in order to satisfy the symmetry constraints derived for all frequency pairs, The constraint is a diagonal matrix where each diagonal element is paired and equal to another element. .
[0045] in, It is a diagonal matrix. A function to construct a diagonal matrix. Let be the first eigenvalue of the diagonal line in the diagonal matrix. It is the second eigenvalue of the diagonal line in the diagonal matrix. The first diagonal line of a diagonal matrix There are eigenvalues, where d is the dimension of the diagonal matrix.
[0046] Under the above constraints, the diagonal matrix position encoding vector is determined using the following formula, and the position and position intersection terms recover the relative distance. Dependency:
[0047] .
[0048] in, Let i be the sinusoidal position encoding vector of the i-th word. Let j be the sinusoidal position encoding vector of the j-th word. For transpose operation, This is another representation of a diagonal matrix, where i and j are the indices of the tokens. Let be the dimension of the diagonal matrix. For dimensional indexing, The first of the diagonal matrices 1 eigenvalue, This is a frequency parameter.
[0049] This principle-based design of position and position intersection terms ensures that the Transformer model receives clear, distortion-free relative offset signals. The learned weights... This can be interpreted as being assigned to each frequency component. The importance of distance scales enables the Transformer model to adaptively focus on the distance scales most relevant to a given task.
[0050] In another exemplary embodiment of this application, step 204 specifically includes: The attention scores of each word to each word are normalized to obtain the attention coefficient of each word to each word.
[0051] The value vector is determined based on the word embedding vector of the lexical, the sinusoidal position encoding vector of the lexical, and the preset value mapping matrix.
[0052] The output sequence of the lexical is determined based on the attention coefficient of each lexical and the value vector.
[0053] Based on the same inventive concept, this application also provides a semantic recognition device based on the DMPE-improved self-attention mechanism for implementing the semantic recognition method based on the DMPE-improved self-attention mechanism described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the semantic recognition device based on the DMPE-improved self-attention mechanism provided below can be found in the limitations of the semantic recognition method based on the DMPE-improved self-attention mechanism described above, and will not be repeated here.
[0054] In one exemplary embodiment, such as Figure 3 As shown, a semantic recognition device based on an improved self-attention mechanism using DMPE is provided, comprising: The sinusoidal positional encoding vector acquisition module 301 is used to segment natural language into words, obtain m word elements, assign a position index to each word element, and generate a sinusoidal positional encoding vector for each word element based on the position index. .
[0055] The diagonal matrix acquisition module 302 is used to determine the interaction matrix based on a preset query mapping matrix and a preset key mapping matrix; calculate the eigenvalues of the interaction matrix; and determine the diagonal matrix based on the eigenvalues and a preset linear layer weight matrix.
[0056] Attention score acquisition module 303 is used to determine the attention score of the i-th word based on the sinusoidal position encoding vector of the word, the sinusoidal position encoding vector of the j-th word, and the diagonal matrix. ; .
[0057] The output sequence acquisition module 304 is used to determine the output sequence of the word based on the attention score of each word and a preset value mapping matrix.
[0058] The semantic recognition module 305 is used to obtain a semantic representation vector based on the output sequence of each word, and to perform semantic recognition based on the semantic representation vector.
[0059] In one exemplary embodiment, a series of comparative language modeling experiments were conducted to verify the beneficial effects of this application. All experiments were performed on a unified hardware and model architecture to ensure fairness in the comparison. The experimental setup was as follows: a dedicated Transformer model with 125M parameters was used, comprising 16 layers, 768 hidden dimensions, and 12 attention heads. Furthermore, the Transformer model was pre-trained for 50,000 steps on the Arxiv and Books3 datasets using sequences of length 512. Arxiv is an academic paper abstracting dataset commonly used for text generation tasks. Books3 is a text corpus consisting of a large number of books, used for language modeling. Finally, regarding the hardware platform: all experiments were performed on a single NVIDIA RTX A6000 GPU with 48GB of VRAM.
[0060] After pre-training, the Transformer model was evaluated using zero-shot tests on sequences ranging from 512 to 8192 bytes in length, with perplexity used as the metric to test its length extrapolation capability. Table 1 shows the comparison results between our proposed Diagonal Matrix Positional Encoding (DMPE) and two mainstream baseline methods: Sinusoidal Positional Encoding (Sinusoidal PE) and Rotary Position Embedding (ROPE). Table 1
[0061] This application significantly improves the length extrapolation capability of Transformer models, reaching an industry-leading level: by mathematically ensuring that the relative distance signal is not distorted, it exhibits strong robustness when processing text sequences that far exceed the training length.
[0062] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores natural language. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a semantic recognition method based on a DMPE-improved self-attention mechanism.
[0063] Those skilled in the art will understand that Figure 4The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.
[0064] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0065] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).
[0066] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0067] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0068] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A semantic recognition method based on an improved self-attention mechanism using DMPE, characterized in that, The method includes: The natural language is segmented into words to obtain m word elements, and a position index is assigned to each word element. Based on the position index, a sinusoidal position encoding vector is generated for each word element. ; The interaction matrix is determined based on the preset query mapping matrix and the preset key mapping matrix; Calculate the eigenvalues of the interaction matrix, and determine the diagonal matrix based on the eigenvalues and the preset linear layer weight matrix; For the i-th word, the attention score of the word to the j-th word is determined based on the sinusoidal position encoding vector of the word, the sinusoidal position encoding vector of the j-th word, and the diagonal matrix. ; ; Based on the attention score of each word and a preset value mapping matrix, the output sequence of the word is determined; Based on the output sequence of each word, a semantic representation vector is obtained, and semantic recognition is performed based on the semantic representation vector.
2. The semantic recognition method based on the improved self-attention mechanism of DMPE according to claim 1, characterized in that, The diagonal matrix is determined based on the eigenvalues and the preset linear layer weight matrix, specifically including: The feature values are sorted by size to obtain sorted feature values; Determine the initial weight matrix based on the real parts of the sorted eigenvalues; Based on the initial weight matrix and the preset linear layer weight matrix, determine the second initial weight matrix; The diagonal matrix is determined based on the second initial weight matrix.
3. The semantic recognition method based on the improved self-attention mechanism of DMPE according to claim 1, characterized in that, The diagonal matrix is represented by the following expression: ； in, It is a diagonal matrix. A function for constructing a diagonal matrix. Let be the first eigenvalue of the diagonal line in the diagonal matrix. It is the second eigenvalue of the diagonal line in the diagonal matrix. The first diagonal line of a diagonal matrix There are eigenvalues, where d is the dimension of the diagonal matrix.
4. The semantic recognition method based on the improved self-attention mechanism of DMPE according to claim 1, characterized in that, After assigning a position index to each lexical unit, the method further includes: mapping each lexical unit to a word embedding vector; for the i-th lexical unit, determining the attention score of the lexical unit to the j-th lexical unit based on the sine position encoding vector of the lexical unit, the sine position encoding vector of the j-th lexical unit, and the diagonal matrix, specifically including: For the i-th word element, based on the word embedding vector of the word element, the preset query mapping matrix, the preset key mapping matrix, and the word embedding vector of the j-th word element, determine the content-content cross vector; Based on the sinusoidal position encoding vector of the word, the sinusoidal position encoding vector of the j-th word, and the diagonal matrix, determine the diagonal matrix position encoding vector; The attention score of the word to the j-th word is determined based on the content-content cross vector and the diagonal matrix position encoding vector.
5. The semantic recognition method based on the improved self-attention mechanism of DMPE according to claim 4, characterized in that, The attention score of the i-th word to the j-th word is determined using the following formula: in, Let be the attention score of the i-th word to the j-th word, where i and j are the indices of the words. Let be the word embedding vector of the i-th word element. This is a preset query mapping matrix. For the preset key mapping matrix, Let j be the word embedding vector of the j-th word element. For content items that overlap, For transpose operation, Let i be the sinusoidal position encoding vector of the i-th word. Let j be the sinusoidal position encoding vector of the j-th word. It is a diagonal matrix. For items where positions intersect, This is a positional encoding for a diagonal matrix.
6. The semantic recognition method based on the improved self-attention mechanism of DMPE according to claim 4, characterized in that, The diagonal matrix position encoding vector is determined using the following formula: ； in, Let i be the sinusoidal position encoding vector of the i-th word. Let j be the sinusoidal position encoding vector of the j-th word. For transpose operation, This is another representation of a diagonal matrix, where i and j are the indices of the tokens. Let be the dimension of the diagonal matrix. For dimensional indexing, The first of the diagonal matrices 1 eigenvalue, This is a frequency parameter.
7. The semantic recognition method based on the improved self-attention mechanism of DMPE according to claim 4, characterized in that, Based on the attention score of each word and a preset value mapping matrix, the output sequence of the word is determined, specifically including: The attention scores of each word to each word are normalized to obtain the attention coefficient of each word to each word. The value vector is determined based on the word embedding vector of the word, the sinusoidal position encoding vector of the word, and the preset value mapping matrix; The output sequence of the lexical is determined based on the attention coefficient of each lexical and the value vector.
8. The semantic recognition method based on the improved self-attention mechanism of DMPE according to claim 1, characterized in that, The sinusoidal positional encoding vector for each word is determined using the following formula: ； ； in, For position pos at the th Sine positional encoding vector in dimension For position pos at the th Sine positional encoding vector in dimension pos is the dimension index, and pos is the location index. This represents the embedding dimension of the self-attention mechanism.
9. The semantic recognition method based on the improved self-attention mechanism of DMPE according to claim 1, characterized in that, The output sequence for each word is determined using the following formula: ； in, For query vector, For key vectors, For value vectors, Let be the dimension of the key vector. For transpose operation, This is the activation function.
10. A semantic recognition device based on a DMPE-improved self-attention mechanism, employing the semantic recognition method based on the DMPE-improved self-attention mechanism as described in any one of claims 1-9, the device comprising: The sinusoidal positional encoding vector acquisition module is used to segment natural language into words, obtain m word elements, assign a position index to each word element, and generate a sinusoidal positional encoding vector for each word element based on the position index. ; The diagonal matrix acquisition module is used to determine the interaction matrix based on the preset query mapping matrix and the preset key mapping matrix; Calculate the eigenvalues of the interaction matrix, and determine the diagonal matrix based on the eigenvalues and the preset linear layer weight matrix; The attention score acquisition module is used to determine the attention score of the i-th word based on the sinusoidal position encoding vector of the i-th word, the sinusoidal position encoding vector of the j-th word, and the diagonal matrix. ; ; The output sequence acquisition module is used to determine the output sequence of the word based on the attention score of each word and a preset value mapping matrix; The semantic recognition module is used to obtain a semantic representation vector based on the output sequence of each word, and to perform semantic recognition based on the semantic representation vector.