Controllable attention method, system, and media based on feature domain partitioning

Abstract

The application provides a controllable attention method and system based on feature domain division and a medium. Local attention output of a local attention head and global attention output of a global attention head are calculated based on an attention matrix, and then the local attention output and the global attention output are fused to obtain an attention output result. By introducing feature domain division, fine control of text features is realized, the computational complexity is reduced, and the controllability and quality of generated text are improved. Local attention helps to focus on the feature area of the text, and global attention tries to capture more extensive context information. The combination of the two can significantly reduce the computational complexity, improve the controllability and quality of the generated content, and improve the harmony and rationality of the generated content.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and relates to a method for calculating attention in natural language text processing and music processing of artificial intelligence, more specifically, relates to a controllable attention method and system based on feature domain division and a medium. BACKGROUND

[0002] In the field of natural language processing, text generation and music generation, modeling and controllable generation technology has emerged a large number of deep learning-based models and methods after years of development. At present, the technology of AI (Artificial Intelligence) generated text mainly relies on a model named 'Transformer'. It can pay attention to every detail of the whole text like a human being, but the disadvantage is that the calculation is large. In order to solve this problem, the improved model (such as Transformer-XL, Sparse Transformer, Transformer-LS) tries to reduce the attention range, but may miss important information.

[0003] Firstly, the Transformer model (Transformer, 2017) adopts global self-attention mechanism, which can capture the dependency between any two positions in the sequence, and has made breakthroughs in machine translation, text summarization and other tasks. However, the computational complexity of global self-attention is quadratic to the sequence length, which greatly increases the computational cost and memory occupation when processing long text and large-scale data, limiting the scalability and real-time generation ability of the model. Even after introducing improvements such as position encoding, the modeling of long sequence still faces the problem of computational bottleneck.

[0004] The later long-distance dependency modeling method based on cycle mechanism prolongs the memory range by introducing fragment overlap and relative position encoding, thereby alleviating the high overhead problem of global attention calculation to a certain extent. Although this method has made progress in capturing long-term dependencies, it still relies on the basic framework of global attention, and still faces high computational cost and difficulty in fine control of generated content when generating large-scale text.

[0005] In view of the above problems, by designing the sparsity of attention matrix, the dependency between some key positions is selectively calculated, thereby significantly reducing the computational complexity. This method solves the efficiency problem of long sequence text generation to a certain extent, but in specific application, the design of sparse mode often needs to make a trade-off between information integrity and computational efficiency, which is easy to appear the situation of insufficient global context capture and incoherent generated content.

[0006] At the same time, the local attention model, by dividing the input sequence into fixed or dynamic windows, only performs attention calculation within a local range, thereby greatly reducing the computational cost. Although local attention has obvious advantages in efficiency over traditional global attention, it has the disadvantage of limiting the model's ability to capture long-distance dependencies, which can cause the generated text to break down in long-distance information association, thereby affecting the coherence and consistency of the overall semantics.

[0007] In addition, axial attention has shown good results in image generation and language modeling, but in language modeling tasks, text data inherently has more complex temporal and contextual dependencies. Simple axial decomposition may not be able to fully capture the semantic information across different positions and levels in the text, and may also cause problems such as mismatch between local information and global semantics when introducing control conditions.

[0008] In addition, in recent years, controllable generation methods based on pre-trained models (such as GPT-2 / CTRL, etc.) often have two main problems in practical operation: first, the fusion of control signals and intrinsic knowledge of language models is not close enough, resulting in generated content that may lose the fluency of natural language while meeting specific control requirements; second, with the continuous expansion of model size, how to balance generation efficiency and controllability while ensuring generation quality remains a difficult problem to be solved.

[0009] In general, the current attention calculation methods have at least the following technical problems:

[0010] (1) High computational resource consumption: Traditional Transformer models use global self-attention mechanisms, with computational complexity proportional to the square of the sequence length, resulting in significant computational resource challenges when processing long text.

[0011] (2) Insufficient capture of long-distance dependencies: Although Transformer-XL addresses the long-distance dependency problem to some extent by introducing segment overlap and relative position encoding, its overall framework still relies on global attention calculation, making it difficult for the model to balance the capture of local details and global semantics when generating very long text.

[0012] (3) Information distribution imbalance: Sparse Transformer and local attention methods reduce computational costs, but due to the limitations of attention calculation range, some key long-distance dependency information may be missed, affecting the coherence and semantic integrity of the generated text.

[0013] (4) Multi-dimensional information fusion problem: Although the axial attention improves the calculation efficiency when decomposing and calculating multi-dimensional data, in the language generation task, how to effectively fuse the temporal information, syntax structure and semantic content in the text to ensure that the generated results are consistent with both local details and overall consistency is still a big challenge.

[0014] (5) Balance between controllability and generation quality: The controllable generation method based on control code is prone to cause the generated content to be abrupt or unnatural when achieving specific control targets, especially when abstract attributes such as emotion and style are involved. The contradiction between the control signal and the intrinsic semantics of the model is difficult to reconcile.

[0015] Therefore, there is an urgent need for a controllable attention scheme based on feature domain division that can accelerate the convergence and reasoning of the model in language modeling tasks, while improving the controllability and quality of the generated content, and improving the harmony and rationality of the generated content. SUMMARY

[0016] In view of the above problems, the purpose of the present application is to provide a controllable attention method based on feature domain division, system and medium, to solve the technical problems of high computational resource consumption, insufficient long-distance dependency capture, unbalanced information distribution, difficult multi-dimensional information fusion, and poor balance between controllability and generation quality in the prior art attention calculation method.

[0017] In a first aspect, the embodiments of the present application provide a controllable attention method based on feature domain division, which comprises:

[0018] The original data is subjected to linear layer and position embedding processing to form a word vector representation, and an attention matrix is obtained based on the word vector representation;

[0019] The preset multi-head attention is divided to obtain a local attention head and a global attention head, and a local attention output of the local attention head and a global attention output of the global attention head are calculated based on the attention matrix;

[0020] The local attention output and the global attention output are fused to obtain an attention output result.

[0021] Optionally, the original data is subjected to linear layer and position embedding processing to form a word vector representation, and an attention matrix is obtained based on the word vector representation, comprising:

[0022] Obtain the event in the original data;

[0023] Each event is subjected to feature domain division corresponding to the attribute quantity of the event to obtain attribute information of at least two dimensions;

[0024] obtain attribute embedding matrices corresponding to respective attribute information;

[0025] obtain embedding vectors of respective attribute information according to the embedding matrices;

[0026] concatenate the embedding vectors to obtain a comprehensive representation of the event, and stack comprehensive representations of respective events to obtain a structured representation of the original data;

[0027] combine the structured representation and pre-obtained position encoding to obtain a hidden layer state, and obtain an attention matrix based on the hidden layer state.

[0028] Optionally, obtaining the attention matrix based on the hidden layer state comprises:

[0029] perform linear transformation on the hidden layer state based on a preset projection matrix to obtain a query matrix, a key matrix and a value matrix;

[0030] divide the query matrix, the key matrix and the value matrix into a preset number of sub-matrices respectively, and take the sub-matrices of the query matrix, the sub-matrices of the key matrix and the sub-matrices of the value matrix as the attention matrix; wherein the preset number is a total number h of heads of the multi-head attention.

[0031] Optionally, dividing the preset multi-head attention to obtain local attention heads and global attention heads comprises:

[0032] obtaining a total number h of heads of the multi-head attention; wherein,

[0033] if the total number h of heads is even, taking attention heads numbered 1 to h / 2 as global attention heads, and taking attention heads numbered h / 2+1 to h as local attention heads;

[0034] if the total number h of heads is odd, taking attention heads numbered 1 to (h+1) / 2 as global attention heads, and taking attention heads numbered (h+3) / 2 to h as local attention heads.

[0035] Optionally, calculating a local attention output of the local attention head based on the attention matrix comprises:

[0036] dividing the local attention head into a specific number of categories of feature domain heads according to a preset indicator function; the specific number is the attribute quantity;

[0037] obtaining a feature attention score matrix based on the attention matrix, and causing the feature domain head to perform attention calculation on a specific attribute corresponding to the feature domain head to obtain an attention output according to the feature attention score matrix; wherein,

[0038] The attention outputs of the specific number of categories are spliced and fused to obtain a local attention output.

[0039] Optionally, a global attention output of the global attention head is calculated based on the attention matrix, including:

[0040] A global attention score matrix is obtained through the attention matrix;

[0041] A global attention output is calculated through the global attention head based on the global attention score matrix.

[0042] Optionally, after obtaining the attention output result, further comprising:

[0043] An output projection matrix is obtained through the attention output result;

[0044] The output projection matrix is linearly transformed to obtain mixed head information;

[0045] The mixed head information is layer normalized to obtain correction data.

[0046] In a second aspect, the embodiments of the present application provide a controllable attention system based on feature domain division, which implements the controllable attention method based on feature domain division as described above, and the system comprises:

[0047] An attention obtaining unit is configured to make the original data pass through a linear layer and a position embedding processing to form a word vector representation, and obtain an attention matrix based on the word vector representation;

[0048] A local-global segmentation unit is configured to divide a preset multi-head attention to obtain a local attention head and a global attention head, and calculate a local attention output of the local attention head and a global attention output of the global attention head based on the attention matrix;

[0049] An attention fusion unit is configured to fuse the local attention output and the global attention output to obtain an attention output result.

[0050] Optionally, the total number of heads of the multi-head attention is h; wherein,

[0051] If h is an even number, the attention heads numbered 1 to h / 2 are taken as the global attention heads, and the attention heads numbered h / 2+1 to h are taken as the local attention heads;

[0052] If h is an odd number, the attention heads numbered 1 to (h+1) / 2 are taken as the global attention heads, and the attention heads numbered (h+3) / 2 to h are taken as the local attention heads.

[0053] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any optional controllable attention method based on feature domain division described in the first aspect.

[0054] From the above technical solutions, it can be seen that the controllable attention method, system and computer-readable storage medium based on feature domain division provided by this application process the original data through a linear layer and position embedding processing to form a word vector representation, and after obtaining the attention matrix based on the word vector representation, the preset multi-head attention is divided to obtain a local attention head and a global attention head, and the local attention output of the local attention head and the global attention output of the global attention head are calculated based on the attention matrix, and then the local attention output and the global attention output are fused to obtain the attention output result. Compared with the prior art, this application has the following beneficial effects:

[0055] Taking music generation as an example, experimental comparisons have shown that chord accuracy, part overlap, perplexity, and value fluctuation are all superior to existing technologies. This allows for higher-quality generated music, closer to real music, while reducing uncertainty, achieving higher efficiency, and enabling faster inference. Simply put, by introducing feature domain partitioning, fine-grained control of text features is achieved, which reduces computational complexity while improving the controllability and quality of generated text. Local attention helps focus on feature regions of the text, while global attention attempts to capture broader contextual information. The combination of the two significantly reduces computational complexity while improving the controllability and quality of generated content, enhancing its harmony and rationality. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] By referring to the following description in conjunction with the accompanying drawings, and with a more complete understanding of the present invention, other objects and results of the present invention will become more apparent and easier to understand. In the accompanying drawings:

[0057] Figure 1 Flowchart of a controllable attention method based on feature domain division according to an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of a specific implementation of a controllable attention method based on feature domain division according to an embodiment of the present invention;

[0059] Figure 3 This is a logical block diagram of a controllable attention system based on feature domain division according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The attention calculation method in the prior art has at least the following technical problems:

[0061] (1) high computational resource consumption; (2) insufficient long-distance dependency capture; (3) uneven information distribution; (4) multi-dimensional information fusion problem; (5) balance between controllability and generation quality.

[0062] In view of the above problems, the application provides a controllable attention method based on feature domain division, a system and a computer readable storage medium. First, the original data is processed through a linear layer and a position embedding to form a word vector representation. After obtaining an attention matrix based on the word vector representation, the preset multi-head attention is divided to obtain a local attention head and a global attention head. The local attention output of the local attention head and the global attention output of the global attention head are calculated based on the attention matrix. Then, the local attention output and the global attention output are fused to obtain an attention output result. Compared with the prior art, the application has the following beneficial effects:

[0063] Taking music generation as an example, through experimental comparison, whether it is chord accuracy, section overlap, perplexity or value fluctuation, the generated music quality is higher and closer to real music, the uncertainty is reduced, the efficiency is higher, and the reasoning speed is faster. Simply put, by introducing feature domain division, fine control of text features is realized, which reduces the computational complexity while improving the controllability and quality of generated text. Local attention helps to focus on the feature area of the text, while global attention tries to capture more extensive context information. The combination of the two can significantly reduce the computational complexity while improving the controllability and quality of the generated content and improving the harmony and rationality of the generated content.

[0064] In order to enable personnel in the technical field to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all embodiments. The description of the following exemplary embodiments is actually only illustrative, but not as any limitation on the present application and its applications or uses. The techniques and equipment known to those skilled in the related art can not be discussed in detail, but should be considered as part of the specification under appropriate circumstances. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of the present application.

[0065] Figure 1 A flowchart of a controllable attention method based on feature domain division provided by the embodiments of the present application is shown in Figure 1 The method comprises:

[0066] S1: making the original data pass through a linear layer and a position embedding processing to form a word vector representation, and obtaining an attention matrix based on the word vector representation;

[0067] S2: dividing a preset multi-head attention to obtain a local attention head and a global attention head, and calculating a local attention output of the local attention head and a global attention output of the global attention head based on the attention matrix;

[0068] S3: attention fusion of the local attention output and the global attention output to obtain an attention output result.

[0069] Specifically, step S1 is a process of making the original data pass through a linear layer and a position embedding processing to form a word vector representation, and obtaining an attention matrix based on the word vector representation.

[0070] In the embodiment, making the original data pass through a linear layer and a position embedding processing to form a word vector representation, and obtaining an attention matrix based on the word vector representation, comprises:

[0071] S11: obtaining an event in the original data;

[0072] S12: dividing each event into a feature domain corresponding to an attribute quantity of the event to obtain attribute information of at least two dimensions;

[0073] S13: obtaining an attribute embedding matrix corresponding to each attribute information;

[0074] S14: obtaining an embedding vector of each attribute information according to the embedding matrix;

[0075] S15: splicing the embedding vectors to obtain a comprehensive representation of the event, and stacking the comprehensive representations of each event to obtain a structured representation of the original data;

[0076] S16: combining the structured representation and a pre-obtained position encoding to obtain a hidden layer state, and obtaining an attention matrix based on the hidden layer state.

[0077] Obtaining an attention matrix based on the hidden layer state comprises:

[0078] S161: linearly transforming the hidden layer state based on a preset projection matrix to obtain a query matrix, a key matrix and a value matrix;

[0079] S162: dividing the query matrix, the key matrix and the value matrix into a preset number of sub-matrices respectively, and taking the sub-matrices of the query matrix, the sub-matrices of the key matrix and the sub-matrices of the value matrix as the attention matrix; wherein the preset number is the total number h of heads of the multi-head attention.

[0080] like Figure 2 As shown, in a specific embodiment, it is assumed that the original data (original language data, original music data, original text data, etc.) includes T events, each containing n attributes, corresponding to n feature domains.

[0081] Let each event i The original representation of is:

[0082] ;

[0083] The meaning of each attribute is as follows:

[0084] : No. i Information about the first attribute in an event;

[0085] : No. i The information of the second attribute in the event;

[0086] : Information about the third attribute in the i-th event;

[0087] : Information about the fourth attribute in the i-th event;

[0088] … and possibly other event properties.

[0089] Taking computer music as an example, ui(1)ui(2)ui(3)ui(4) can represent the instrument, relative time, pitch and duration attribute information of the i-th note event, in no particular order.

[0090] Taking dance skeleton data as an example, ui(1)ui(2)ui(3)ui(4) can represent the torso, hand, leg and head attribute information of the i-th action, in no particular order.

[0091] For each attribute, an independent embedding matrix can be used to map discrete category information or continuous values ​​into a fixed-dimensional vector space. The embedding matrices of each attribute are:

[0092] ;

[0093] in, Indicates the vocabulary size of the attribute, represents the dimension of the corresponding embedding vector.

[0094] For events i , the embedding vectors obtained by looking up the table are:

[0095] ;

[0096] Concatenate the attribute embedding vectors to obtain a comprehensive representation of the event:

[0097] ;

[0098] Among them, the semicolon ";" represents the cascade operation of the vector. is the dimension of the final vector.

[0099] All T The comprehensive representations of events are stacked to obtain the structured representation input matrix with structural alignment:

[0100] ;

[0101] The structured representation (structured representation input matrix) X Provides a unified vector representation for subsequent attention calculations.

[0102] In order to capture the sequential information between events, position encoding needs to be introduced. Let the position encoding matrix be , then the final hidden state can be written as:

[0103] ；

[0104] The hidden state is the vector representation of the input data after it passes through the linear layer and position encoding, reflecting the initial feature extraction results of the data. The hidden state provides a complete representation of the data content and position features for the subsequent attention mechanism.

[0105] in, For the i The encoding vector of each position is usually composed of sine and cosine functions:

[0106] ,

[0107] here k This encoding method allows the model to distinguish different positions and learn relative position information.

[0108] In this specific embodiment, the specific steps of obtaining the attention matrix are as follows:

[0109] For the hidden layer input , obtain the query by linear transformation ( Query ),key(Key ) and value (V) matrices: Value ) matrices:

[0110] ,

[0111] where projection matrices , , are used to map to the same dimensional space. Typically, d is evenly divided into h heads, each with dimension .

[0112] Here, , , are learnable projection matrices used to map hidden layer states to query, key, and value spaces, respectively. These matrices are of size d x d , ensuring that the output Q, K, V matrices have consistent dimensions with the input, i.e., all are of size .

[0113] In multi-head attention mechanisms, to allow the model to capture information from different perspectives, the dimension d is typically evenly divided into h heads. Each head has a dimension of . Specifically:

[0114] The Q, K, V matrices are split into h sub-matrices, and the split sub-matrices are used as subsequent attention matrices. Each sub-matrix has a dimension of . Here, h is equal to the total number of heads h mentioned later.

[0115] Each head independently calculates attention, and then the results are concatenated and fused to generate the final output.

[0116] Through linear transformation by projection matrix , , , the following effects are mainly achieved:

[0117] Feature extraction: Different projection matrices extract suitable features for queries, keys, and values from , making Q, K, V focus on different roles.

[0118] Dimension alignment: Ensuring that Q, K, V have the same dimension d, facilitating subsequent calculations (such as dot product and weighted sum).

[0119] Flexibility: Projection matrices are learnable, and the model can adjust the mapping method according to task requirements.​​

[0120] Need to explain, Q, K, V matrix in the attention mechanism, together to complete the selection and transmission of information. Here are their roles:

[0121] Query (Q) matrix:

[0122] Meaning: Q represents the current word (or a certain position in the sequence) attention needs of other word.

[0123] Role: Each query vector (Q's first i row) is like a "question", used to "ask" the information of other positions in the sequence. It determines which parts the current word wants to pay attention to.

[0124] Key (K) matrix:

[0125] Meaning: K represents the information "key" of each word in the sequence, used to match the query.

[0126] Role: Key vector (K's first j row) and query vector Calculate similarity (usually by dot product), determine The degree of attention to position j .

[0127] Value (V) matrix:

[0128] Meaning: V contains the actual information content to be transmitted.

[0129] Role: Value vector (V's first j row) is weighted and summed in attention calculation to form the final output. The weight is determined by the similarity of Q and K.

[0130] Step S2 is the process of dividing the preset multi-head attention to obtain local attention heads and global attention heads, and calculating local attention outputs of the local attention heads and global attention outputs of the global attention heads based on the attention matrix.

[0131] In this embodiment, the preset multi-head attention is divided to obtain local attention heads and global attention heads, including:

[0132] S21: Obtain the total number of heads h of the multi-head attention; wherein,

[0133] S21A: If the total number of heads h is even, the attention heads numbered 1 to h / 2 are taken as global attention heads, and the attention heads numbered h / 2+1 to h are taken as local attention heads.

[0134] S21B: If the total number of heads h is odd, the attention heads numbered 1 to (h+1) / 2 are global attention heads, and the attention heads numbered (h+3) / 2 to h are local attention heads.

[0135] calculating a local attention output of the local attention head based on the attention matrix, comprising:

[0136] S221: dividing the local attention head into a specific number of categories of feature domain heads according to a preset indication function; the specific number is the attribute quantity;

[0137] S222: obtaining a feature attention score matrix based on the attention matrix, and causing the feature domain head to perform attribute-specific attention calculation for the attribute corresponding to the feature domain head to obtain an attention output according to the feature attention score matrix, wherein;

[0138] S223: splicing and fusing the attention outputs of the specific number of categories to obtain a local attention output.

[0139] calculating a global attention output of the global attention head based on the attention matrix, comprising:

[0140] S231: obtaining a global attention score matrix through the attention matrix;

[0141] S232: performing standard attention calculation based on the global attention score matrix through the global attention head to obtain a global attention output.

[0142] The core of the multi-head attention mechanism in the Transformer is to capture information in different subspaces through multiple independent attention heads. In order to achieve the focused capture of local features and the fusion of global information, the traditional multi-head attention is divided into two types of heads, global and local, in this embodiment, and the local attention is calculated within the feature domain, so that the local attention helps to focus on the feature area of the text, while the global attention tries to capture more extensive context information, and the step of feature domain division is introduced, thereby realizing fine control of the text features. This reduces the computational complexity while improving the controllability and quality of the generated text.

[0143] In one specific embodiment, the h heads are divided into two groups:

[0144] Global attention head: head number i= 1,…,h / 2. The traditional global attention calculation method is retained, allowing information exchange between any two positions, thereby capturing global dependency relationships;

[0145] Local attention head: head number i= h / 2+1,…,h. Each local attention head only focuses on the word pieces within the same feature domain. For example, for the same music event, its different features (instrument, time, pitch, duration, etc.) form several "feature domains". Local attention heads only interact information within the same feature domain.

[0146] The purpose of the embodiment is to consider that local and global information are both equally contributed to the generation quality, so the attention heads are evenly divided. Considering that h may be odd, when h is even, half of the local attention heads and half of the global attention heads; when h is odd, it is recommended to set the local attention heads to (floor) and the global attention heads to (ceiling). In addition, in some specific tasks, the proportion of the two can be adjusted according to actual needs to optimize the balance between local detail capture and global dependency modeling. The indicator function of the embodiment is as follows, assuming that the representation of the input event has been sequentially arranged or blocked according to each feature domain. Define the indicator function as follows:

[0147] ;

[0148] In the local attention head , according to the indicator function , the n feature domain head is divided:

[0149] ;

[0150] Wherein, the working principle of the indicator function is to select word piece 1 as a, and then loop b to 10 to match the same feature, so that 1~10 are divided; then 11 is found not to belong to the same feature, so 11 is selected as a, and b is continued to loop backward. So on and so forth.

[0151] "Feature domain" refers to a subset of features related to this part.

[0152] For the music generation task, the input data can be divided into 4 feature domains: pitch, duration, instrument, and start time, and the boundaries of each feature domain are preset.

[0153] Through such feature domain division, the model can focus on information interaction of specific attributes in the local attention head, providing a more fine-grained input basis for subsequent attention calculation.

[0154] For the first i head, the attention score matrix is defined as:

[0155] ;

[0156] The attention output is then:

[0157] ;

[0158] Wherein:

[0159] represents the query matrix of the i-th head, i

[0160] represents the key matrix,

[0161] represents the value matrix,

[0162] is the dimension of each head.

[0163] The softmax here represents normalizing the rows so that the weight sum of each row is 1.

[0164] Each feature field head divided by the local attention head is calculated by the standard attention to obtain:

[0165] ;

[0166] The local attention output is then obtained by splicing the feature field output:

[0167] ;

[0168] In this embodiment, the global attention head directly calculates the standard attention as steps S231 and S232, and no further description is given here.

[0169] The global and local attention are then fused to obtain the attention output result:

[0170] ;

[0171] Wherein, "concat" represents a concatenation operation along the last dimension, and the dimension of the concatenated vector is d. Concat Thus, the complete attention output result is obtained.

[0172]

[0173] ​​As described above, the controllable attention method based on feature domain division provided by the present application first obtains the word vector representation (word embedding), i.e., the original data obtains the hidden state (hidden state) through the linear layer (linear layer) and the position embedding (position embedding), i.e., the word vector representation of the original data; then projection (projection), i.e., projecting the word vector representation through the weight matrix 、 、 into the query (Q), key (K), and value (V) matrix; then the head and feature domain division, i.e., dividing the h heads (the meaning of an operation unit is directly translated from English) of the multi-head attention into two types of heads: h / 2 global attention heads (global attention head) that retain the traditional global attention mechanism, and h / 2 local attention heads (local attention head), where the tokens of each head are divided and arranged according to the number of preset feature domains; step four, fusion and output of attention information, the attention calculation results of different feature domains are aggregated into local attention, the local and global attention are fused to obtain controllable local attention based on feature domain, and then the attention results are output through the linear layer and layer normalization (layer normalization). The local attention mechanism is similar to focusing on the conversation of only a part of people in a group of people, while ignoring the voices of others, which can improve the processing efficiency, and the global attention refers to everyone in the whole group of people can hear the voices of others, which helps to capture more extensive information. Thus, the technical problems of high attention calculation complexity, poor fine-grained control ability, and unstable generation quality in the prior art are effectively solved, which helps to accelerate the convergence and reasoning of the model in language modeling tasks, while improving the controllability and quality of the generated content, and improving the harmony and rationality of the generated content.

[0174] In addition, in the present embodiment, after obtaining the attention output result, the following steps are further included:

[0175] S41: obtaining an output projection matrix through the attention output result;

[0176] S42: performing linear transformation on the output projection matrix to obtain mixed head information;

[0177] S43: performing layer normalization on the mixed head information to obtain correction data.

[0178] Specifically, the concatenated vector needs to be linearly transformed through an output projection matrix to mix the information of each head:

[0179] ;

[0180] Next, in order to stabilize the training process and correct the scale of each part's output, layer normalization is introduced. The formula for layer normalization is:

[0181] ;

[0182] Layer normalization is similar to 'standardizing' the output of each neuron to make its numerical distribution more stable and avoid gradient explosion or disappearance during training.

[0183] Among them, the layer normalization operation is defined as:

[0184] ;

[0185] here:

[0186] represents the representation of the i-th position in the vector Z,

[0187] μ and σ represent The mean and standard deviation of

[0188] γ and β are learnable scaling and translation parameters,

[0189] is a small constant that prevents division by zero,

[0190] “⊙” means element-by-element multiplication.

[0191] In order to demonstrate the progress of the controllable attention method based on feature domain partitioning in this embodiment compared with the existing technology, the effectiveness of the controllable local attention method based on feature domain partitioning (FCLA) in language modeling and controllable generation tasks is demonstrated. Then, multiple groups of experiments are designed on the symbolic music (especially computer music in MIDI format) generation task, covering objective evaluation and comparative analysis of the experiments.

[0192] While this example primarily uses language modeling as its theoretical foundation, music generation can also be considered a sequence generation problem, and its technical principles are highly similar to those of natural language processing. Therefore, in the experimental section, the MIDI music generation task was used as a validation platform to demonstrate the applicability of this method to various types of sequence generation tasks. Subsequent work could further introduce data from natural language generation tasks to more comprehensively verify the method's universality.

[0193] The experiment adopts the Lakh MIDI Dataset (LMD) as the benchmark dataset, which is available at https: / / colinraffel.com / projects / lmd. We select popular song fragments (about 12,000 songs) in the LMD-matched subset.

[0194] To compare the performance of FCLA, we select the state-of-the-art (SOTA) model as the baseline:

[0195] The large model based on the feature domain division controllable attention method in this embodiment is based on the Transformer architecture, integrating feature domain division and local and global attention mechanisms. We conduct experiments on the self-developed model Pop-Diffuseq using the FCLA mechanism.

[0196] The model is trained on 4 NVIDIA 4090 graphics cards, with a batch size of 64, using the Adam optimizer, a learning rate of 2e-4, and a training period of 50 rounds.

[0197] I. Evaluation indicators are as follows:

[0198] ① Music measurement indicators

[0199] Chord accuracy (CA): Chord accuracy measures the degree of matching between the generated music's harmonic progression (i.e., the sequence of chords) and the true music's harmonic progression. Specifically, it calculates the proportion of correctly identified chord sequences in the total chord sequences in the generated music.

[0200] Distribution overlap: Distribution overlap is used to measure the similarity between generated music and true music in certain musical properties:

[0201] 1. Pitch distribution (DP): DP measures the similarity between the frequency of various pitches (such as C, D, E, etc., without considering high or low octaves) in the generated music and the frequency of pitches in the true music.

[0202] 2. Duration distribution (DD): DD measures the similarity between the distribution of note durations (such as whole notes, half notes, etc.) in the generated music and the distribution of note durations in the true music.

[0203] 3. Onset time distribution (DO): DO measures the similarity between the distribution of note onset time intervals (i.e., the time difference between notes) in the generated music and the distribution of note onset time intervals in the true music.

[0204] DP, DD, and DO are the average values of the overlapped areas of pitch, duration, and onset time distributions, respectively.

[0205] Let the generated distribution be DG (generated distributions) and the true distribution be DT (truth distributions),

[0206] The generated and true distributions of pitch are DP G and DP T

[0207] The generated distribution and true distribution of duration are DD G , DD T

[0208] The generated distribution and true distribution of the start time are DOG and DOT

[0209] but:

[0210] DP= E[min(DPG,DPT) / max(DPG,DPT)]

[0211] DD = E[min(DDG,DDT) / max(DDG,DDT)]

[0212] DO = E[ min(DOG, DOT) / max(DOG, DOT)]

[0213] Among them, the function E represents the mean of all test set data, the function min represents the minimum value, and the function max represents the maximum value.

[0214] The pitch distribution DP of a piece of music is the frequency of occurrence of each type of pitch. There are 12 types in total, regardless of octave groups.

[0215] The duration distribution DD and start time distribution DO of a piece of music respectively represent the average duration and average start time in the music, both of which are scalar information.

[0216] Assume that a piece of real music has the following pitch data:

[0217] The note C appears 50 times, the note D appears 30 times, and the note E appears 20 times (100 times in total).

[0218] Frequency: C=50%, D=30%, E=20%.

[0219] The pitch data for generating music is:

[0220] The note C appears 45 times, the note D appears 35 times, and the note E appears 20 times (100 times in total).

[0221] Frequency: C=45%, D=35%, E=20%.

[0222] Calculate DP:

[0223] C: min(50%, 45%) / max(50%, 45%) = 45% / 50% = 0.9

[0224] D: min(30%, 35%) / max(30%, 35%) = 30% / 35% = 0.857

[0225] E: min(20%, 20%) / max(20%, 20%) = 20% / 20% = 1.0

[0226] DP = mean(0.9, 0.857, 1.0) ≈ 0.919

[0227] This value is close to 1, indicating that the pitch distribution of the generated music is very similar to the real music

[0228] ② Generation quality indicators

[0229] Perplexity (PPL) is a commonly used indicator to measure the quality of sequence generation, and its calculation method is based on cross-entropy loss. Specifically, for a given generated sequence and the corresponding real sequence, first calculate the cross-entropy loss by token, then take the exponential of the average cross-entropy loss of all tokens, that is:

[0230] where N represents the total number of tokens, represents the cross-entropy loss of the i-th token. Here, the output of the average cross-entropy loss is taken as an exponential, in order to convert the loss into a direct numerical value related to probability, so as to facilitate the understanding of the uncertainty of the model in generation. A lower PPL value means that the model fits the real data distribution better, and the quality of the generated sequence is higher.

[0231] In actual experiments, we calculate the token-level PPL every 20 training steps, and evaluate the training process of the model by observing the stability of the PPL curve.

[0232] II. Experimental results are as follows:

[0233] ① Performance comparison

[0234] Table 1 shows the objective indicators of the method (FCLA) and the baseline model. The experimental results show that:

[0235] FCLA is better than the baseline in chord accuracy (CA = 0.64 ± 0.05), pitch (DP = 0.78 ± 0.1), duration (DD = 0.83 ± 0.1), and onset time (DO = 0.68 ± 0.1), and the generated results are closer to the real data distribution.

[0236]

[0237] Table 1 Comparison of objective indicators (mean ± standard deviation, the higher the value, the higher the generation quality);

[0238] Perplexity (PPL): The PPL curve of FCLA is stable, converges faster, and has smaller fluctuations than DiffuSeq.

[0239] In the reference experiment, the PPL value of Pop-Diffuseq remains stable and low, while the PPL value of DiffuSeq fluctuates greatly, indicating that the former has a significant advantage in generation quality.

[0240] Further experiments found that when DiffuSeq introduced the FCLA mechanism, its PPL value decreased significantly and became more stable. This shows that the FCLA mechanism can effectively improve the performance of DiffuSeq, making the generated music of higher quality and closer to real music, while reducing the uncertainty in the generation process. The perplexity (PPL) curve (the lower the value, the higher the generation quality) obtained by the controllable attention method based on feature domain division in this embodiment greatly shows that the generation quality has been greatly improved.

[0241] ②Efficiency comparison

[0242] GPU memory usage: The peak GPU memory usage of FCLA is reduced by 26.9% compared to DiffuSeq, which meets the needs of long sequence generation.

[0243] Inference speed: It only takes 37.86 seconds to generate 447 music segments on a single card, which is 18.89% faster than DiffuSeq. Therefore, the inference speed and efficiency have also been greatly improved compared to the prior art.

[0244] As can be seen, the controllable attention method based on feature domain division provided in this embodiment makes the original data undergo linear layer and position embedding processing to form word vector representation, and obtains an attention matrix based on the word vector representation. Then, the preset multi-head attention is divided to obtain local attention heads and global attention heads, and the local attention output of the local attention heads and the global attention output of the global attention heads are calculated based on the attention matrix. Then, the local attention output and the global attention output are fused to obtain an attention output result. Compared with the prior art, the present application has the following beneficial effects:

[0245] Through experimental comparison, whether chord accuracy, section overlap, or value fluctuation, the generated music is of higher quality, closer to real music, with less uncertainty, higher efficiency, and faster reasoning speed than the prior art. Simply put, by introducing feature domain division, fine control of text features is achieved, which reduces computational complexity while improving the controllability and quality of generated text. Local attention helps focus on the feature area of the text, while global attention tries to capture more extensive context information. The combination of the two improves the controllability and quality of the generated content, improves the harmony and reasonableness of the generated content.

[0246] In summary, the controllable attention method based on feature domain division provided by the embodiment combines local and global attention and introduces feature domain division to achieve fine control of text features. This reduces computational complexity while improving the controllability and quality of generated text, allowing local attention to focus on the feature area of the text, while global attention tries to capture more extensive context information. Specifically, the following beneficial effects can be achieved:

[0247] (1) Reducing computational complexity: By localizing part of the attention head, large-scale matrix operations are divided into multiple small-scale matrix operations, and the overall complexity is significantly reduced compared to traditional Transformers. This improvement is particularly significant in long sequence generation tasks, significantly reducing computational and memory overhead.

[0248] The feature domain described herein refers to the area divided according to the attributes (such as syntax, semantics, emotion, etc.) of the input data (such as text or music events) so that the model can handle different features specifically.

[0249] (2) Balancing local and global information: The invention combines local and global attention through a hybrid attention scheme, capturing local details while not missing global context information, resulting in coherent and detailed content.

[0250] By dividing the multi-head attention into local attention heads and global attention heads, detailed information within the feature domain and long-distance dependencies in the sequence are processed respectively.

[0251] (3) Improving the controllability of generated text: By introducing feature domain division technology, the invention can more accurately control the grammar, semantics, emotion, style, and other dimensions of text generation, making the generated text not only realistic in natural language expression, but also meeting specific control requirements.

[0252] (4) Enhance the expansibility and application range of the model: the method proposed in the application can be widely applied to machine translation, text summarization, content creation and other language modeling tasks, and has good popularization value and practical application prospect.

[0253] Next, a feature domain division based controllable attention system provided by an embodiment of the application is introduced, and the feature domain division based controllable attention system described below can be correspondingly referred to the feature domain division based controllable attention method described above.

[0254] As shown in Figure 3 The application further provides a feature domain division based controllable attention system 100, which can implement the feature domain division based controllable attention method as described above, and the system comprises:

[0255] An attention obtaining unit 110 is configured to make the original data pass through a linear layer and a position embedding processing to form a word vector representation, and obtain an attention matrix based on the word vector representation.

[0256] A local-global segmentation unit 120 is configured to divide a preset multi-head attention to obtain a local attention head and a global attention head, and calculate a local attention output of the local attention head and a global attention output of the global attention head based on the attention matrix.

[0257] An attention fusion unit 130 is configured to perform attention fusion on the local attention output and the global attention output to obtain an attention output result.

[0258] Wherein, the total number of the multi-head attention is h; wherein,

[0259] If h is an even number, the attention heads numbered 1 to h / 2 are taken as the global attention heads, and the attention heads numbered h / 2+1 to h are taken as the local attention heads.

[0260] If h is an odd number, the attention heads numbered 1 to (h+1) / 2 are taken as the global attention heads, and the attention heads numbered (h+3) / 2 to h are taken as the local attention heads.

[0261] The feature domain division based controllable attention system 100 provided by the application is calculated in the same way as the feature domain division based controllable attention method described above, and the more specific implementation process can refer to the specific embodiments of the feature domain division based controllable attention method described above.

[0262] As described above, the controllable attention system 100 based on feature domain division provided by the present application, such as the controllable attention method based on feature domain division described above, can also solve the technical problems of high computational resource consumption, insufficient long-distance dependency capture, uneven information distribution, difficult multi-dimensional information fusion, poor controllability and balance of generation quality of the attention calculation method in the prior art. By introducing feature domain division, fine control of text features is realized, which reduces the computational complexity while improving the controllability and quality of the generated text. Local attention helps to focus on the feature area of the text, while global attention tries to capture more extensive context information. The combination of the two can significantly reduce the computational complexity while improving the controllability and quality of the generated content and improving the harmony and rationality of the generated content.

[0263] The embodiment of the present application also provides a computer readable storage medium (not shown in the figure), which can be non-volatile or volatile, and stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0264] subjecting original data to linear layer and position embedding processing to form a word vector representation, and obtaining an attention matrix based on the word vector representation;

[0265] dividing a preset multi-head attention to obtain a local attention head and a global attention head, and calculating a local attention output of the local attention head and a global attention output of the global attention head based on the attention matrix;

[0266] performing attention fusion on the local attention output and the global attention output to obtain an attention output result.

[0267] Specifically, when the computer program is executed by the processor, the specific implementation method can refer to the description of the related steps in the controllable attention method based on feature domain division in the embodiment, which will not be described here.

[0268] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the modules is only a logical function division. There can be another division manner in actual implementation.

[0269] The modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical units, i.e. they can be located in one place, or distributed on multiple network units. According to actual needs, some or all of the modules can be selected to achieve the purpose of the present embodiment.

[0270] In addition, each functional module in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software function modules.

[0271] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

[0272] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any additional reference signs in the claims should not be considered as limiting the claims involved.

[0273] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The plurality of units or devices stated in the system claims can also be implemented by one unit or device through software or hardware. The second word is used to indicate the name, and does not mean any specific order.

[0274] It should be noted that each embodiment in the specification is described in a progressive manner, and the same and similar parts between each embodiment can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for the method, device, electronic equipment and computer readable storage medium, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant part can be referred to the part of the method embodiment. The above described method, device, electronic equipment and medium are only illustrative, and the units described as separate components can be or can not be physically separated, and the components indicated as units can be or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to the actual needs, part or all of the modules can be selected to achieve the purpose of the embodiment. Those skilled in the art can understand and implement without creative labor.

[0275] The feature domain division based controllable attention method, system and storage medium according to the present application are described above with reference to the accompanying drawings in an exemplary manner. However, those skilled in the art should understand that various improvements can be made to the above feature domain division based controllable attention method, system and storage medium according to the present application without departing from the content of the present application. Therefore, the protection scope of the present application should be determined by the content of the appended claims.

Claims

1. A controllable attention method based on feature domain division, characterized in that: include: The raw data is processed through a linear layer and position embedding to form a word vector representation, and an attention matrix is ​​obtained based on the word vector representation; wherein the method includes: obtaining events in the raw data; dividing each event into a feature domain corresponding to the attribute quantity of the event to obtain attribute information of at least two dimensions; obtaining an attribute embedding matrix corresponding to each attribute information; obtaining an embedding vector for each attribute information based on the embedding matrix; concatenating the embedding vectors to obtain a comprehensive representation of the event, and stacking the comprehensive representations of each event to obtain a structured representation of the raw data; combining the structured representation with the pre-acquired position encoding to obtain a hidden layer state, and obtaining an attention matrix based on the hidden layer state; The preset multi-head attention is divided to obtain local attention heads and global attention heads, and the local attention output of the local attention heads and the global attention output of the global attention heads are calculated based on the attention matrix; wherein the local attention output of the local attention heads is calculated based on the attention matrix, including: dividing the local attention heads into feature domain heads of a specific number of categories according to a preset indicator function; the specific number is the attribute quantity; obtaining a feature attention score matrix based on the attention matrix, and causing the feature domain heads to perform attention calculation of specific attributes for the attributes corresponding to the feature domain heads according to the feature attention score matrix to obtain attention output; wherein the attention outputs of the specific number of categories are spliced ​​and fused to obtain the local attention output; The local attention output and the global attention output are subjected to attention fusion to obtain an attention output result.

2. The controllable attention method based on feature domain division according to claim 1, characterized in that: Obtaining an attention matrix based on the hidden layer state, including: Performing a linear transformation on the hidden layer state based on a preset projection matrix to obtain a query matrix, a key matrix, and a value matrix; The query matrix, key matrix and value matrix are respectively divided into a preset number of sub-matrices, and the sub-matrix of the query matrix, the sub-matrix of the key matrix and the sub-matrix of the value matrix are used as the attention matrix; wherein the preset number is the total number of heads h of the multi-head attention.

3. The controllable attention method based on feature domain division according to claim 1, characterized in that The preset multi-head attention is divided into local attention heads and global attention heads, including: Get the total number of heads h of the multi-head attention; where, If the total number of heads h is an even number, the attention heads numbered 1 to h / 2 are regarded as global attention heads, and the attention heads numbered h / 2+1 to h are regarded as local attention heads; If the total number of heads h is an odd number, the attention heads numbered from 1 to (h+1) / 2 are regarded as global attention heads, and the attention heads numbered from (h+3) / 2 to h are regarded as local attention heads.

4. The controllable attention method based on feature domain division according to claim 3, characterized in that: Calculating a global attention output of the global attention head based on the attention matrix includes: Obtain a global attention score matrix through the attention matrix; A standard attention calculation is performed by the global attention head based on the global attention score matrix to obtain a global attention output.

5. The controllable attention method based on feature domain division as claimed in claim 4, characterized in that: After obtaining the attention output results, it also includes: Obtain an output projection matrix through the attention output result; performing a linear transformation on the output projection matrix to obtain mixing head information; Layer normalization is performed on the mixing head information to obtain calibrated data.

6. A controllable attention system based on feature domain division, characterized in that: To implement the controllable attention method based on feature domain division as described in any one of claims 1 to 5, the system includes: An attention acquisition unit is configured to process raw data through a linear layer and position embedding to form a word vector representation, and to acquire an attention matrix based on the word vector representation; wherein the process includes: acquiring events in the raw data; dividing each event into a feature domain corresponding to the attribute quantity of the event to acquire attribute information of at least two dimensions; acquiring an attribute embedding matrix corresponding to each attribute information; acquiring an embedding vector for each attribute information based on the embedding matrix; concatenating the embedding vectors to acquire a comprehensive representation of the event, and stacking the comprehensive representations of each event to acquire a structured representation of the raw data; combining the structured representation with a pre-acquired position encoding to acquire a hidden layer state, and acquiring an attention matrix based on the hidden layer state; A local-global segmentation unit, configured to divide the preset multi-head attention to obtain local attention heads and global attention heads, and calculate the local attention output of the local attention heads and the global attention output of the global attention heads based on the attention matrix; wherein the local attention output of the local attention heads is calculated based on the attention matrix, including: dividing the local attention heads into feature domain heads of a specific number of categories according to a preset indicator function; the specific number is the attribute quantity; obtaining a feature attention score matrix based on the attention matrix, and causing the feature domain heads to perform attention calculation of specific attributes for the attributes corresponding to the feature domain heads according to the feature attention score matrix to obtain attention output; wherein the attention outputs of the specific number of categories are spliced ​​and fused to obtain the local attention output; An attention fusion unit is used to fuse the local attention output and the global attention output to obtain an attention output result.

7. The controllable attention system based on feature domain division according to claim 6, characterized in that: The total number of heads of the multi-head attention is h; where If h is an even number, the attention heads numbered from 1 to h / 2 are regarded as global attention heads, and the attention heads numbered from h / 2+1 to h are regarded as local attention heads; If h is an odd number, the attention heads numbered from 1 to (h+1) / 2 are regarded as global attention heads, and the attention heads numbered from (h+3) / 2 to h are regarded as local attention heads.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the controllable attention method based on feature domain division according to any one of claims 1 to 5 is implemented.

Images