Multi-head attention grouping method and device and storage medium

Abstract

The invention discloses a multi-head attention grouping method and device and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: firstly compressing at least two multi-head vectors corresponding to multi-head attention into single-dimensional vectors, then determining a cluster order according to the single-dimensional vectors, and carrying out the clustering of the multi-head attention according to the cluster order; the multi-head attention key and value grouping method comprises the following steps: clustering a plurality of multi-head attention keys, extracting a clustering result in a cluster sequence based on a clustering algorithm, carrying out average pooling based on a target matrix with matched types in the clustering result, merging into a result matrix, and finally grouping the multi-head attention keys and values according to the result matrix to obtain a multi-head attention key and value grouping result. And determining the number of groups according to the grouping result. According to the method, the multi-head vectors are clustered through the self-adaptive clustering algorithm, so that the clustering result is generated, grouping of the multi-head vectors is completed according to the clustering result, and the efficiency of model training is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multi-head attention grouping method, device and storage medium. Background Art

[0002] In the field of natural language processing and machine learning, GQA (Grouped-Query Attention) is a key technology to improve the efficiency of Transformer models. At present, the relevant solution for grouping Key and Value in attention heads is to manually set the hyperparameter of the number of groups, so as to group the GQA heads according to the number of groups and make the heads in each group share the transformation matrix of Key and Value. However, manually setting the number of groups will increase the complexity of model parameter adjustment, which will lead to low efficiency of model training.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The present application provides a multi-head attention grouping method, device and storage medium, aiming to solve the problem that related solutions increase the complexity of model parameter adjustment by manually setting the number of groups, thereby leading to low efficiency of model training.

[0005] To achieve the above purpose, the present application provides a multi-head attention grouping method, which comprises the following steps:

[0006] Compress at least two multi-head vectors corresponding to the multi-head attention into single-dimensional vectors respectively;

[0007] Determine a cluster order according to the single-dimensional vector, and extract a clustering result from the cluster order based on a clustering algorithm, wherein the clustering result is composed of at least one matrix;

[0008] Perform average pooling based on the target matrices of type matching in the clustering results to merge into a result matrix;

[0009] The keys and values ​​of the multi-head attention are grouped according to the result matrix, and the number of groups is determined according to the grouping result.

[0010] In one embodiment, the step of compressing at least two multi-head vectors corresponding to the multi-head attention into single-dimensional vectors includes:

[0011] Determine the number of multi-head vectors in the multi-head attention, and construct a vector matrix according to the number and the multi-head vectors;

[0012] Calculating the covariance matrix of the multi-head vector according to the relationship between the variables in the vector matrix;

[0013] Decomposing the covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors;

[0014] A dot product operation is performed according to the eigenvalue and the eigenvector to generate the one-dimensional vector, wherein the one-dimensional vector is a representation of the multi-headed vector in a one-dimensional space.

[0015] In one embodiment, the step of determining the cluster order according to the single-dimensional vector and extracting the clustering result in the cluster order based on a clustering algorithm includes:

[0016] Determine the one-dimensional vector as a data point, and determine the specific number of the data points;

[0017] Obtaining a preset distance threshold, and determining a sample parameter according to the specific quantity;

[0018] Processing the data points according to the distance threshold and the sample parameters to generate the cluster ranking;

[0019] A clustering result in the cluster sorting is extracted based on a clustering algorithm.

[0020] In one embodiment, the step of extracting the clustering results in the cluster sorting based on the clustering algorithm includes:

[0021] Determining a target distance threshold and a target sample parameter according to the cluster sorting;

[0022] Determining the number of categories in the cluster sorting according to the target distance threshold and the target sample parameter, and dividing the cluster sorting into at least one data point set according to the number of categories;

[0023] The data point set is output and determined as the clustering result.

[0024] In one embodiment, the step of performing average pooling based on the target matrix of type matching in the clustering result to merge into a result matrix includes:

[0025] Determine a first matrix whose adjacent distance is less than the target distance threshold in the clustering result, and determine the first matrix as the target matrix of type matching;

[0026] Performing dimensionality reduction processing on the target matrix;

[0027] The target matrix after dimension reduction is average pooled to generate the result matrix.

[0028] In one embodiment, after the step of determining in the clustering result a first matrix whose adjacent distance is less than the target distance threshold and determining the first matrix as the target matrix of type matching, the step includes:

[0029] Determine a second matrix in the clustering result whose adjacent distance is greater than the target distance threshold;

[0030] The second matrix is ​​deleted from the clustering result, and the step of determining the first matrix whose adjacent distance is less than the target distance threshold in the clustering result is performed on the deleted clustering result.

[0031] In one embodiment, the step of grouping the keys and values ​​of the multi-head attention according to the result matrix and determining the number of groups according to the grouping result includes:

[0032] determining the number of the result matrices;

[0033] Grouping the keys and values ​​of the multi-head attention according to the number of the result matrix;

[0034] A target number of the keys and values ​​after grouping is determined, and the target number is determined as the number of groups.

[0035] In one embodiment, after the step of grouping the keys and values ​​of the multi-head attention according to the result matrix and determining the number of groups according to the grouping result, the method further includes:

[0036] Inputting the number of groups as a parameter into the multi-head attention, and fine-tuning the parameters of the model according to the multi-head attention;

[0037] The model is trained according to the fine-tuned parameters.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a multi-head attention grouping device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the multi-head attention grouping method as described above.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, the steps of the multi-head attention grouping method as described above are implemented.

[0040] The present application provides a multi-head attention grouping method, a multi-head attention grouping device and a storage medium, by compressing at least two multi-head vectors corresponding to the multi-head attention into single-dimensional vectors respectively, and then determining the cluster order according to the single-dimensional vector, and extracting the clustering result in the cluster order based on the clustering algorithm, the clustering result is composed of at least one matrix, and then average pooling is performed based on the target matrix of type matching in the clustering result, and merged into a result matrix, and finally the keys and values ​​of the multi-head attention are grouped according to the result matrix, and the number of groups is determined according to the grouping result. The present application clusters the multi-head vectors through an adaptive clustering algorithm to generate a clustering result, and then completes the grouping of the multi-head vectors according to the clustering result to improve the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 This is a flow chart of the first embodiment of the multi-head attention grouping method of the present application;

[0044] Figure 2 This is a schematic diagram of a target matrix determination process involved in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of clustering result visualization involved in the embodiment of the present application;

[0046] Figure 4 This is a schematic diagram of multi-head attention grouping involved in the embodiment of the present application;

[0047] Figure 5 A flowchart of a second embodiment of the multi-head attention grouping method of the present application;

[0048] Figure 6 This is a schematic diagram of the architecture of the hardware operating environment of the multi-head attention grouping device involved in the embodiment of the present application.

[0049] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] In order to better understand the above technical solution, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0052] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of this application is: compress at least two multi-head vectors corresponding to the multi-head attention into single-dimensional vectors respectively; determine the cluster order according to the single-dimensional vector, and extract the clustering result in the cluster order based on the clustering algorithm, wherein the clustering result is composed of at least one matrix; perform average pooling based on the target matrix of type matching in the clustering result and merge them into a result matrix; group the keys and values ​​of the multi-head attention according to the result matrix, and determine the number of groups according to the grouping results.

[0054] Currently, in the self-attention mechanism, the scheme for grouping Key and Value is to manually set the hyperparameter of the number of groups, so as to group the query heads according to the number of groups and make the heads in each group share the transformation matrix of Key and Value. However, manually setting the number of groups will increase the complexity of model parameter adjustment, which leads to low efficiency of model training.

[0055] This application first compresses at least two multi-head vectors corresponding to the multi-head attention into single-dimensional vectors, then determines the cluster order based on the single-dimensional vector, and extracts the clustering results in the cluster order based on the clustering algorithm. The clustering result consists of at least one matrix, and then average pooling is performed based on the target matrix of type matching in the clustering result to merge into a result matrix. Finally, the keys and values ​​of the multi-head attention are grouped according to the result matrix, and the number of groups is determined according to the grouping results. This application clusters the multi-head vectors through an adaptive clustering algorithm to generate clustering results, and then completes the grouping of the multi-head vectors according to the clustering results to improve the efficiency of model training.

[0056] Embodiment 1

[0057] Please refer to Figure 1 In a first embodiment, the multi-head attention grouping method comprises the following steps:

[0058] Step S10: Compress at least two multi-head vectors corresponding to the multi-head attention into single-dimensional vectors respectively.

[0059] In this embodiment, the processing action is performed by the model training system. During the model training process, the multi-head attention is an attention mechanism widely used in deep learning, especially in natural language processing (NLP) and computer vision (CV) and other fields. It extends the basic attention mechanism and enhances the model's representation ability and data understanding ability by introducing multiple parallel attention "heads". The multi-head attention consists of at least two multi-head vectors, and the multi-head vector can be obtained by obtaining the multi-head attention.

[0060] It should be noted that if there are two multi-headed vectors in the multi-headed attention, the two multi-headed vectors are compressed into two corresponding single-dimensional vectors; if there are three multi-headed vectors in the head attention, three single-dimensional vectors are obtained by compression; similarly, if there are at least three multi-headed vectors in the head attention, at least three single-dimensional vectors are obtained by compression.

[0061] The multi-head attention mechanism is usually used in the Transformer model. When using the multi-head attention mechanism, the input sequence is first mapped to multiple query (Q), key (K), and value (V) vectors, and each head generates these vectors through different linear transformations. Then, each head calculates the dot product of Q and K, normalizes it through the softmax function, and finally multiplies these weights with V to get the output of each head. The outputs of all heads are concatenated and then transformed through a linear transformation to get the final output. Its core advantage is that it can capture multiple different characteristics of the input data at the same time. Different "heads" can focus on different aspects of the word sequence, such as semantics, grammar, etc.

[0062] Specifically, in the Transformer model, the output of the multi-head attention layer is obtained, where the multi-head attention usually contains multiple heads (for example, 8 or 16 heads), each head will output a vector sequence, namely a multi-head vector, which includes the number of heads of the multi-head attention, the vector dimension of each head, and the sequence length. Then, the output of multiple heads, namely the multi-head vector, is output to a matrix. Then, the multi-head vector in the matrix is ​​compressed into a single-dimensional vector based on the principal component analysis method.

[0063] In this embodiment, the step of obtaining a multi-head vector in the multi-head attention and compressing the multi-head vector into a single-dimensional vector includes:

[0064] Determine the number of multi-head vectors in the multi-head attention, and construct a vector matrix according to the number and the multi-head vectors; calculate the covariance matrix of the multi-head vectors according to the relationship between the variables in the vector matrix; decompose the covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors; perform dot product operation according to the eigenvalues ​​and the eigenvectors to generate the single-dimensional vector, wherein the single-dimensional vector is the representation of the multi-head vector in a single-dimensional space.

[0065] Specifically, the multi-head vector can be compressed into a single-dimensional vector through principal component analysis. First, the multi-head vector (matrix) in the attention mechanism is used as the input data of the principal component analysis. Assuming there are n heads and the length of the vector corresponding to each head is m, then the input data is an n×m matrix. Before performing principal component analysis, it is usually necessary to center the data (i.e., subtract the mean) to ensure that the results of the principal component analysis are not affected by data deviation. For the vector corresponding to each head, calculate its mean separately and subtract the mean from each element.

[0066] For the centered data, the covariance matrix can be obtained by calculating the covariance between the variables. The covariance matrix is ​​used to describe the relationship between the variables in the data (here, the vector elements corresponding to each head). Since the input data is a matrix, the covariance matrix obtained will be an m×m matrix. The covariance matrix is ​​then subjected to eigenvalue decomposition to obtain the eigenvalues ​​and corresponding eigenvectors, where the eigenvalues ​​represent the variance of each principal component and the eigenvectors represent the direction in the multidimensional space. Since the purpose is to compress the data into a single-dimensional vector, we only need to pay attention to the largest eigenvalue and its corresponding eigenvector, that is, project the original data onto the largest eigenvector to obtain a single-dimensional vector. By performing a dot product operation on the vector corresponding to each head and the largest eigenvector, the result obtained is the representation of the head in the single-dimensional space.

[0067] In addition, if a unified single-dimensional vector is needed to represent the information of all heads, the representation of each head in the single-dimensional space can be further processed, such as taking the average, the maximum value, or performing a weighted sum.

[0068] As an optional implementation method, the multi-head vector can be compressed into a single-dimensional vector by inputting the multi-head vector sequence into a recurrent neural network (RNN) or a long short-term memory network (LSTM), and using the last hidden state or output state of the recurrent neural network or the long short-term memory network as a single-dimensional vector.

[0069] Specifically, the multi-head vector (assuming that each head represents a feature sequence) is first organized into a format that can be processed by RNN, usually a three-dimensional vector (number of samples, time steps, number of features). The preprocessed data is then input into the RNN layer. The RNN layer processes the input vector at each time step and updates its internal state. The state of the RNN layer contains information from all previous time steps, so it can capture the temporal dependencies in the sequence. At the last time step of the RNN layer, its state is extracted as the output. This state vector already contains all the information in the sequence and is a one-dimensional vector. In addition, the output state of the RNN layer can be mapped to the desired feature space by adding a fully connected layer to obtain the final one-dimensional vector.

[0070] As another optional implementation, the multi-head vector can also be compressed into a single-dimensional vector by an autoencoder. An autoencoder is an unsupervised learning model that performs data compression by learning a low-dimensional representation of the input data. The multi-head vector can be used as the input of the autoencoder and trained to output a single-dimensional vector. The autoencoder learns the compressed representation of the data by minimizing the reconstruction error, thereby compressing the multi-head vector into a unit vector and outputting it.

[0071] Step S20: determining a cluster order according to the single-dimensional vector, and extracting a clustering result from the cluster order based on a clustering algorithm, wherein the clustering result is composed of at least one matrix.

[0072] In this embodiment, after compressing the multi-head vector into a single-dimensional vector, the cluster sorting is determined according to the sorting method of the single-dimensional vector, wherein the cluster sorting is composed of multiple single-dimensional vectors, and the sorting method is that the distance between the single-dimensional vectors is less than a preset threshold, so that the data with the distance between the single-dimensional vectors less than the preset threshold is selected as the cluster sorting, and the cluster sorting represents the density-based clustering structure of each sample point. Then, based on the clustering algorithm, multiple cluster sortings are merged into a clustering result, wherein the clustering result is composed of multiple matrices.

[0073] Specifically, since the single-dimensional vector is generated by compressing the multi-head vector, the single-dimensional vector is processed on the same scale. Then, a suitable distance measurement method (such as Euclidean distance, Manhattan distance, etc.) is selected to calculate the distance between each point in the single-dimensional vector, and a suitable clustering algorithm, such as K-means, hierarchical clustering, etc., is selected according to the characteristics and requirements of the data. Then, the parameters required by the clustering algorithm are set, such as the number of clusters K in the K-means algorithm, the distance measurement method and link type in the hierarchical clustering, etc. The processed single-dimensional vector is input into the clustering algorithm, and the clustering process is executed to obtain the clustering result. Then, according to the clustering result, the clusters are sorted according to certain rules. The sorting rules can be determined according to actual needs, such as the size of the cluster, the similarity of the samples in the cluster, etc. The clustering result is represented in the form of at least one matrix. This matrix can contain multiple aspects of information, such as the number of clusters, the index of the samples in the cluster, the distance between clusters, etc. For example, a cluster assignment matrix can be created, in which each row represents a sample, each column represents a cluster, and the elements in the matrix represent the cluster number to which the sample belongs. Finally, the clustering results are analyzed to evaluate the quality and effect of clustering. The silhouette coefficient, Calinski-Harabasz index and other indicators can be used to evaluate the effect of clustering.

[0074] Step S30: performing average pooling based on the target matrices of type matching in the clustering results to merge into a result matrix.

[0075] In this embodiment, after the clustering result is output based on the clustering algorithm, the clustering result is first processed to remove the noise in the clustering result to ensure that the samples in the clustering result have similar characteristics or attributes. The clustering result is usually expressed in the form of cluster centers, and these clustering results need to be organized into the form of a target matrix. The target matrix is ​​then average-pooled and merged into a result matrix.

[0076] In addition, it is necessary to construct a target matrix, that is, to construct a target matrix based on the clustering results, where the rows represent the elements in the clustering results and the columns represent the features of the elements. You can also directly use numerical values ​​to construct a matrix.

[0077] Specifically, average pooling is a commonly used merging method, which can obtain a feature vector representing the cluster by calculating the feature average of all elements in each cluster. When merging the target matrix by average pooling, the average pooling method is first selected, and then for each cluster in the target matrix, the average of all its features (ie, each column) is calculated. This will result in a new matrix (result matrix) in which each row represents the feature vector of a cluster and each column represents the average of a feature. If there are missing values ​​or outliers in the target matrix, preprocessing is required before calculating the average value. For example, an interpolation method can be used to fill in missing values, or an outlier detection algorithm can be used to identify and process outliers.

[0078] In addition, the clustering results and the average pooling results need to be verified and evaluated to ensure their accuracy and reliability. This can be done using external validity evaluation, internal validity evaluation, and correlation test evaluation.

[0079] Optionally, in this embodiment, the step of performing average pooling based on the target matrix of type matching in the clustering result to merge into a result matrix includes:

[0080] Determine a first matrix whose adjacent distance is less than the target distance threshold in the clustering result, and determine the first matrix as the target matrix of type matching; perform dimensionality reduction processing on the target matrix; and perform average pooling on the target matrix after dimensionality reduction to generate the result matrix.

[0081] Specifically, the target distance threshold is a parameter set in the clustering algorithm, and the clustering result is composed of multiple matrices. Figure 2 As shown, Figure 2 This is a schematic diagram of the process of determining the target matrix involved in the embodiment of the present application. In the clustering result, the first matrix whose adjacent distance is less than the target distance threshold is queried according to the target distance threshold, and these first matrices are used as target matrices of type matching. Then, the target matrix is ​​reduced in dimension to ensure that the first matrix obtained is in the same dimension, and then the target matrix after dimension reduction is averaged and pooled to generate the result matrix, that is, Figure 2 The sample set outlined by the box.

[0082] Further, in this embodiment, after the step of determining in the clustering result a first matrix whose adjacent distance is less than the target distance threshold and determining the first matrix as the target matrix of type matching, the method further includes:

[0083] Determine a second matrix in the clustering result whose adjacent distance is greater than the target distance threshold; delete the second matrix in the clustering result, and perform the step of determining a first matrix in the clustering result whose adjacent distance is less than the target distance threshold on the deleted clustering result.

[0084] Specifically, Figure 3 As shown, Figure 3 This is a visualization diagram of the clustering results involved in the embodiment of this application. Figure 3 In the clustering result, the second matrix, i.e., the discrete points, whose adjacent distances are greater than the target distance threshold are removed, and the second matrix is ​​deleted in the clustering result, thereby completing the noise removal. Then, the first matrix whose adjacent distances are less than the target distance threshold is determined in the clustering result for the deleted clustering result.

[0085] Step S40: Group the keys and values ​​of the multi-head attention according to the result matrix, and determine the number of groups according to the grouping results.

[0086] In this embodiment, the result matrix is ​​composed of multiple target matrices, and one or more key features are selected as the basis for grouping according to the feature average value in the result matrix. These features should be key attributes that can distinguish different clusters or categories. For each selected feature, one or more grouping thresholds are set. These thresholds are used to divide data points into different groups. The setting of thresholds can be determined according to the distribution of data, actual needs or statistical methods. Then each multi-head vector (i.e., each data point) is traversed and compared with the set grouping threshold according to its feature value. Based on the comparison result, each data point is assigned to the corresponding group. If a data point meets the conditions of multiple groups, it can be selected according to priority or other rules. After completing the grouping operation, the number of different groups, i.e., the number of groups, is counted.

[0087] Optionally, in this embodiment, the step of grouping the keys and values ​​of the multi-head attention according to the result matrix and determining the number of groups according to the grouping result includes:

[0088] Determine the number of the result matrices; group the keys and values ​​of the multi-head attention according to the number of the result matrices; determine the target number of the keys and values ​​after grouping, and determine the target number as the number of groups.

[0089] Specifically, Figure 4 As shown, Figure 4 This is a schematic diagram of the multi-head attention grouping involved in the embodiment of the present application. Figure 4 It can be seen that before grouping, the number of key and value groups in the multi-head attention is equal to the number of heads in the multi-head vector query, that is, a group includes a key, a value and a multi-head vector query. When grouping the multi-head attention, the result matrix is ​​composed of cluster data points of different types. The different types here refer to the cluster data points in different regions extracted by the clustering algorithm mentioned above. By confirming the number of regions of cluster data points, and dividing the number of key groups corresponding to the number according to the number, and then dividing the number of values, the above key groups are assigned to the value, thereby completing the grouping of the keys and values ​​of the multi-head vector. The final result is as follows. Figure 4 As shown in the figure, a group consists of two multi-headed vectors and a single key and value. After the grouping is completed, the number of key groups is determined as the hyperparameter of the number of groups.

[0090] Furthermore, in this embodiment, after the step of grouping the keys and values ​​of the multi-head attention according to the result matrix and determining the number of groups according to the grouping result, it also includes:

[0091] The number of groups is input as a parameter into the multi-head attention, and the parameters of the model are fine-tuned according to the multi-head attention; and the model is trained according to the fine-tuned parameters.

[0092] Specifically, after the number of groups is determined, the number of groups of input data is input as a parameter into the multi-head attention mechanism. According to the input data and the output results of the multi-head attention, the parameters of the model are fine-tuned. The model is then trained using the fine-tuned parameters to improve its performance on specific tasks.

[0093] It should be noted that here, "number of groups" refers to the number of different parts or batches that the input data is divided into. These numbers of groups are used as input parameters of the multi-head attention mechanism. By splitting the input data into multiple "heads" (i.e., multiple subspaces) and then calculating the attention weights in these subspaces in parallel, the model's ability to capture different aspects of information is improved. Fine-tuning refers to the process of making small adjustments to the model parameters based on the data of a specific task based on the use of a pre-trained model. Based on the number of input groups and the output results of the multi-head attention, the parameters of the model are fine-tuned to better suit the needs of the current task. After parameter fine-tuning, these updated parameters are used to continue training the model, where the training process includes applying the model to the training dataset and adjusting the parameters based on the difference between the model's predicted results and the actual results to minimize the loss function.

[0094] In the technical solution provided in this embodiment, at least two multi-head vectors corresponding to the multi-head attention are first compressed into single-dimensional vectors, and then the cluster order is determined according to the single-dimensional vector, and the clustering result in the cluster order is extracted based on the clustering algorithm. The clustering result is composed of at least one matrix, and then average pooling is performed based on the target matrix of type matching in the clustering result to merge into a result matrix. Finally, the keys and values ​​of the multi-head attention are grouped according to the result matrix, and the number of groups is determined according to the grouping result. The solution of this embodiment clusters the multi-head vectors through an adaptive clustering algorithm to generate clustering results, and then completes the grouping of the multi-head vectors according to the clustering results to improve the efficiency of model training.

[0095] Embodiment 2

[0096] Please refer to Figure 5 In the second embodiment, step S20 includes:

[0097] Step S50: Determine the one-dimensional vector as a data point, and determine the specific number of the data points.

[0098] Step S60: Obtain a preset distance threshold, and determine a sample parameter according to the specific quantity.

[0099] In this embodiment, the single-dimensional vector is processed by a density-based clustering algorithm. First, the set of single-dimensional vectors is regarded as data points in a coordinate system, and the set is traversed to calculate the total number of data points, which is the specific number. Then, a preset distance threshold is determined, and sample parameters are set according to the specific number.

[0100] Specifically, the distance threshold is eps, which is an important parameter for determining the neighborhood range between a sample and other samples, and min_samples represents a sample parameter, which is used to define the minimum number of samples of a core point. In OPTICS (Ordering Points To Identify the Clustering Structure, a density-based clustering algorithm), data points can be clustered by eps and min_samples.

[0101] Exemplarily, the processing steps of the OPTICS clustering algorithm are as follows: by selecting an unprocessed point as the starting point, calculating its core distance, and setting the reachable distance of all points to undefined. For the current point, find its direct density reachable points and calculate the reachable distances of these points. This usually involves determining a neighborhood radius, and the number of points contained in the radius should be greater than or equal to min_samples. The direct density reachable points of the current point are inserted into the clustering sequence graph according to the size of the reachable distance. The clustering sequence graph is an ordered data structure. Then select a point from the unprocessed points, usually the point with the smallest reachable distance, and repeat the steps of finding its direct density reachable points and inserting the direct density reachable points of the current point into the clustering sequence graph according to the size of the reachable distance until all points are processed.

[0102] Step S70: Processing the data points according to the distance threshold and the sample parameters to generate the cluster ranking.

[0103] Step S80: extracting clustering results in the cluster sorting based on a clustering algorithm.

[0104] In this embodiment, an empty data structure is initialized to store the generated cluster ordering. For each data point, its distance to other data points is calculated. Figure 2As shown, if the distance is less than or equal to the preset distance threshold eps, the data point is added to the nearest cluster. If the distance is greater than eps, check whether a new cluster needs to be created to include the data point. Repeat the above steps until all data points are assigned to a cluster. According to the distribution and distance relationship of the data points in the cluster, the clusters are sorted to obtain a cluster sorting. Finally, the clustering results in the cluster sorting are extracted based on the OPTICS clustering algorithm.

[0105] It should be noted that when generating the cluster sorting, eps is set to infinity and min_samples is set according to one-fourth of the number of heads in the multi-head vector.

[0106] Optionally, in this embodiment, the step of extracting the clustering result in the cluster sorting based on the clustering algorithm includes:

[0107] Determine a target distance threshold and a target sample parameter according to the cluster sorting; determine the number of categories in the cluster sorting according to the target distance threshold and the target sample parameter, and divide the cluster sorting into at least one data point set according to the number of categories; output the data point set and determine it as the clustering result.

[0108] Specifically, the clustering results in the cluster sorting are extracted by the OPTICS clustering algorithm. The generated cluster sorting is first obtained. The cluster sorting is an ordered list, which contains the sorting of data points according to eps, and the sorting reflects the structural relationship between the data points. Then determine the target distance threshold. The target distance threshold is a new eps, which is determined based on the distance distribution in the cluster sorting, and is usually selected at the bottom of the distance range. The threshold can be manually selected by observing the histogram or curve graph of the distance distribution, or it can be determined using an automated method such as a threshold selection algorithm based on statistics. Similarly, determine the target sample parameter, and the target sample parameter min_samples can be set according to the number of clusters in the cluster sorting.

[0109] After the target distance threshold eps and the target sample parameter min_samples are set, the clusters are divided according to the target distance threshold. Traverse the cluster sorting and classify the data points whose distance is less than or equal to the target distance threshold into the same cluster. Then determine the number of categories. The number of categories is the number of clusters divided. In the above steps, the number of categories increases by 1 each time a new cluster is formed. Then, based on the formed data point set, for each cluster, the data points contained in it are grouped into a set to obtain at least one data point set, and each set corresponds to a clustering result. Finally, the data point set is output, and each data point set is output as a clustering result. It can be represented in the form of a list, array, or other data structure.

[0110] In the technical solution provided in this embodiment, by determining the single-dimensional vector as a data point, determining the specific number of the data points, obtaining a preset distance threshold, and determining sample parameters according to the specific number, and then processing the data points according to the distance threshold and the sample parameters to generate the cluster sorting, and finally extracting the clustering results in the cluster sorting based on the clustering algorithm, the multi-head vector can be preliminarily processed and divided into cluster sorting to improve the accuracy of multi-head vector grouping.

[0111] Since the system introduced in the embodiment of the present application is a system used to implement the method of the embodiment of the present application, based on the method introduced in the embodiment of the present application, the person skilled in the art can understand the specific structure and deformation of the system, so it is not repeated here. All systems used in the method of the embodiment of the present application belong to the scope of protection of this application.

[0112] The present application provides a multi-head attention grouping device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-head attention grouping method in the above-mentioned embodiment one.

[0113] Reference below Figure 6 , which shows a schematic diagram of the structure of a multi-head attention grouping device suitable for implementing the embodiment of the present application. The multi-head attention grouping device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The multi-head attention grouping device shown is merely an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present application.

[0114] like Figure 6As shown, the multi-head attention grouping device may include a processing device 1001 (e.g., a core processor, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the multi-head attention grouping device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the multi-head attention grouping device to communicate wirelessly or wired with other devices to exchange data. Although the multi-head attention grouping device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0115] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0116] The multi-head attention grouping device provided by the present application adopts the multi-head attention grouping method in the above embodiment, which can solve the technical problem that the complexity of model parameter adjustment will increase by manually setting the number of groups in the related solution, thereby resulting in low efficiency of model training. Compared with the prior art, the beneficial effects of the multi-head attention grouping device provided by the present application are the same as the beneficial effects of the multi-head attention grouping method provided by the above embodiment, and the other technical features in the multi-head attention grouping device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0117] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0118] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0119] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the multi-head attention grouping method in the above-mentioned embodiment.

[0120] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0121] The above-mentioned computer-readable storage medium may be included in the multi-head attention grouping device; or it may exist independently without being assembled into the multi-head attention grouping device.

[0122] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the multi-head attention grouping device, the multi-head attention grouping device: compresses at least two multi-head vectors corresponding to the multi-head attention into single-dimensional vectors respectively; determines the cluster order according to the single-dimensional vector, and extracts the clustering results in the cluster order based on the clustering algorithm, wherein the clustering results are composed of at least one matrix; performs average pooling based on the target matrix of type matching in the clustering results, and merges them into a result matrix; groups the keys and values ​​of the multi-head attention according to the result matrix, and determines the number of groups according to the grouping results.

[0123] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0126] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned multi-head attention grouping method, and can solve the technical problem that the related scheme increases the complexity of model parameter adjustment by manually setting the number of groups, thereby resulting in low efficiency of model training. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the multi-head attention grouping method provided in the above-mentioned embodiment, and will not be repeated here.

[0127] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multi-head attention grouping method as described above.

[0128] The computer program product provided by the present application can solve the technical problem that the complexity of model parameter adjustment will increase by manually setting the number of groups in the related solutions, thereby resulting in low efficiency of model training. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as the beneficial effects of the multi-head attention grouping method provided by the above embodiment, which will not be repeated here.

[0129] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A multi-head attention grouping method, characterized in that: The multi-head attention grouping method comprises the following steps: Compress at least two multi-head vectors corresponding to the multi-head attention into single-dimensional vectors respectively; Determine a cluster order according to the single-dimensional vector, and extract a clustering result from the cluster order based on a clustering algorithm, wherein the clustering result is composed of at least one matrix; Perform average pooling based on the target matrices of type matching in the clustering results to merge into a result matrix; The keys and values ​​of the multi-head attention are grouped according to the result matrix, and the number of groups is determined according to the grouping result.

2. The method according to claim 1, characterized in that The step of compressing at least two multi-head vectors corresponding to the multi-head attention into single-dimensional vectors comprises: Determine the number of multi-head vectors in the multi-head attention, and construct a vector matrix according to the number and the multi-head vectors; Calculating the covariance matrix of the multi-head vector according to the relationship between the variables in the vector matrix; Decomposing the covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors; A dot product operation is performed according to the eigenvalue and the eigenvector to generate the one-dimensional vector, wherein the one-dimensional vector is a representation of the multi-headed vector in a one-dimensional space.

3. The method according to claim 1, characterized in that The step of determining the cluster order according to the single-dimensional vector and extracting the clustering result in the cluster order based on a clustering algorithm comprises: Determine the one-dimensional vector as a data point, and determine the specific number of the data points; Obtaining a preset distance threshold, and determining a sample parameter according to the specific quantity; Processing the data points according to the distance threshold and the sample parameters to generate the cluster ranking; A clustering result in the cluster sorting is extracted based on a clustering algorithm.

4. The method according to claim 3, characterized in that The step of extracting the clustering results in the cluster sorting based on the clustering algorithm comprises: Determining a target distance threshold and a target sample parameter according to the cluster sorting; Determining the number of categories in the cluster sorting according to the target distance threshold and the target sample parameter, and dividing the cluster sorting into at least one data point set according to the number of categories; The clustering result is determined according to the set of data points.

5. The method according to any one of claims 1 to 4, characterized in that: The step of performing average pooling on the target matrix based on the type matching in the clustering result to merge into a result matrix comprises: Determine a first matrix whose adjacent distance is less than the target distance threshold in the clustering result, and determine the first matrix as the target matrix of type matching; Performing dimensionality reduction processing on the target matrix; The target matrix after dimension reduction is average pooled to generate the result matrix.

6. The method according to claim 5, characterized in that After the step of determining in the clustering result a first matrix whose adjacent distance is less than the target distance threshold, and determining the first matrix as the target matrix of type matching, the method further comprises: Determine a second matrix in the clustering result whose adjacent distance is greater than the target distance threshold; The second matrix is ​​deleted from the clustering result, and the step of determining the first matrix whose adjacent distance is less than the target distance threshold in the clustering result is performed on the deleted clustering result.

7. The method according to claim 1, characterized in that The step of grouping the keys and values ​​of the multi-head attention according to the result matrix and determining the number of groups according to the grouping result comprises: determining the number of the result matrices; Grouping the keys and values ​​of the multi-head attention according to the number of the result matrix; A target number of the keys and values ​​after grouping is determined, and the target number is determined as the number of groups.

8. The method according to claim 1, characterized in that After the step of grouping the keys and values ​​of the multi-head attention according to the result matrix and determining the number of groups according to the grouping results, the method further includes: Inputting the number of groups as a parameter into the multi-head attention, and fine-tuning the parameters of the model according to the multi-head attention; The model is trained according to the fine-tuned parameters.

9. A multi-head attention grouping device, characterized in that: The multi-head attention grouping device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the multi-head attention grouping method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the multi-head attention grouping method as described in any one of claims 1 to 8 are implemented.

Images