Context information processing method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the field of artificial intelligence, and relates to a contextual information processing method, which comprises the following steps: acquiring an input feature set of voice signal data, performing multi-head segmentation processing on the input feature set to obtain a plurality of attention subspaces, acquiring a weight matrix in the attention subspaces, performing vector conversion based on the weight matrix, and obtaining a plurality of contextual information. The invention further provides a context information processing device, computer equipment and a storage medium. The context information processing method comprises the steps of obtaining a feature vector, carrying out matrix transformation processing based on the feature vector and a weight matrix to obtain a branch target matrix, and carrying out cascade processing on the branch target matrixes of all attention subspaces to generate target context information. According to the invention, interaction between different attention heads can be sensed at the same time, the dependency relationship between different voice signal features is improved, and context information can be efficiently captured at extremely low calculation cost in a voice recognition task.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to context information processing methods, devices, computer equipment and storage media. Background Art

[0002] In the field of speech recognition, the attention mechanism can greatly enhance the model's ability to understand and process complex contexts by giving weighted attention to the contextual information most relevant to the current processing task, thereby improving recognition accuracy and ensuring detailed and in-depth analysis of speech data.

[0003] There are currently a large number of improvements based on the attention mechanism, but there are still some problems that have not been solved. For example, in speech recognition tasks, although the self-attention mechanism can capture the global context, it is weak in extracting local features and requires a lot of computational cost in the process. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a context information processing method, apparatus, computer device and storage medium to solve the problem of how to efficiently capture context information and local features at a very low computational cost in speech recognition tasks.

[0005] In order to solve the above technical problems, the present application provides a context information processing method, which adopts the following technical solution:

[0006] Obtaining an input feature set of speech signal data;

[0007] Performing multi-head segmentation processing on the input feature set to obtain a plurality of attention subspaces;

[0008] Obtaining a weight matrix in the attention subspace, and performing vector transformation based on the weight matrix to obtain a feature vector;

[0009] Performing matrix transformation processing based on the eigenvector and the weight matrix to obtain a branch target matrix;

[0010] The branch target matrices of all attention subspaces are cascaded to generate target context information.

[0011] Furthermore, the step of performing multi-head segmentation processing on the input feature set to obtain a plurality of attention subspaces includes:

[0012] Obtaining a similarity score between every two of the input feature subsets;

[0013] Distributing the similarity scores through a normalization function to obtain attention weights for all input feature subsets;

[0014] The attention weights of the input feature subset are stored in the attention subspace corresponding to the input feature subset, wherein one feature subset corresponds to one attention subspace.

[0015] Furthermore, the weight matrix includes a query matrix, a key matrix and a value matrix, wherein the query matrix contains a plurality of query vectors, the feature vector includes the query vector, the key vector and the value vector, and the step of performing vector conversion based on the weight matrix to obtain the feature vector specifically includes:

[0016] Performing linear mapping on the input feature subset based on the query matrix to obtain the query vector;

[0017] Performing linear mapping on the input feature subset based on the key matrix to obtain the key vector;

[0018] The input feature subset is linearly mapped based on the value matrix to obtain the value vector.

[0019] Furthermore, the step of performing matrix transformation processing based on the eigenvector and the weight matrix to obtain a branch target matrix specifically includes:

[0020] Obtaining adjacent feature vectors of the query vector;

[0021] Performing an element-by-element multiplication operation on the query matrix and the key matrix corresponding to the adjacent feature vector to obtain a first target matrix;

[0022] Performing position encoding processing on the first target matrix to obtain a second target matrix;

[0023] Performing dimensional summation processing on the second target matrix to obtain a first target vector;

[0024] Multiplying the first target vector by the value matrix to obtain a branch context information vector;

[0025] A Kronecker product operation is performed on the branch context information vector and the value matrix to obtain a branch target matrix.

[0026] Furthermore, it is characterized in that the step of performing matrix transformation processing based on the eigenvector and the weight matrix further includes:

[0027] Performing a dot product operation on the key vector corresponding to the adjacent feature vector and the query vector to obtain a second target vector;

[0028] The second target vector is saved in a score matrix, and the score matrix is ​​saved in the attention subspace.

[0029] Furthermore, the step of cascading the branch target matrices of all attention subspaces to generate target context information specifically includes:

[0030] The attention subspaces are sorted in order according to the score weights of the score matrix to obtain a branch target matrix sequence;

[0031] The branch target matrices are sequentially subjected to matrix fusion processing according to the sorting order of the branch target matrix sequence to obtain target context information.

[0032] Furthermore, the step of inputting a feature set of the speech signal data specifically includes:

[0033] Preprocessing the speech signal data to obtain target processed data;

[0034] Performing short-time Fourier transform processing on the target processed data to obtain a spectrogram, wherein the spectrogram stores characteristic values ​​of the target processed data;

[0035] Perform Z-score normalization processing on the eigenvalues ​​of the target processed data to obtain a real number matrix, and use the real number matrix as the input feature set.

[0036] In order to solve the above technical problems, the embodiment of the present application further provides a context information processing device, which adopts the following technical solution:

[0037] A first acquisition module is used to acquire an input feature set of data to be processed, wherein the data to be processed is speech signal data;

[0038] A multi-head segmentation processing module, used for performing multi-head segmentation processing on the input feature set to obtain a plurality of attention subspaces;

[0039] A second acquisition module is used to acquire a weight matrix in the attention subspace, perform vector conversion based on the weight matrix, and obtain a feature vector, wherein the feature vector includes a query vector, a key vector, and a value vector, and the weight matrix includes a query matrix, a key matrix, and a value matrix, wherein the query matrix includes a plurality of query vectors, the key matrix includes a plurality of key vectors, and the value matrix includes a plurality of value vectors;

[0040] A matrix transformation processing module, used for performing matrix transformation processing based on the eigenvector and the weight matrix to obtain a branch target matrix;

[0041] The cascade module is used to cascade the branch target matrices of all attention subspaces to generate target context information.

[0042] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0043] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the context information processing method when executing the computer-readable instructions.

[0044] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0045] A computer-readable storage medium, characterized in that computer-readable instructions are stored on the computer-readable storage medium, and when the computer-readable instructions are executed by a processor, the steps of the context information processing method are implemented.

[0046] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0047] The embodiment of the present application obtains an input feature set of data to be processed, wherein the data to be processed is speech signal data, performs multi-head segmentation processing on the input feature set to obtain several attention subspaces, obtains a weight matrix in the attention subspace, performs vector conversion based on the weight matrix to obtain a feature vector, performs matrix transformation processing based on the feature vector and the weight matrix to obtain a branch target matrix, cascades the branch target matrices of all attention subspaces to generate target context information. The present application can extract local features, thereby enhancing the perception and expression capabilities of the model, while perceiving the interaction between different heads, improving the dependency between different features, enhancing the generalization and robustness of the model, and can also reduce computational complexity and improve generalization performance while maintaining high-precision classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0050] Figure 2 A flowchart of an embodiment of a context information processing method according to the present application;

[0051] Figure 3is a structural diagram of an embodiment of a context information processing device according to the present application;

[0052] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0054] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0055] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0056] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0057] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0058] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers and desktop computers, etc.

[0059] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0060] It should be noted that the context information processing method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the context information processing device is generally arranged in the server / terminal device.

[0061] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0062] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for processing context information according to the present application. The context information processing method comprises the following steps:

[0063] Step S201, obtaining an input feature set of speech signal data.

[0064] In this embodiment, the context information processing method can be deployed in a self-attention module in a speech recognition system. The speech recognition processing platform can be constructed by a server or a server cluster. The server or server cluster can be any electronic device with functions such as speech recognition, speech processing, data transmission, and data storage. The context information processing method runs on the electronic device (for example, Figure 1 The server / terminal device shown in the figure can receive the data to be processed through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultrawideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0065] In this embodiment, the voice signal data may be digital information extracted from a user's call record. Specifically, the voice signal data may include frequency characteristics, amplitude, duration, and a time-frequency diagram of the sound.

[0066] In this embodiment, the input feature set may be a real number matrix, which may include a real number matrix of frequency features, a real number matrix of amplitude, and a real number matrix of timbre features, etc. The real number matrix may be a matrix with a dimension of T×D, and the matrix may be expressed as:

[0067] x∈R T×D

[0068] Among them, the above x is a real number matrix (i.e., the above input feature set), the above T is the total sequence length (number of rows) of the real number matrix, i.e., the number of elements in the sequence, D is the feature dimension (number of columns) representing each sequence element, and the above R is the identifier of the current matrix. Each row of the above real number matrix represents a data point, and each column represents a speech feature. Representing the input feature set of the data to be processed in the form of a real number matrix can provide standardized input data for subsequent self-attention module processing, thereby improving the processing efficiency of the self-attention module.

[0069] Step S202, performing multi-head segmentation processing on the input feature set to obtain several attention subspaces.

[0070] In this embodiment, the independent parts generated after the input feature set is segmented and processed by the multi-head attention mechanism can be simply referred to as "heads", and the above-mentioned attention subspace can be the dimensional space of the feature subset processed by each "head".

[0071] In this embodiment, the multi-head segmentation processing can be to segment the feature dimension of the input feature set according to a preset number of segmentations. For example, assuming that the feature dimension of the input feature set is 512, and we set 8 heads (i.e., the attention subspaces mentioned above), then each head will process a 64-dimensional feature subset. By performing multi-head segmentation processing on the input feature set, several attention subspaces can be obtained, which can realize the capture of information from multiple angles at the same time when processing speech signal data, thereby increasing the expressive power and flexibility of the self-attention module.

[0072] Specifically, regarding the specific implementation process of performing multi-head segmentation processing on the input feature set, this application will provide further detailed descriptions in subsequent specific embodiments, and will not be elaborated on here.

[0073] Step S203, obtaining a weight matrix in the attention subspace, performing vector transformation based on the weight matrix, and obtaining a feature vector.

[0074] In this embodiment, the feature vector includes a query vector, a key vector and a value vector.

[0075] Specifically, the query vector may be a representation of the currently processed input element, which may be a word or a piece of speech data, and is used to match key vectors of other elements. The query vector may be used as a way to find and evaluate the strength of the relationship between other input elements and the current element.

[0076] The key vector may be a representation of each element in the input feature set, which is paired with the query vector to calculate the similarity or association between them. The matching degree of each key vector with the query vector may be evaluated by the dot product or other similarity measure between the query vector and the key vector.

[0077] The above value vector can be the content of each element in the input feature set. When the query vector matches the key vector highly, the relevant value vector will be used to calculate the final output. The above value vector provides the actual content with high attention in a given query context.

[0078] In this embodiment, the weight matrix includes a query matrix, a key matrix, and a value matrix. The weight matrix can be used to generate a query vector, a key vector, and a value vector.

[0079] Specifically, the query matrix, key matrix and value matrix can be obtained by linearly transforming the input feature set through the fully connected layer in the attention module, wherein the query matrix can be a weight matrix for converting the input feature set into a query vector, the key matrix can be a weight matrix for converting the input feature set into a key vector, and the value matrix can be a weight matrix for converting the input feature set into a value vector, wherein the query matrix can contain several query vectors, the key matrix can contain several key vectors, and the value matrix can contain several value vectors.

[0080] Step S204: Perform matrix transformation processing based on the eigenvector and the weight matrix to obtain a branch target matrix.

[0081] In this embodiment, the above-mentioned matrix transformation processing can be linear transformation, affine transformation, orthogonal transformation, dot product and element-by-element multiplication operations. The complexity of the network in the self-attention module can be effectively reduced through matrix transformation processing.

[0082] In this embodiment, the branch target matrix may be a matrix representation of the output result of each head in the self-attention mechanism, the branch context information vector may be used to capture the core context information of the attention subspace, and the branch target matrix includes a plurality of branch context information vectors;

[0083] Specifically, regarding the specific implementation process of performing matrix transformation processing based on the feature vector and the weight matrix to obtain the branch target matrix and the branch context information vector, this application will provide further detailed descriptions in subsequent specific embodiments and will not be elaborated on here.

[0084] Step S205, cascade processing is performed on the branch target matrices of all attention subspaces to generate target context information.

[0085] In this embodiment, the above-mentioned target context information can be the product of the step-by-step fusion of branch context information in all branch target matrices. The above-mentioned cascade processing can be to connect multiple processing units (i.e., the above-mentioned attention subspace) in sequence, and the output of each unit will become the input of the next unit. This can solve the problem that the information between different heads in the multi-head self-attention mechanism is relatively independent and cannot be interacted in the attention mechanism.

[0086] Specifically, regarding the specific implementation process of cascading the branch target matrices of all attention subspaces to generate target context information, this application will provide further detailed descriptions in subsequent specific embodiments and will not be elaborated on here.

[0087] The present application obtains an input feature set of data to be processed, wherein the data to be processed is speech signal data, performs multi-head segmentation processing on the input feature set to obtain several attention subspaces, obtains a weight matrix in the attention subspace, performs vector conversion based on the weight matrix to obtain a feature vector, performs matrix transformation processing based on the feature vector and the weight matrix to obtain a branch target matrix, cascades the branch target matrices of all attention subspaces to generate target context information. The present application can extract local features, thereby enhancing the perception and expression capabilities of the model, while perceiving the interaction between different heads, improving the dependency between different features, enhancing the generalization and robustness of the model, and can also reduce computational complexity and improve generalization performance while maintaining high-precision classification.

[0088] In some optional implementations, step S201 includes the following steps:

[0089] Preprocessing the speech signal data to obtain target processed data;

[0090] Performing short-time Fourier transform processing on the target processed data to obtain a spectrogram, wherein the spectrogram stores characteristic values ​​of the target processed data;

[0091] Perform Z-score normalization processing on the eigenvalues ​​of the target processed data to obtain a real number matrix, and use the real number matrix as the input feature set.

[0092] In this embodiment, the preprocessing may include voice sampling rate adjustment, noise removal, pre-emphasis and other processing.

[0093] In this embodiment, Short-Time Fourier Transform (STFT) can be a method for analyzing the time-varying frequency of a signal. By dividing the speech signal data into short-time segments and then applying Fourier transform to each segment, the frequency components of the signal at different time points can be revealed.

[0094] In a possible embodiment of the present invention, the speech signal data is subjected to speech sampling rate adjustment, noise removal, pre-emphasis and other processing to obtain target processing data, and the target processing data can be divided into short-time frames by fixed-length framing, overlapping framing or window function weighted framing, and the discontinuity of the edge of the short-time frame is reduced by applying a window function such as a Blackman window, a Dolph Chebyshev window and a Knott window to each short-time frame to obtain a windowed frame, and a spectrum is calculated using a fast Fourier transform for the window frame to obtain a frequency domain representation of each window frame, and the spectrum of each frame is divided into two parts by time. The data in the above spectrogram are arranged in sequence to form a two-dimensional spectrum diagram (spectrogram), in which the horizontal axis is time and the vertical axis is the speech signal frequency. The value of each point represents the intensity or amplitude of the speech signal frequency at that time point. The data in the above spectrogram is subjected to maximum and minimum normalization or Z-score standardization processing to keep the eigenvalues ​​in the spectrogram within a uniform range. Finally, the processed spectrogram is arranged into a real number matrix, in which each column represents the speech signal frequency distribution characteristics of a time frame, and each row represents the speech signal intensity of different speech signal frequencies at different times.

[0095] This application can effectively extract and structure speech information by converting speech signal data into a spectrogram and forming a real number matrix, providing a standard input data set for subsequent self-attention model processing and improving the efficiency of subsequent context information processing.

[0096] In some optional implementations, step S202 includes the following steps:

[0097] Divide the input feature set into several input feature subsets and generate a corresponding number of attention subspaces;

[0098] Get the similarity score between each two input feature subsets;

[0099] Distribute the similarity scores through a normalization function to obtain the attention weights of the input feature subsets;

[0100] The attention weights of the input feature subset are stored in the attention subspace corresponding to the input feature subset.

[0101] In this embodiment, one feature subset corresponds to one attention subspace.

[0102] Specifically, the above input feature subset can be expressed as:

[0103] X∈R T×D / H

[0104] Among them, the above X represents the real matrix of the input feature subset, T is the total sequence length, D is the feature dimension, TXD is the matrix dimension, the above R is the identifier of the current matrix, and H refers to the number of heads that process information in parallel in the multi-head attention mechanism;

[0105] Specifically, the feature dimension of the real number matrix (input feature set) can be divided according to the preset number of heads (H) to obtain H local matrices (input feature subsets) of the same matrix size, and the above local matrices can be stored in the corresponding attention subspaces respectively.

[0106] In a possible embodiment, after obtaining the above-mentioned input feature subsets, in order to determine the correspondence between the above-mentioned input feature subsets and the above-mentioned attention subspace, the similarity score between every two input feature subsets can be calculated by cosine similarity, and the attention weights of all input feature subsets are allocated according to the calculated similarity scores through the softmax normalization function to ensure that the sum of the attention weights of each input feature subset to all other input feature subsets is 1, and the input feature subsets are stored in the attention subspace in order according to the size of the attention weights.

[0107] The present application divides the input feature set into several input feature subsets and stores them in different attention subspaces, so that the self-attention module can process the input feature subsets in parallel, and each attention subspace can focus on capturing different aspects or features of the input feature subset.

[0108] In some optional implementations, step S203 includes the following steps:

[0109] Linearly map the input feature subset based on the query matrix to obtain the query vector;

[0110] Linearly map the input feature subset based on the key matrix to obtain the key vector;

[0111] Linearly map the input feature subset based on the value matrix to obtain a value vector.

[0112] In this embodiment, the linear mapping may be a function from one vector space to another vector space. Specifically, the query vector may be determined by multiplying the input feature subset by the query matrix, the key vector may be determined by multiplying the input feature subset by the key matrix, and the value vector may be determined by multiplying the input feature subset by the value matrix. The specific formulas are as follows:

[0113] Q = xW Q

[0114] K=xW K

[0115] V=xW V

[0116] Among them, the above x is the input feature subset, Q is the query vector, K is the key vector, V is the value vector, and W is Q is the query matrix, the above W K is the bond matrix, the above W V is a value matrix.

[0117] Through specialized linear transformations, each head can focus on a different set of features in the input and optimize its response to these features, thereby making the attention mechanism more precise and effective.

[0118] In some optional implementations, step S204 includes the following steps:

[0119] Get the adjacent feature vectors of the query vector;

[0120] Performing an element-by-element multiplication operation on the query matrix and the key matrix corresponding to the adjacent feature vector to obtain a first target matrix;

[0121] Performing position encoding processing on the first target matrix to obtain a second target matrix;

[0122] Perform dimension sum processing on the second target matrix to obtain the first target vector;

[0123] Multiply the first target vector by the value matrix to obtain a branch context information vector;

[0124] Perform a Kronecker product operation on the branch context information vector and the value matrix to obtain the branch target matrix.

[0125] In this embodiment, the adjacent feature vector may be tokens adjacent to the query vector. The tokens may be basic units of the input data subset, and may be represented as a vector, a word, or a word unit, etc. The key matrix corresponding to the adjacent feature vector may be represented as:

[0126] K=T i ×D / H(Ti < <T)

[0127] Among them, K is the key matrix corresponding to the adjacent feature vector, T is the total sequence length of the input data subset, Ti is the local sequence length less than the total sequence length T, D / H is the feature dimension of the input data subset, D is the feature dimension of the input data set, and H is the number of heads of the multi-head self-attention mechanism.

[0128] In this embodiment, the element-by-element multiplication operation can be a Hadamard Product. The above-mentioned Hadamard Product can be used to describe the operation of multiplying the elements of two matrices of the same dimension at corresponding positions. The result of the Hadamard Product operation is another matrix of the same dimension (the first target matrix). The matrix dimension size of the above-mentioned first target matrix is ​​the same as the key matrix corresponding to the adjacent eigenvector, and each element thereof is the product of the corresponding elements of the original two matrices.

[0129] In this embodiment, the position coding processing can be a technology for adding sequence information to the first target matrix in the input data subset. The position coding processing can be implemented through coding techniques such as sine and cosine function coding, relative position coding, RPE (Rotary Position Encoding), etc. to obtain the encoded second target matrix.

[0130] In this embodiment, the dimensional summation process for the second target matrix may be to perform dimensionality reduction process on the second target matrix, and sum the values ​​on the characteristic dimension of the second target matrix into a scalar value or a vector (first target vector) after dimensionality reduction. The representation of the first target vector may be:

[0131] 1×T i

[0132] Among them, 1 represents the sequence length of the first target vector, T i The feature dimension of the first target vector.

[0133] In this embodiment, the Kronecker product operation can be a binary operation on a matrix, which can be used to expand the matrix operation. Specifically, the branch context information vector is subjected to the Kronecker product operation with the value matrix. The branch context information vector can be treated as a matrix for operation to generate a branch target matrix.

[0134] In a possible embodiment, when calculating the interaction relationship between each query vector and the key vector, the entire T×D / H matrix (input feature subset) is not multiplied, but only the tokens between the adjacent positions of the query vector are multiplied element by element to obtain a first target matrix of the same size as the key matrix, which greatly reduces the complexity of the network, and adds position information to the first target matrix, and then verifies the D / H dimension of the second target matrix and performs matrix summation to obtain a 1×T i The first target vector is then multiplied by the value matrix, i.e., a 1×T i The vector and T i ×D / H matrices are multiplied to obtain a 1×D / H vector (branch context information vector), and then the branch context information vector is Kronecker-producted with the value matrix to obtain the branch target matrix.

[0135] This application can effectively reduce the computational complexity of the attention module and improve the processing speed by performing element-by-element multiplication operations on the query vector and its adjacent feature vectors, and adding position encoding and dimensional summation processing. It can also enhance the attention subspace's utilization of position information in the input feature subset, thereby improving the flexibility and generalization ability of the attention subspace.

[0136] In some optional implementations, step S204 further includes the following steps:

[0137] Perform a dot product operation on the key vector corresponding to the adjacent feature vector and the query vector to obtain a second target vector;

[0138] Save the second target vector in the score matrix and save the score matrix in the attention subspace.

[0139] In this embodiment, the dot product operation can be a calculation method for measuring the relationship between two vectors in space. The purpose of the dot product operation is to quantify the similarity between the two vectors. The above score matrix can be used to store the similarity score of the second target vector.

[0140] By storing the score matrix into the attention subspace, the present application can process the attention subspace using the weight value of the score matrix, such as how to allocate the computing resources and attention of the attention subspace, which can reduce the computational complexity of the self-attention module and improve the generalization performance while maintaining high-precision classification.

[0141] In some optional implementations, step S205 includes the following steps:

[0142] Sort the attention subspaces in order according to the score weights of the score matrix to obtain a branch target matrix sequence;

[0143] The branch target matrices are sequentially subjected to matrix fusion processing according to the sorting order of the branch target matrix sequence to obtain target context information.

[0144] In this embodiment, the output results obtained by sequentially sorting the score weights of the score matrix determine the sequence order of each attention subspace, and the above branch target matrix sequence defines the priority of the attention subspace in the subsequent processing flow.

[0145] In this embodiment, matrix fusion can be: the output result (branch context information vector) calculated by the attention module of the first head (branch target matrix) is fused with the output result of the second head, and the cycle is repeated until the last head fuses the branch context information vectors of all heads to generate the target context information.

[0146] This application uses the score weights of the score matrix to sequentially sort and fuse the branch target matrix of the attention subspace, ensuring that the priority of branch target matrix processing is consistent with the relative importance of the data, and sequentially fuses the output results of different heads. The self-attention module can comprehensively consider information from multiple angles, and the target context information finally generated is more comprehensive and accurate.

[0147] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned voice signal data, the above-mentioned voice signal data can also be stored in a node of a blockchain.

[0148] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0149] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0150] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0151] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a context information processing device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0152] like Figure 3 As shown, the context information processing device 300 described in this embodiment includes: a first acquisition module 301, a multi-head segmentation processing module 302, a second acquisition module 303, a matrix transformation processing module 304 and a cascade module 305. Among them:

[0153] A first acquisition module 301 is used to acquire an input feature set of speech signal data;

[0154] A multi-head segmentation processing module 302 is used to perform multi-head segmentation processing on the input feature set to obtain a plurality of attention subspaces;

[0155] A second acquisition module 303 is used to acquire a weight matrix in the attention subspace, and perform vector conversion based on the weight matrix to obtain a feature vector;

[0156] A matrix transformation processing module 304 is used to perform matrix transformation processing based on the feature vector and the weight matrix to obtain a branch target matrix;

[0157] The cascade module 305 is used to cascade the branch target matrices of all attention subspaces to generate target context information.

[0158] The first acquisition module 301 includes:

[0159] A preprocessing submodule, used for preprocessing the speech signal data to obtain target processed data;

[0160] A first processing submodule is used to perform short-time Fourier transform processing on the target processing data to obtain a spectrogram, wherein the spectrogram stores characteristic values ​​of the target processing data;

[0161] The standardization processing submodule is used to perform Z-score standardization processing on the eigenvalues ​​of the target processing data to obtain a real number matrix, and use the real number matrix as the input feature set.

[0162] The multi-head segmentation processing module 302 includes:

[0163] A partitioning submodule, used to partition the input feature set into a plurality of input feature subsets and generate a corresponding number of attention subspaces;

[0164] A first acquisition submodule, used for acquiring a similarity score between every two of the input feature subsets;

[0165] An allocation submodule, used for allocating the similarity scores through a normalization function to obtain attention weights of input feature subsets;

[0166] The storage submodule is used to store the attention weight of the input feature subset into the attention subspace corresponding to the input feature subset, wherein one feature subset corresponds to one attention subspace.

[0167] The second acquisition module 303 includes:

[0168] A first linear mapping submodule, configured to perform linear mapping on the input feature subset based on the query matrix to obtain the query vector;

[0169] A second linear mapping submodule, configured to perform linear mapping on the input feature subset based on the key matrix to obtain the key vector;

[0170] The third linear mapping submodule is used to perform linear mapping on the input feature subset based on the value matrix to obtain the value vector.

[0171] The matrix transformation processing module 304 includes:

[0172] A second acquisition submodule, used to acquire an adjacent feature vector of the query vector;

[0173] An operator module, configured to perform an element-by-element multiplication operation on the query matrix and a key matrix corresponding to the adjacent feature vector to obtain a first target matrix;

[0174] A second processing submodule is used to perform position encoding processing on the first target matrix to obtain a second target matrix;

[0175] A third processing submodule is used to perform dimensional summation processing on the second target matrix to obtain a first target vector;

[0176] a fourth processing submodule, configured to multiply the first target vector by the value matrix to obtain a branch context information vector;

[0177] The fifth processing submodule is used to perform a Kronecker product operation on the branch context information vector and the value matrix to obtain a branch target matrix.

[0178] After the matrix transformation processing module, 304 further includes:

[0179] A dot product operation module, used for performing a dot product operation on the key vector corresponding to the adjacent feature vector and the query vector to obtain a second target vector;

[0180] A saving module is used to save the second target vector in a score matrix and save the score matrix in the attention subspace.

[0181] The cascade module 305 includes:

[0182] A sorting submodule, used to sort the attention subspaces in order according to the score weights of the score matrix to obtain a branch target matrix sequence;

[0183] The fusion submodule is used to perform matrix fusion processing on the branch target matrices in sequence according to the sorting order of the branch target matrix sequence to obtain target context information.

[0184] In this embodiment, the operations performed by the above units or modules correspond one-to-one to the steps of the context information processing method in the above implementation mode, and are not described in detail here.

[0185] In this embodiment, by obtaining an input feature set of data to be processed, the data to be processed is speech signal data, the input feature set is subjected to multi-head segmentation processing to obtain several attention subspaces, the weight matrix in the attention subspace is obtained, vector conversion is performed based on the weight matrix to obtain a feature vector, matrix transformation processing is performed based on the feature vector and the weight matrix to obtain a branch target matrix, the branch target matrices of all attention subspaces are cascaded to generate target context information. The present application can extract local features, thereby enhancing the perception and expression capabilities of the model, while perceiving the interaction between different heads, improving the dependency between different features, enhancing the generalization and robustness of the model, and can also reduce computational complexity and improve generalization performance while maintaining high-precision classification.

[0186] To solve the above technical problems, the present application also provides a computer device. Figure 4, Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0187] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0188] The computer device may be a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0189] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the context information processing method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0190] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the context information processing method.

[0191] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0192] The present embodiment provides a computer device, which obtains an input feature set of data to be processed, wherein the data to be processed is speech signal data, performs multi-head segmentation processing on the input feature set to obtain several attention subspaces, obtains a weight matrix in the attention subspace, performs vector conversion based on the weight matrix to obtain a feature vector, performs matrix transformation processing based on the feature vector and the weight matrix to obtain a branch target matrix, cascades the branch target matrices of all attention subspaces to generate target context information. The present application can extract local features, thereby enhancing the perception and expression capabilities of the model, while perceiving the interaction between different heads, improving the dependency between different features, enhancing the generalization and robustness of the model, and can also reduce computational complexity and improve generalization performance while maintaining high-precision classification.

[0193] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the context information processing method as described above.

[0194] The present embodiment provides a computer-readable storage medium, which obtains an input feature set of data to be processed, wherein the data to be processed is speech signal data, performs multi-head segmentation processing on the input feature set to obtain several attention subspaces, obtains a weight matrix in the attention subspace, performs vector conversion based on the weight matrix to obtain a feature vector, performs matrix transformation processing based on the feature vector and the weight matrix to obtain a branch target matrix, cascades the branch target matrices of all attention subspaces to generate target context information. The present application can extract local features, thereby enhancing the perception and expression capabilities of the model, while perceiving the interaction between different heads, improving the dependency between different features, enhancing the generalization and robustness of the model, and can also reduce computational complexity and improve generalization performance while maintaining high-precision classification.

[0195] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0196] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A context information processing method, applied to a self-attention module of a speech recognition model, characterized in that: The steps include: Obtaining an input feature set of speech signal data; Performing multi-head segmentation processing on the input feature set to obtain a plurality of attention subspaces; Obtaining a weight matrix in the attention subspace, and performing vector transformation based on the weight matrix to obtain a feature vector; Performing matrix transformation processing based on the feature vector and the weight matrix to obtain a branch target matrix; The branch target matrices of all attention subspaces are cascaded to generate target context information.

2. The context information processing method according to claim 1, characterized in that: The step of performing multi-head segmentation processing on the input feature set to obtain a plurality of attention subspaces includes: Dividing the input feature set into a plurality of input feature subsets and generating a corresponding number of attention subspaces; Obtaining a similarity score between every two of the input feature subsets; Distributing the similarity scores through a normalization function to obtain attention weights of the input feature subsets; The attention weights of the input feature subset are stored in the attention subspace corresponding to the input feature subset, wherein one feature subset corresponds to one attention subspace.

3. The context information processing method according to claim 2, characterized in that: The weight matrix includes a query matrix, a key matrix and a value matrix, wherein the query matrix includes a plurality of query vectors, the feature vector includes a query vector, a key vector and a value vector, and the step of performing vector conversion based on the weight matrix to obtain the feature vector specifically includes: Performing linear mapping on the input feature subset based on the query matrix to obtain the query vector; Performing linear mapping on the input feature subset based on the key matrix to obtain the key vector; The input feature subset is linearly mapped based on the value matrix to obtain the value vector.

4. The context information processing method according to claim 3, characterized in that: The step of performing matrix transformation processing based on the eigenvector and the weight matrix to obtain a branch target matrix specifically includes: Obtaining adjacent feature vectors of the query vector; Performing an element-by-element multiplication operation on the query matrix and the key matrix corresponding to the adjacent feature vector to obtain a first target matrix; Performing position encoding processing on the first target matrix to obtain a second target matrix; Performing dimensional summation processing on the second target matrix to obtain a first target vector; Multiplying the first target vector by the value matrix to obtain a branch context information vector; A Kronecker product operation is performed on the branch context information vector and the value matrix to obtain a branch target matrix.

5. The context information processing method according to claim 4, characterized in that: After the step of performing matrix transformation processing based on the feature vector and the weight matrix, the method further includes: Performing a dot product operation on the key vector corresponding to the adjacent feature vector and the query vector to obtain a second target vector; The second target vector is saved in a score matrix, and the score matrix is ​​saved in the attention subspace.

6. The context information processing method according to claim 5, characterized in that: The step of cascading the branch target matrices of all attention subspaces to generate target context information specifically includes: The attention subspaces are sorted in order according to the score weights of the score matrix to obtain a branch target matrix sequence; The branch target matrices are sequentially subjected to matrix fusion processing according to the sorting order of the branch target matrix sequence to obtain target context information.

7. The context information processing method according to claim 1, characterized in that: The step of obtaining an input feature set of speech signal data specifically includes: Preprocessing the speech signal data to obtain target processed data; Performing short-time Fourier transform processing on the target processed data to obtain a spectrogram, wherein the spectrogram stores characteristic values ​​of the target processed data; Perform Z-score normalization processing on the eigenvalues ​​of the target processed data to obtain a real number matrix, and use the real number matrix as the input feature set.

8. A context information processing device, characterized in that: include: A first acquisition module, used to acquire an input feature set of speech signal data; A multi-head segmentation processing module, used for performing multi-head segmentation processing on the input feature set to obtain a plurality of attention subspaces; A second acquisition module is used to acquire a weight matrix in the attention subspace, and perform vector conversion based on the weight matrix to obtain a feature vector; A matrix transformation processing module, used for performing matrix transformation processing based on the feature vector and the weight matrix to obtain a branch target matrix; The cascade module is used to cascade the branch target matrices of all attention subspaces to generate target context information.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the context information processing method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the context information processing method according to any one of claims 1 to 7 are implemented.

Images