Data processing method and device, electronic equipment and storage medium

Abstract

The application discloses a data processing method and device, electronic equipment and storage medium, comprising: obtaining to-be-processed sequence data, and projecting the to-be-processed sequence data to a first subspace, a second subspace and a third subspace to generate first branch sequence data, a state space trunk subsequence and a state space gating subsequence, and third branch sequence data; performing local attention processing on the first branch sequence data to obtain a local attention result; determining a state space adjustment output based on the state space trunk subsequence and the state space gating subsequence; generating fusion weight information based on the third branch sequence data; and determining a fusion output result based on the fusion weight information, the local attention result and the state space adjustment output; and fusing the fusion output result with the to-be-processed sequence data to obtain an output feature. The application can improve the dependence on local fine-grained semantics and improve the capturing ability of spatial correlation features.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid growth of processing tasks such as long text understanding and long video processing, Structured State Space Models (SSMs), represented by Mamba, have become an important development direction for industrial-grade sequence modeling due to their ability to process long sequences with linear time complexity.

[0003] The core of the standard Mamba architecture is the Mamba Block. Its basic technical solution is as follows: After layer normalization and linear projection, the input sequence features are divided into two branches. One branch undergoes depthwise convolution with a fixed kernel size and activation function, then is fed into the Selective State Space Model (SSM) module. The other branch is activated by an activation function as a gating signal, which is multiplied by the output of the SSM module to obtain the product. The product is then linearly projected and connected to the residuals to achieve the block output.

[0004] However, the existing standard Mamba architecture relies on the SSM module for sequence modeling, which has a weak ability to capture local fine-grained semantic dependencies and spatial correlation features. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, and storage medium that can improve the existing Mamba architecture's weak ability to capture local fine-grained semantic dependencies and spatial correlation features.

[0006] This application provides a data processing method, the method comprising: The process involves: acquiring the sequence data to be processed and projecting it onto a first subspace to generate first branch sequence data; projecting the sequence data onto a second subspace to generate second branch sequence data, which reflects state space characteristics and includes a state space backbone subsequence and a state space gating subsequence; projecting the sequence data onto a third subspace to generate third branch sequence data; performing local attention processing on the first branch sequence data to obtain a local attention result; determining a state space adjustment output based on the state space backbone subsequence and the state space gating subsequence; generating fusion weight information based on the third branch sequence data; determining a fusion output result based on the fusion weight information, the local attention result, and the state space adjustment output; and fusing the fusion output result with the sequence data to be processed to obtain output features.

[0007] This application provides a data processing apparatus, the apparatus comprising: A sequence projection unit is used to acquire the sequence data to be processed, project the sequence data to be processed onto a first subspace to generate first branch sequence data; project the sequence data to be processed onto a second subspace to generate second branch sequence data, the second branch sequence data being sequence data reflecting state space characteristics, the second branch sequence data including state space backbone subsequences and state space gated subsequences; and project the sequence data to be processed onto a third subspace to generate third branch sequence data. A local attention unit is used to perform local attention processing on the first branch sequence data to obtain a local attention result. A state-space adjustment unit is used to determine the state-space adjustment output based on the state-space backbone subsequence and the state-space gating subsequence. The fusion weighting unit is used to generate fusion weight information based on the third branch sequence data; and to determine the fusion output result based on the fusion weight information, the local attention result, and the state space adjustment output. The data fusion unit is used to fuse the fusion output result with the sequence data to be processed to obtain output features.

[0008] In one embodiment, the state-space adjustment unit includes: A matrix generation subunit is used to generate a state matrix based on the main subsequence of the state space; The output determination subunit is used to determine the state space adjustment output based on the state matrix, the state space backbone subsequence, and the state space gating subsequence.

[0009] In one implementation, the matrix generating subunit includes: The frequency domain decomposition subunit is used to perform frequency domain decomposition on the main subsequence of the state space to obtain frequency domain features. The entropy calculation sub-unit is used to calculate the energy distribution entropy value of the frequency domain feature; The matrix generation sub-unit is used to generate the state matrix based on the energy distribution entropy value.

[0010] In one implementation, the frequency domain features include Fourier transform results and wavelet transform results; correspondingly, the frequency domain decomposition sub-units include: The Fourier three-subunit is used to perform discrete Fourier transform on the main subsequence of the state space to obtain the Fourier transform result; Three wavelet sub-units are used to perform discrete wavelet transform on the main sub-sequence of the state space to obtain the wavelet transform result; The entropy calculation sub-unit includes: The Fourier entropy value sub-unit is used to calculate the energy distribution entropy value of the Fourier transform result; The wavelet entropy value three sub-units are used to calculate the energy distribution entropy value of the wavelet transform result.

[0011] In one implementation, the matrix generates secondary sub-units, including: The first component has three sub-units, which are used to calculate the first component based on the energy distribution entropy value of the Fourier transform result, the energy distribution entropy value of the wavelet transform result, and the diagonal state matrix; The second component has three sub-units, which are used to calculate the second component based on the energy distribution entropy value of the Fourier transform result, the energy distribution entropy value of the wavelet transform result, and the discrete wavelet matrix; A component summation subunit is used to calculate the component summation of the first component and the second component, wherein the component summation is the state matrix. In one embodiment, an output determination subunit includes: The main output sub-unit is used to determine the state space main output based on the state matrix and the state space main sub-sequence; The adjustment output sub-unit is used to adjust the state space backbone output based on the state space gating sub-sequence to obtain the state space adjusted output.

[0012] In one implementation, the main output sub-unit includes: The input extraction subunit is used to extract the input features at time t from the main subsequence of the state space, where t is an integer greater than 1; The hidden state three sub-unit is used to calculate the hidden state at time t based on the hidden state at time t-1, the input features at time t, and the state matrix; The output feature three sub-units are used to calculate the output feature at time t based on the hidden state at time t and the input feature at time t; The feature combination three sub-units are used to combine the output features at multiple time points to obtain the state space backbone output.

[0013] In one implementation, adjusting the output secondary subunit includes: The activation processing subunit is used to perform activation function processing on the state space gated subsequence to obtain the activation processing result. The result multiplication sub-unit is used to perform element-wise multiplication of the activation processing result with the state space backbone output to obtain the state space adjustment output.

[0014] In one implementation, the fusion weighting unit is specifically used to: normalize the third branch sequence data to obtain the fusion weight information; And used for: performing a weighted operation on the local attention result and the state space adjustment output based on the fusion weight information to obtain the fusion output result.

[0015] In one implementation, the data fusion unit includes: A linear projection subunit is used to perform linear projection on the fusion output result to obtain a linear projection result, wherein the dimension of the linear projection result is the same as the dimension of the sequence data to be processed. The residual connection subunit is used to perform a residual connection between the linear projection result and the sequence data to be processed to obtain the output feature.

[0016] In one embodiment, the sequence projection unit includes: The second subspace projection subunit is used to project the sequence data to be processed onto the second subspace to generate a state space projection result. The projection splitting subunit is used to split the state space projection result to obtain the state space backbone subsequence and the state space gating subsequence.

[0017] In one implementation, the local attention unit includes: The molecular segmentation unit is used to segment the first branch sequence data according to preset window information to obtain multiple segmentation results; A self-attention processing subunit is used to perform self-attention processing on each of the segmentation results to obtain the corresponding self-attention result; The splicing subunit is used to splice multiple self-attention results according to the order of the multiple segmentation results in the first branch sequence data to obtain a splicing result; The local attention subunit is used to process the splicing result using an activation function to obtain the local attention result.

[0018] In one embodiment, the method is implemented by a trained data processing model; correspondingly, the apparatus further includes: A training projection unit is used to acquire training sequence data and project the training sequence data into a first subspace to generate training first branch sequence data; project the training sequence data into a second subspace to generate training second branch sequence data, the training second branch sequence data including training state space backbone subsequence and training state space gating subsequence; and project the training sequence data into a third subspace to generate training third branch sequence data. A local attention unit is trained to perform local attention processing on the training first branch sequence data to obtain the training local attention result. The training state matrix unit is used to generate a training state matrix based on the main subsequence of the training state space; An adjustment output determination unit is used to determine the training state space adjustment output based on the training state matrix, the training state space backbone subsequence, and the training state space gating subsequence. The training fusion weight unit is used to generate training fusion weight information based on the training third branch sequence data; and to determine the training fusion output result based on the training fusion weight information, the training local attention result, and the training state space adjustment output. The training fusion output unit is used to fuse the training fusion output result with the training sequence data to obtain training output features; The parameter update unit is used to backpropagate and update the parameters of the data processing model to be trained based on the training output features and the target output features, until the preset training termination condition is met, and the trained data processing model is obtained.

[0019] This application also provides an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute steps in any of the data processing methods provided in this application.

[0020] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the data processing methods provided in this application.

[0021] This application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps in any of the data processing methods provided in this application.

[0022] The data processing method provided in this application embodiment can acquire sequence data to be processed and project it into a first subspace, a second subspace, and a third subspace respectively to obtain corresponding first branch sequence data, second branch sequence data, and third branch sequence data. The second branch sequence data includes a state space backbone subsequence and a state space gating subsequence. Local attention processing is performed on the first branch sequence data to obtain a local attention result. A state matrix is ​​generated based on the state space backbone subsequence; then, based on the state matrix, the state space backbone subsequence, and the state space gating subsequence, the output of the second branch sequence data—the state space adjustment output—is obtained. Fusion weight information is generated based on the third branch sequence data, and then, based on the fusion weight information, the local attention result, and the state space adjustment output, the fusion output result is determined. The fusion output result is fused with the sequence data to be processed to obtain output features.

[0023] In this embodiment, three branch sequences can be obtained for parallel processing: a first branch sequence data, a second branch sequence data (including a state space backbone subsequence and a state space gated subsequence), and a third branch sequence data. The first branch sequence data is a local attention branch, the second branch sequence data is an SSM branch, and the third branch sequence data is a gated branch dynamically fusing the first two. The local attention branch and the SSM branch are connected in parallel, and a gated branch is introduced to dynamically fuse the local attention branch and the SSM branch. The SSM branch, based on the inherent recursive computation characteristics and linear time complexity of SSM, effectively captures long-distance dependency features in the sequence through the iterative evolution of state variables. The local attention branch and the SSM branch are set up in parallel, utilizing the local attention mechanism to specifically compensate for the potential shortcomings of the SSM branch in extracting local details, capturing short-distance, high-precision dependencies. Furthermore, this parallel architecture can maintain the efficient collaboration between the two branches, improving the capture of fine-grained local semantic dependencies. The gated branch can flexibly balance the local attention results and the SSM branch output, improving the ability to capture spatial correlation features. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1a This is a schematic diagram illustrating an application scenario of the data processing method provided in this application; Figure 1b This is a flowchart illustrating the data processing method provided in an embodiment of this application; Figure 1c A schematic flowchart of a data processing method provided in an embodiment of this application is shown; Figure 2 This is a flowchart illustrating a data processing method provided in another specific embodiment of this application; Figure 3 This is a schematic diagram of a data processing apparatus provided in one embodiment of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] This application provides a data processing method, apparatus, electronic device, and storage medium.

[0028] Specifically, the data processing device can be integrated into an electronic device, such as a terminal or server. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal and server can be directly or indirectly connected via wired or wireless communication; this application does not impose any restrictions on this connection.

[0029] In some embodiments, the data processing apparatus may also be integrated into multiple electronic devices, such as multiple servers, with the data processing method of this application being implemented by the multiple servers.

[0030] In some embodiments, the terminal can also be used as a server to implement some or all of the functions of a server.

[0031] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0032] Please see details Figure 1a The method provided in this application embodiment may include: acquiring sequence data to be processed, and projecting the sequence data to be processed onto a first subspace to generate first branch sequence data; projecting the sequence data to be processed onto a second subspace to generate second branch sequence data, wherein the second branch sequence data is sequence data reflecting state space characteristics, and the second branch sequence data includes a state space backbone subsequence and a state space gating subsequence; projecting the sequence data to be processed onto a third subspace to generate third branch sequence data; performing local attention processing on the first branch sequence data to obtain a local attention result; determining a state space adjustment output based on the state space backbone subsequence and the state space gating subsequence; generating fusion weight information based on the third branch sequence data; and determining a fusion output result based on the fusion weight information, the local attention result, and the state space adjustment output; and fusing the fusion output result with the sequence data to be processed to obtain output features.

[0033] The above method yields three parallel processing branch sequences: the first branch sequence data, the second branch sequence data (including the state space backbone subsequence and the state space gated subsequence), and the third branch sequence data. The first branch sequence data is the local attention branch, the second branch sequence data is the SSM branch, and the third branch sequence data is the gated branch dynamically fusing the first two. The local attention branch and the SSM branch are connected in parallel, and a gated branch is introduced to dynamically fuse the two branches. The SSM branch, based on the inherent recursive computation characteristics and linear time complexity of SSM, effectively captures long-distance dependency features in the sequence through the iterative evolution of state variables. The local attention branch, set up in parallel with the SSM branch, utilizes the local attention mechanism to specifically compensate for the potential shortcomings of the SSM branch in extracting local details, capturing short-distance, high-precision dependencies. Furthermore, this parallel architecture maintains efficient collaboration between the two branches, improving the capture of fine-grained local semantic dependencies. The gated branch can flexibly balance the local attention results and the SSM branch output, improving the ability to capture spatial correlation features.

[0034] The data processing method provided in this application can replace the Transformer block in the existing Transformer architecture or the Mamba block in the existing Mamba architecture without requiring large-scale modifications to the overall model framework.

[0035] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent should be explicitly obtained, and the collection, use and processing of related data should comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0036] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0037] In this embodiment, a data processing method is provided, such as... Figure 1b As shown, the specific process of this method may include the following steps 110 to 150: 110. Obtain the sequence data to be processed and project it onto the first subspace to generate the first branch sequence data; project the sequence data to be processed onto the second subspace to generate the second branch sequence data, wherein the second branch sequence data is sequence data reflecting the characteristics of the state space, and the second branch sequence data includes the state space backbone subsequence and the state space gating subsequence; project the sequence data to be processed onto the third subspace to generate the third branch sequence data.

[0038] The sequence data to be processed refers to the input sequence data to be processed, specifically the feature sequence after processing by the previous layer of the network, which forms the basis for subsequent calculations. Optionally, the sequence data to be processed can be image data segmented into multiple image patches, forming a sequence; it can also be a sequence of natural language text after word segmentation and vectorization; or it can be a time-series signal acquired by a sensor. It should be understood that the specific data type of the sequence data to be processed should not be construed as a limitation of this application. The sequence data to be processed can be represented as... .in, L For sequence length, D For feature dimensions.

[0039] Projection is the process of mapping data from the original space to a new feature space through a linear projection layer. Specifically, projection can be achieved through matrix multiplication.

[0040] A subspace refers to a feature space specifically partitioned for a particular task. Specifically, the first subspace is the feature space prepared for local attention processing; the second subspace is the feature space prepared for the State Space Model (SSM); and the third subspace is the feature space prepared for dynamic fusion gating. SSM is a mathematical model that describes the relationship between the input and output of a system through state variables. In sequence modeling, it uses discretized state iterations to model the temporal dependencies of a sequence, possessing the ability to process long sequences with linear time complexity.

[0041] Optionally, the step 110 of "projecting the sequence data to be processed onto the first subspace to generate the first branch sequence data" can be specifically implemented by the following formula: (1) in, This is the first branch sequence data; Refers to the sequence data to be processed The result of the normalization process is obtained by performing normalization. , These are all parameters determined during model training.

[0042] Optionally, step 110, "projecting the sequence data to be processed onto the second subspace to generate second branch sequence data, wherein the second branch sequence data is sequence data reflecting the characteristics of the state space, and the second branch sequence data includes state space backbone subsequences and state space gated subsequences," may specifically include the following steps A1 to A2: A1. Project the sequence data to be processed onto the second subspace to generate a state space projection result.

[0043] Alternatively, step A1 can be implemented using the following formula: (2) in, This is the result of the state space projection; Refers to the sequence data to be processed The result of the normalization process is obtained by performing normalization. , These are all parameters determined during model training.

[0044] A2. The state space projection result is split to obtain the state space backbone subsequence and the state space gating subsequence.

[0045] Continuing with the example above, the state space projection result can be... The process involves splitting the sequence to obtain the main subsequence of the state space. and state-space gated subsequence .

[0046] In the above process, it can be The former D Dimension split into ;Will After D Dimension split into .

[0047] Optionally, the step 110 of "projecting the sequence data to be processed onto the third subspace to generate the third branch sequence data" can be specifically implemented by the following formula: (3) in, This is the third branch sequence data; Refers to the sequence data to be processed The result of the normalization process is obtained by performing normalization. , These are all parameters determined during model training.

[0048] 120. Perform local attention processing on the first branch sequence data to obtain the local attention result.

[0049] Local attention processing refers to a mechanism that limits the scope of attention computation. Unlike standard self-attention mechanisms, which focus on the entire sequence, local attention processing focuses on a defined range around the current data point. The specific processing steps of local attention processing will be described in detail in steps 121 to 124 below.

[0050] Optionally, in one embodiment, step 120 may specifically include steps 121 to 124: 121. Based on the preset window information, the first branch sequence data is segmented to obtain multiple segmentation results.

[0051] Preset window information refers to the size of the sliding window. The preset window information is used to determine how much data to split for each attention calculation.

[0052] The segmentation result refers to the data of the first branch sequence. According to the preset window information The resulting data blocks after segmentation.

[0053] After segmenting the first branch sequence data, if the actual length of the last segment among multiple segmentation results is less than... This can be achieved by padding the last segment with zeros. For example, let's assume the first branch sequence data... The length is 106, and the preset window information is... The value is 10. This results in a total of 11 segmentation results. The first 10 of these contain... Complete raw data, number 11 contains The last 6 original data points, plus 4 padding 0s.

[0054] 122. Perform self-attention processing on each of the segmentation results to obtain the corresponding self-attention result.

[0055] Self-attention processing refers to independently calculating the value of each segmentation result. Q , K , V The matrix is ​​then weighted by attention.

[0056] Specifically, let's take the i-th segmentation result as an example. For example, the corresponding self-attention result can be calculated using the following formula: (4) in, For querying the matrix, These are the parameters determined during model training. The key matrix, These are the parameters determined during model training. For value matrices, These are the parameters determined during model training. , , . The feature dimension of the key matrix can take the following values: D . M The value is a pre-defined value, and its corresponding value varies in different application scenarios. In autoregressive scenarios, M This serves as a causal mask within the sliding window; in a bidirectional modeling scenario, M It is a zero matrix.

[0057] 123. The multiple self-attention results are concatenated according to the order of the multiple segmentation results in the first branch sequence data to obtain the concatenation result.

[0058] By reassembling the multiple self-attention results calculated in step 122 in their original order (i.e., the order of the multiple segmentation results in the first branch sequence data), the spliced ​​result can be obtained.

[0059] In step 123, through splicing, the data dimensions can be restored to match those of the first branch sequence data. The results are consistent, and the spliced ​​result has already incorporated local contextual information.

[0060] 124. The splicing result is processed by an activation function to obtain the local attention result.

[0061] Optionally, the activation function can be the SiLU activation function.

[0062] In step 124, by introducing a nonlinear factor (i.e., activation function processing), the feature representation ability can be enhanced, resulting in a local attention result. The local attention result can be used... To express.

[0063] In the above implementation, a long sequence can be divided into multiple fragments by a sliding window, and fine attention calculations can be performed in each fragment, thereby efficiently extracting local features and making up for the shortcomings of the existing Mamba model in capturing local details.

[0064] 130. Based on the state space backbone subsequence and the state space gating subsequence, determine the state space adjustment output.

[0065] Optionally, in one embodiment, step 130 may specifically include steps 131 to 132 as follows: 131. Generate a state matrix based on the main subsequence of the state space.

[0066] Optionally, in one embodiment, step 130 may specifically include steps 1311 to 1313: 1311. Perform frequency domain decomposition on the state space backbone subsequence to obtain frequency domain features.

[0067] State space main subsequence The raw data stream that enters the SSM branch.

[0068] Frequency domain decomposition refers to... Shift from the time / space dimension to the frequency dimension.

[0069] Optionally, in one implementation, the frequency domain features include Fourier transform results and wavelet transform results. Accordingly, step 1311 may specifically include the following steps 13111 to 13112: 13111. Perform a discrete Fourier transform on the main subsequence of the state space to obtain the Fourier transform result.

[0070] The Discrete Fourier Transform (DFT) excels at capturing global, periodic frequency information. The DFT is a mathematical transformation that converts a time-domain sequence signal into a frequency-domain representation, decomposing the sequence into a superposition of sinusoidal components of different frequencies.

[0071] Optionally, for the main subsequence of the state space Performing a discrete Fourier transform yields the Fourier transform result. .

[0072] 13112. Perform discrete wavelet transform on the state space backbone subsequence to obtain the wavelet transform result.

[0073] The Discrete Wavelet Transform (DWT) excels at capturing local, abrupt frequency information. DWT is a mathematical transformation that decomposes a time-domain sequence into frequency-domain components of different scales using wavelet basis functions. Compared to the Fourier Transform, it can simultaneously preserve the local features of both the time and frequency domains of the sequence.

[0074] Optionally, for the main subsequence of the state space By performing a discrete wavelet transform, the wavelet transform result can be obtained. .

[0075] In the above implementation, by combining the Discrete Fourier Transform and the Discrete Wavelet Transform, both global periodic information and local abrupt change information can be obtained.

[0076] 1312. Calculate the energy distribution entropy value of the frequency domain feature.

[0077] The energy distribution entropy value is used to measure the complexity of the frequency components of a signal. A low energy distribution entropy value indicates that the signal is relatively simple, mainly concentrated on a few frequencies; a high energy distribution entropy value indicates that the signal is more complex, with more disordered frequency components.

[0078] Continuing with the example above, step 1312 can specifically include steps 13121 to 13122 as follows: 13121. Calculate the energy distribution entropy value of the Fourier transform result.

[0079] Specifically, the Fourier transform result can be first... The energy distribution is converted into an energy distribution probability, and then the energy distribution entropy is calculated using the energy distribution probability. Specifically, this can be achieved using the following formulas (5) and (6): (5) in, Indicates the firsti Energy of each frequency component; Indicate the Fourier transform result Total energy; Indicates the first i The energy probability of each frequency component in the overall signal.

[0080] (6) in, This represents the entropy value of the energy distribution resulting from the Fourier transform.

[0081] 13122. Calculate the energy distribution entropy value of the wavelet transform result.

[0082] Specifically, the wavelet transform result can be first... The energy distribution is converted into an energy distribution probability, and then the energy distribution entropy is calculated using the energy distribution probability. Specifically, this can be achieved using the following formulas (7) and (8): (7) in, Indicates the first k The energy of each wavelet coefficient; Represents the wavelet transform result Total energy; Indicates the first k The energy probability of each frequency component in the overall signal.

[0083] (8) in, This represents the entropy value of the energy distribution of the wavelet transform result.

[0084] 1313. Based on the energy distribution entropy value, generate the state matrix.

[0085] Optionally, in one embodiment, step 1313 may specifically include the following steps 13131 to 13133: 13131. Calculate the first component based on the energy distribution entropy value of the Fourier transform result, the energy distribution entropy value of the wavelet transform result, and the diagonal state matrix.

[0086] The diagonal state matrix refers to the diagonal state matrix constructed based on the Fourier transform basis, which can be used... This matrix represents the characteristics of a linear time-invariant system. Fourier bases excel at capturing the characteristics of periodic, global, and stationary signals.

[0087] Specifically, the first component can be expressed by the formula (*) as follows: First component = (*) in, It is a Sigmoid gated function; These are the parameters determined during model training. It is a diagonal state matrix constructed based on the Fourier transform basis, and the parameters in the matrix are learned during the model training process; It is the entropy value of the energy distribution as a result of the Fourier transform; It is the energy distribution entropy value of the wavelet transform result.

[0088] 13132. Calculate the second component based on the energy distribution entropy value of the Fourier transform result, the energy distribution entropy value of the wavelet transform result, and the discrete wavelet matrix.

[0089] Discrete wavelet matrix refers to the state matrix constructed based on discrete wavelet basis, which can be used as... This matrix represents the characteristics of a simulated multi-resolution analysis system. Discrete wavelet bases excel at capturing local, transient, and non-stationary signal characteristics.

[0090] Specifically, the second component can be expressed by the formula (#) as follows: Second component = (#) in, It is a Sigmoid gated function; These are the parameters determined during model training. The state matrix is ​​constructed based on the discrete wavelet basis, and the parameters in the matrix are learned during the model training process. It is the entropy value of the energy distribution as a result of the Fourier transform; It is the energy distribution entropy value of the wavelet transform result.

[0091] 13133. Calculate the component sum of the first component and the second component, where the component sum is the state matrix. Specifically, the calculation process of the component sum (i.e., the state matrix) can be achieved using the following formula (9): (9) in, The state matrix; It is a Sigmoid gated function; These are the parameters determined during model training. It is a diagonal state matrix constructed based on the Fourier transform basis, and the parameters in the matrix are learned during the model training process; The state matrix is ​​constructed based on the discrete wavelet basis, and the parameters in the matrix are learned during the model training process. It is the entropy value of the energy distribution as a result of the Fourier transform; It is the energy distribution entropy value of the wavelet transform result.

[0092] In the above implementation method, the solved state matrix It is an adaptive state matrix that dynamically adjusts according to changes in frequency domain characteristics. Specifically, the state matrix... The generation process is determined by the frequency domain complexity of the data, which can be specifically represented by the energy distribution entropy value. A higher energy distribution entropy value indicates greater complexity in the data to be processed, allowing for the generation of a more sensitive model capable of capturing rapidly changing state matrices. When the energy distribution entropy value is low, it indicates that the data to be processed is relatively stable, which can generate a state matrix that makes the model smoother and can filter out noise. .

[0093] 132. Based on the state matrix, the state space backbone subsequence, and the state space gating subsequence, determine the state space adjustment output.

[0094] State-space gated subsequence It is a control signal sequence. This state-space gating subsequence is used to determine the state-space backbone subsequence. Which parts can pass?

[0095] Optionally, in one embodiment, step 132 may specifically include the following steps 1321 to 1322: 1321. Based on the state matrix and the state space backbone subsequence, determine the state space backbone output.

[0096] Optionally, in one embodiment, step 1321 may specifically include the following steps 132111 to 13214: 13211. Extract the input features at time t from the main subsequence of the state space, where t is an integer greater than 1.

[0097] The input features at time t can be represented as Specifically, this can be derived from the main subsequence of the state space. Extracting the data at time t, the extracted data is the input feature at time t. .

[0098] 13212. Based on the hidden state at time t-1, the input features at time t, and the state matrix, calculate the hidden state at time t.

[0099] The hidden state at time t can be represented as , NLet be the state dimension. Specifically, the hidden state at time t can be calculated using the following formula (10): (10) in, Let be the hidden state at time t; The state matrix; This represents the hidden state at time t-1; The input features at time t; These are the parameters learned during model training.

[0100] 13213. Based on the hidden state at time t and the input features at time t, calculate the output features at time t.

[0101] Continuing with the example above, specifically, the output characteristics at time t can be calculated using the following formula (11): (11) in, The output feature at time t; Let be the hidden state at time t; The input features at time t; , D These are the parameters learned during model training.

[0102] 13214. Combine the output features at multiple time points to obtain the state space backbone output.

[0103] Steps 13211 to 13213 above can be executed multiple times to obtain output features at multiple time points: , , , … The output features at multiple time points are compiled into a list, which constitutes the state-space backbone output. The state-space backbone output can be used... express.

[0104] 13212. Based on the state space gating subsequence, the state space backbone output is adjusted to obtain the state space adjusted output.

[0105] Optionally, in one embodiment, step 13212 may specifically include the following steps B1 to B2: B1. Apply activation function processing to the state space gated subsequence to obtain the activation processing result.

[0106] Step B1 involves gating the state space subsequence. The activation function is applied to compress the value of the gating signal (i.e. the activation result) to the range (0,1), thus making it a standard gating switch.

[0107] The activation function can be the SiLU activation function or other activation functions, such as the Sigmoid activation function. It should be understood that the specific type of activation function should not be construed as a limitation of this application.

[0108] B2. Multiply the activation processing result element-wise with the state space backbone output to obtain the state space adjustment output.

[0109] Element-wise multiplication refers to the direct multiplication of corresponding values ​​between two vectors or matrices of the same dimension. This is a mathematical technique for implementing gating mechanisms. Here, 0 represents suppression, and 1 represents enhancement.

[0110] The activation process result is compared with the state space backbone output obtained from steps 13211 to 13214 above. By performing element-wise multiplication, we can obtain the state-space adjusted output. .

[0111] In the above implementation method, a dynamically generated state matrix can be used first. For the main subsequence of the state space Perform scanning and memory updates to generate preliminary long-sequence features (i.e., the state-space backbone output). Then, parallel gating branches (i.e., state-space gating subsequences) are used. The aforementioned long sequence features are filtered and weighted to obtain high-quality SSM features (i.e., state-space regulation output). ).

[0112] 140. Based on the third branch sequence data, generate fusion weight information; and based on the fusion weight information, the local attention result, and the state space adjustment output, determine the fusion output result.

[0113] The third branch sequence data is used to generate control signals for the fusion weights, and can be used... This indicates that the third branch sequence data contains strategy information about how features are combined.

[0114] The fusion weight information is a weight coefficient with a value between 0 and 1. The fusion weight information determines the proportion of the local attention result and the state-space adjustment output in the final result.

[0115] Optionally, in one embodiment, step 140 may specifically include the following steps 141 to 142: 141. Normalize the third branch sequence data to obtain the fusion weight information.

[0116] Specifically, the fusion weight information can be calculated using the following formula (12): (12) in, g To integrate weighted information; It is a Sigmoid gated function; This is the third branch sequence data.

[0117] 142. Based on the fusion weight information, perform a weighted operation on the local attention result and the state space adjustment output to obtain the fusion output result.

[0118] Continuing with the example above, specifically, the fusion output result can be calculated using the following formula (13): (13) in, To merge the output results; g To integrate weighted information; This is a result of local attention; Adjust the output for state space.

[0119] Alternatively, in one implementation, weight information is fused. g The values ​​can be adaptively generated based on the sequence data to be processed. When the sequence data to be processed requires stronger local fine-grained feature capture, the model can automatically enhance the attention branch (corresponding to the local attention result). The weights of the SSM branch (corresponding to the state-space adjustment output) are adjusted; when the sequence data to be processed requires stronger long-sequence dependency modeling, the model can automatically increase the weights of the SSM branch (corresponding to the state-space adjustment output). The weight of ).

[0120] Specifically, if the sequence data to be processed is a complex image or text containing a large number of rare characters or complex sentence structures, then the third branch, after training, can recognize such features with high local complexity and can output a third branch sequence data with a larger value. ; as shown by equation (12) After function processing, the weight information is fused. g The value of approaches 1. For example, let's assume... g =0.9, then substituting into equation (13), we get:

[0121] In this application scenario, the attention branch (corresponding to the local attention result) The weight of the branch (corresponding to the state-space adjustment output) is automatically amplified, accounting for 90%; while the SSM branch (corresponding to the state-space adjustment output) is... The weights of ) are reduced. The model mainly relies on the attention mechanism to handle details.

[0122] If the sequence data to be processed is derived from a video background with strong periodicity or from a long text with logical coherence and coherence, then the third branch, after training, can recognize this global coherence feature and output a third branch sequence data with a smaller numerical value. ; as shown by equation (12) After function processing, the weight information is fused. g The value of approaches 0. For example, let's assume... g =0.1, then substituting into equation (13), we get:

[0123] In this application scenario, the SSM branch (corresponding to state-space adjustment output) The weight of the attention branch (corresponding to the local attention result) is automatically amplified, accounting for 90%; while the weight of the attention branch (corresponding to the local attention result) is amplified. The weights of ) are reduced. The model mainly relies on SSM to capture long-range patterns.

[0124] The aforementioned gated fusion branch (i.e. the branch corresponding to the third subspace) can dynamically balance the output weights of the local attention branch (i.e. the branch corresponding to the first subspace) and the SSM branch (i.e. the branch corresponding to the second subspace) according to the feature distribution of the sequence data to be processed, so as to achieve adaptive adaptation to different modeling requirements and at the same time ensure the parallel computing efficiency of the architecture.

[0125] 150. The fusion output result is fused with the sequence data to be processed to obtain the output features.

[0126] Optionally, in one embodiment, step 150 may specifically include the following steps 151 to 152: 151. Perform linear projection on the fusion output to obtain a linear projection result, wherein the dimension of the linear projection result is the same as the dimension of the sequence data to be processed.

[0127] In step 151, the fused output can be linearly projected to map the fused feature dimensions back to the dimensions of the original input (i.e., the sequence data to be processed). The dimensions of the linear projection result are adjusted to match the dimensions of the sequence data to be processed.

[0128] 152. The linear projection result is residually concatenated with the sequence data to be processed to obtain the output feature.

[0129] Residual connection refers to directly adding the processed features (i.e., the linear projection result) to the original input (i.e., the sequence data to be processed).

[0130] In the above implementation, the output features are obtained by residually concatenating the linear projection result with the sequence data to be processed. This ensures that even if the features extracted from the fusion output result corresponding to the linear projection result are not perfect, the original data (sequence data to be processed) can still be preserved, thus effectively avoiding information loss.

[0131] For details of the schematic flowchart of the data processing procedure shown in steps 110 to 150 above, please refer to [link / reference needed]. Figure 1c As shown. Please see below. Figure 1c The sequence data to be processed is the original input. After layer normalization, the sequence data can be standardized. After linear projection, the normalized data can be mapped to the internal dimensions of the model and split into three parallel branches: local attention branch, adaptive frequency SSM branch, and fusion gating branch.

[0132] The local attention branch is used to capture local details and short-range dependencies. Specifically, the sliding window partitioning divides the long sequence (i.e., the first branch sequence data) into multiple short windows. For each short window, a self-attention mechanism can be used to extract corresponding local features, as described in steps 121 to 122 above. Then, the multiple local features are concatenated and processed using an activation function (not shown in the figure) to obtain the local attention result. .

[0133] The adaptive frequency SSM branch is used to capture global trends and long-range dependencies, and performs adaptive filtering in the frequency domain. Discrete Fourier Transform (DFT) and Discrete Wavelet Transform (DWT) are performed on the state-space backbone subsequences to transform the signal into the frequency domain. The entropy of the frequency domain features is then calculated, and the state matrix is ​​generated through a Sigmoid Gate. The above process corresponds to steps 1311 to 1313.

[0134] Subsequently, selective scanning is performed, and using equations (10) and (11), the main subsequences of the state space are recursively processed to model long-range dependencies and obtain the main output of the state space. Then, based on state-space gated subsequences... Output to the state space backbone Adjustments are made to obtain the state-space regulated output. The above process corresponds to steps 1321 to 1322.

[0135] The fusion gating branch is used to dynamically determine the fusion ratio between "local" and "global" data. The third branch contains sequence data. Equation (12) is used to generate fusion weight information. g .

[0136] Based on fusion weight information g Dynamic blending of local details With global trends The fused output result is obtained. .

[0137] Fusion output results After linear projection (not shown in the figure), residual concatenation is performed with the sequence data to be processed to obtain the output features.

[0138] The above implementation uses three parallel branches to process local details, global trends, and fusion decisions, respectively. Finally, through weighted summation and residual concatenation, it outputs a high-quality representation that retains the original information while incorporating multi-scale features. Figure 1c The data processing method shown removes the deep convolutional module that limits scaling performance from the standard Mamba block, replacing it with a parallel hybrid architecture of sliding window local attention and Mamba SSM. This addresses the representational bottleneck of fixed convolutional kernels while simultaneously improving long sequence modeling efficiency and local feature capture accuracy. Furthermore, for the adaptive frequency SSM branch, an adaptive frequency domain transform basis selection module is designed. This module dynamically adapts the discrete Fourier transform basis and multiple types of discrete wavelet bases based on the frequency domain characteristics of the sequence data to be processed, optimizing the SSM's state matrix (i.e.,...). This enhances the frequency domain representation capability of the model, improving its adaptability and robustness to different types of sequences.

[0139] exist Figure 1c The data processing method presented employs a parallel hybrid architecture of local attention and SSM (Simplified Character Set Model). This retains the advantages of SSM's linear complexity in long sequence modeling while compensating for SSM's insufficient ability to capture local fine-grained semantic dependencies through the local attention mechanism. Compared to existing technologies that serially combine attention mechanisms with SSM, the parallel design significantly improves model inference efficiency and reduces the computing power cost for industrial deployments.

[0140] Optionally, in one embodiment, steps 110 to 150 described above can be implemented by a trained data processing model. The training method for this data processing model may specifically include the following steps S1 to S7: S1. Acquire training sequence data and project the training sequence data into a first subspace to generate training first branch sequence data; project the training sequence data into a second subspace to generate training second branch sequence data, the training second branch sequence data including training state space backbone subsequence and training state space gating subsequence; project the training sequence data into a third subspace to generate training third branch sequence data.

[0141] S2. Perform local attention processing on the training first branch sequence data to obtain the training local attention result.

[0142] S3. Generate a training state matrix based on the main subsequence of the training state space.

[0143] S4. Based on the training state matrix, the training state space backbone subsequence, and the training state space gating subsequence, determine the training state space adjustment output.

[0144] S5. Based on the training third branch sequence data, generate training fusion weight information; and based on the training fusion weight information, the training local attention result, and the training state space adjustment output, determine the training fusion output result.

[0145] S6. The training fusion output result is fused with the training sequence data to obtain the training output features.

[0146] Steps S1 to S6 above correspond to steps 110 to 150 above, and will not be repeated here.

[0147] S7. Based on the training output features and target output features, backpropagate and update the parameters of the data processing model to be trained until the preset training termination condition is met, and the trained data processing model is obtained.

[0148] Backpropagation and update refers to the process of calculating the gradient from the output layer to the input layer layer by layer based on the calculated loss value using the gradient descent algorithm, and adjusting the internal parameters of the model accordingly.

[0149] The preset training termination condition is a pre-defined training cutoff condition. This condition can be: the calculated loss function value is less than a preset loss threshold. In this case, a preset loss threshold can be set. , It is a positive number and can take values ​​such as 0.01 or 0.001. If the average loss value calculated by the loss function for the current batch or the entire epoch is less than... If this is the case, then it can be determined that the model for processing the data to be trained is accurate enough, that is, it meets the preset training termination condition. The loss function mentioned above can specifically be the cross-entropy loss function.

[0150] The data processing method provided in this application embodiment can acquire the sequence data to be processed and project it into a first subspace, a second subspace, and a third subspace respectively to obtain corresponding first branch sequence data, second branch sequence data, and third branch sequence data. The second branch sequence data includes a state space backbone subsequence and a state space gating subsequence. Local attention processing is performed on the first branch sequence data to obtain a local attention result. A state matrix is ​​generated based on the state space backbone subsequence; then, based on the state matrix, the state space backbone subsequence, and the state space gating subsequence, the output quantity of the second branch sequence data is obtained: the state space adjustment output. Fusion weight information is generated based on the third branch sequence data, and then, based on the fusion weight information, the local attention result, and the state space adjustment output, the fusion output result is determined. The fusion output result is fused with the sequence data to be processed to obtain output features. In this application embodiment, three branch sequences can be processed in parallel: first branch sequence data, second branch sequence data (including the state space backbone subsequence and the state space gating subsequence), and third branch sequence data. The architecture consists of three branches: the first branch is a local attention branch, the second branch is a SSM branch, and the third branch is a gated branch that dynamically fuses the first two. The local attention branch and the SSM branch are connected in parallel, and a gated branch is introduced to dynamically fuse them. The SSM branch, based on its inherent recursive computation characteristics and linear time complexity, effectively captures long-range dependency features in the sequence through iterative evolution of state variables. The local attention branch, also connected in parallel, utilizes the local attention mechanism to specifically compensate for the potential shortcomings of the SSM branch in extracting local details, capturing short-range, high-precision dependencies. This parallel architecture maintains efficient collaboration between the two branches, improving the ability to capture fine-grained semantic dependencies. The gated branch flexibly balances the results of the local attention branch with the output of the SSM branch, enhancing the ability to capture spatially related features.

[0151] The data processing method provided in this application can improve the ability to capture local fine-grained semantic dependencies and spatial correlation features.

[0152] In this embodiment, the method of this application embodiment is described in detail. The data processing method provided in this application embodiment can be executed by an electronic device. In this embodiment, such as... Figure 2 As shown, the specific process of a data processing method is as follows: 201. Obtain the sequence data to be processed.

[0153] 202. Project the sequence data to be processed onto the first subspace to generate the first branch sequence data.

[0154] 203. Project the sequence data to be processed onto the second subspace to generate a state space projection result.

[0155] 204. The state space projection result is split to obtain the state space backbone subsequence and the state space gating subsequence.

[0156] 205. Project the sequence data to be processed onto the third subspace to generate the third branch sequence data.

[0157] 206. Based on the preset window information, the first branch sequence data is segmented to obtain multiple segmentation results.

[0158] 207. Perform self-attention processing on each of the segmentation results to obtain the corresponding self-attention result.

[0159] 208. The multiple self-attention results are concatenated according to the order of the multiple segmentation results in the first branch sequence data to obtain the concatenation result.

[0160] 209. Apply an activation function to the splicing result to obtain the local attention result.

[0161] 210. Perform frequency domain decomposition on the state space backbone subsequence to obtain frequency domain features.

[0162] The frequency domain features include Fourier transform results and wavelet transform results. Accordingly, step 210 may specifically include the following steps: performing a discrete Fourier transform on the state space backbone subsequence to obtain the Fourier transform result; performing a discrete wavelet transform on the state space backbone subsequence to obtain the wavelet transform result.

[0163] 211. Calculate the energy distribution entropy value of the frequency domain feature.

[0164] Step 211 may specifically include the following steps: calculating the energy distribution entropy value of the Fourier transform result; and calculating the energy distribution entropy value of the wavelet transform result.

[0165] 212. Generate the state matrix based on the energy distribution entropy value.

[0166] 213. Based on the state matrix and the state space backbone subsequence, determine the state space backbone output.

[0167] Optionally, in one embodiment, step 213 above may specifically include the following steps: extracting input features at time t from the state space backbone subsequence, where t is an integer greater than 1; calculating the hidden state at time t based on the hidden state at time t-1, the input features at time t, and the state matrix; calculating the output features at time t based on the hidden state at time t and the input features at time t; and combining the output features at multiple times to obtain the state space backbone output.

[0168] 214. Based on the state-space gated subsequence, the output of the state-space backbone is adjusted to obtain the state-space adjusted output.

[0169] Optionally, in one embodiment, step 214 may specifically include the following steps: performing activation function processing on the state space gated subsequence to obtain an activation processing result; multiplying the activation processing result element-wise with the state space backbone output to obtain the state space adjustment output.

[0170] 215. Normalize the third branch sequence data to obtain the fusion weight information.

[0171] 216. Based on the fusion weight information, perform a weighted operation on the local attention result and the state space adjustment output to obtain the fusion output result.

[0172] 217. Perform linear projection on the fusion output to obtain a linear projection result, wherein the dimension of the linear projection result is the same as the dimension of the sequence data to be processed.

[0173] 218. The linear projection result is residually concatenated with the sequence data to be processed to obtain the output feature.

[0174] The specific execution process of steps 201 to 218 has been explained in detail above, and will not be repeated here.

[0175] The data processing method provided in this application embodiment can acquire the sequence data to be processed and project it into a first subspace, a second subspace, and a third subspace respectively to obtain corresponding first branch sequence data, second branch sequence data, and third branch sequence data. The second branch sequence data includes a state space backbone subsequence and a state space gating subsequence. Local attention processing is performed on the first branch sequence data to obtain a local attention result. A state matrix is ​​generated based on the state space backbone subsequence; then, based on the state matrix, the state space backbone subsequence, and the state space gating subsequence, the output quantity of the second branch sequence data is obtained: the state space adjustment output. Fusion weight information is generated based on the third branch sequence data, and then, based on the fusion weight information, the local attention result, and the state space adjustment output, the fusion output result is determined. The fusion output result is fused with the sequence data to be processed to obtain output features. In this application embodiment, three branch sequences can be processed in parallel: first branch sequence data, second branch sequence data (including the state space backbone subsequence and the state space gating subsequence), and third branch sequence data. The architecture consists of three branches: the first branch is a local attention branch, the second branch is a SSM branch, and the third branch is a gated branch that dynamically fuses the first two. The local attention branch and the SSM branch are connected in parallel, and a gated branch is introduced to dynamically fuse them. The SSM branch, based on its inherent recursive computation characteristics and linear time complexity, effectively captures long-range dependency features in the sequence through iterative evolution of state variables. The local attention branch, also connected in parallel, utilizes the local attention mechanism to specifically compensate for the potential shortcomings of the SSM branch in extracting local details, capturing short-range, high-precision dependencies. This parallel architecture maintains efficient collaboration between the two branches, improving the ability to capture fine-grained semantic dependencies. The gated branch flexibly balances the results of the local attention branch with the output of the SSM branch, enhancing the ability to capture spatially related features.

[0176] The data processing method provided in this application can improve the ability to capture local fine-grained semantic dependencies and spatial correlation features.

[0177] To better implement the above methods, embodiments of this application also provide a data processing apparatus. For example... Figure 3 As shown, the device includes: The sequence projection unit 301 is used to acquire the sequence data to be processed, project the sequence data to be processed onto a first subspace to generate first branch sequence data; project the sequence data to be processed onto a second subspace to generate second branch sequence data, the second branch sequence data being sequence data reflecting the characteristics of the state space, the second branch sequence data including the state space backbone subsequence and the state space gated subsequence; and project the sequence data to be processed onto a third subspace to generate third branch sequence data. Local attention unit 302 is used to perform local attention processing on the first branch sequence data to obtain local attention results; The state space adjustment unit 303 is used to determine the state space adjustment output based on the state space backbone subsequence and the state space gating subsequence; The fusion weighting unit 304 is used to generate fusion weight information based on the third branch sequence data; and to determine the fusion output result based on the fusion weight information, the local attention result, and the state space adjustment output. The data fusion unit 305 is used to fuse the fusion output result with the sequence data to be processed to obtain output features.

[0178] In one embodiment, the state space adjustment unit 303 includes: A matrix generation subunit is used to generate a state matrix based on the main subsequence of the state space; The output determination subunit is used to determine the state space adjustment output based on the state matrix, the state space backbone subsequence, and the state space gating subsequence.

[0179] In one implementation, the matrix generating subunit includes: The frequency domain decomposition subunit is used to perform frequency domain decomposition on the main subsequence of the state space to obtain frequency domain features. The entropy calculation sub-unit is used to calculate the energy distribution entropy value of the frequency domain feature; The matrix generation sub-unit is used to generate the state matrix based on the energy distribution entropy value.

[0180] In one implementation, the frequency domain features include Fourier transform results and wavelet transform results; correspondingly, the frequency domain decomposition sub-units include: The Fourier three-subunit is used to perform discrete Fourier transform on the main subsequence of the state space to obtain the Fourier transform result; Three wavelet sub-units are used to perform discrete wavelet transform on the main sub-sequence of the state space to obtain the wavelet transform result; The entropy calculation sub-unit includes: The Fourier entropy value sub-unit is used to calculate the energy distribution entropy value of the Fourier transform result; The wavelet entropy value three sub-units are used to calculate the energy distribution entropy value of the wavelet transform result.

[0181] In one implementation, the matrix generates secondary sub-units, including: The first component has three sub-units, which are used to calculate the first component based on the energy distribution entropy value of the Fourier transform result, the energy distribution entropy value of the wavelet transform result, and the diagonal state matrix; The second component has three sub-units, which are used to calculate the second component based on the energy distribution entropy value of the Fourier transform result, the energy distribution entropy value of the wavelet transform result, and the discrete wavelet matrix; The component summation subunit is used to calculate the component summation of the first component and the second component, wherein the component summation is the state matrix.

[0182] In one implementation, the output determination subunit includes: The main output sub-unit is used to determine the state space main output based on the state matrix and the state space main sub-sequence; The adjustment output sub-unit is used to adjust the state space backbone output based on the state space gating sub-sequence to obtain the state space adjusted output.

[0183] In one implementation, the main output sub-unit includes: The input extraction subunit is used to extract the input features at time t from the main subsequence of the state space, where t is an integer greater than 1; The hidden state three sub-unit is used to calculate the hidden state at time t based on the hidden state at time t-1, the input features at time t, and the state matrix; The output feature three sub-units are used to calculate the output feature at time t based on the hidden state at time t and the input feature at time t; The feature combination three sub-units are used to combine the output features at multiple time points to obtain the state space backbone output.

[0184] In one implementation, adjusting the output secondary subunit includes: The activation processing subunit is used to perform activation function processing on the state space gated subsequence to obtain the activation processing result. The result multiplication sub-unit is used to perform element-wise multiplication of the activation processing result with the state space backbone output to obtain the state space adjustment output.

[0185] In one implementation, the fusion weighting unit 304 is specifically used to: normalize the third branch sequence data to obtain the fusion weight information; And used for: performing a weighted operation on the local attention result and the state space adjustment output based on the fusion weight information to obtain the fusion output result.

[0186] In one embodiment, the data fusion unit 305 includes: A linear projection subunit is used to perform linear projection on the fusion output result to obtain a linear projection result, wherein the dimension of the linear projection result is the same as the dimension of the sequence data to be processed. The residual connection subunit is used to perform a residual connection between the linear projection result and the sequence data to be processed to obtain the output feature.

[0187] In one embodiment, the sequence projection unit 301 includes: The second subspace projection subunit is used to project the sequence data to be processed onto the second subspace to generate a state space projection result. The projection splitting subunit is used to split the state space projection result to obtain the state space backbone subsequence and the state space gating subsequence.

[0188] In one embodiment, the local attention unit 302 includes: The molecular segmentation unit is used to segment the first branch sequence data according to preset window information to obtain multiple segmentation results; A self-attention processing subunit is used to perform self-attention processing on each of the segmentation results to obtain the corresponding self-attention result; The splicing subunit is used to splice multiple self-attention results according to the order of the multiple segmentation results in the first branch sequence data to obtain a splicing result; The local attention subunit is used to process the splicing result using an activation function to obtain the local attention result.

[0189] In one embodiment, the method is implemented by a trained data processing model; correspondingly, the apparatus further includes: A training projection unit is used to acquire training sequence data and project the training sequence data into a first subspace to generate training first branch sequence data; project the training sequence data into a second subspace to generate training second branch sequence data, the training second branch sequence data including training state space backbone subsequence and training state space gating subsequence; and project the training sequence data into a third subspace to generate training third branch sequence data. A local attention unit is trained to perform local attention processing on the training first branch sequence data to obtain the training local attention result. The training state matrix unit is used to generate a training state matrix based on the main subsequence of the training state space; An adjustment output determination unit is used to determine the training state space adjustment output based on the training state matrix, the training state space backbone subsequence, and the training state space gating subsequence. The training fusion weight unit is used to generate training fusion weight information based on the training third branch sequence data; and to determine the training fusion output result based on the training fusion weight information, the training local attention result, and the training state space adjustment output. The training fusion output unit is used to fuse the training fusion output result with the training sequence data to obtain training output features; The parameter update unit is used to backpropagate and update the parameters of the data processing model to be trained based on the training output features and the target output features, until the preset training termination condition is met, and the trained data processing model is obtained.

[0190] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0191] The data processing method provided in this application embodiment can acquire the sequence data to be processed and project it into a first subspace, a second subspace, and a third subspace respectively to obtain corresponding first branch sequence data, second branch sequence data, and third branch sequence data. The second branch sequence data includes a state space backbone subsequence and a state space gating subsequence. Local attention processing is performed on the first branch sequence data to obtain a local attention result. A state matrix is ​​generated based on the state space backbone subsequence; then, based on the state matrix, the state space backbone subsequence, and the state space gating subsequence, the output quantity of the second branch sequence data is obtained: the state space adjustment output. Fusion weight information is generated based on the third branch sequence data, and then, based on the fusion weight information, the local attention result, and the state space adjustment output, the fusion output result is determined. The fusion output result is fused with the sequence data to be processed to obtain output features. In this application embodiment, three branch sequences can be processed in parallel: first branch sequence data, second branch sequence data (including the state space backbone subsequence and the state space gating subsequence), and third branch sequence data. The architecture consists of three branches: the first branch is a local attention branch, the second branch is a SSM branch, and the third branch is a gated branch that dynamically fuses the first two. The local attention branch and the SSM branch are connected in parallel, and a gated branch is introduced to dynamically fuse them. The SSM branch, based on its inherent recursive computation characteristics and linear time complexity, effectively captures long-range dependency features in the sequence through iterative evolution of state variables. The local attention branch, also connected in parallel, utilizes the local attention mechanism to specifically compensate for the potential shortcomings of the SSM branch in extracting local details, capturing short-range, high-precision dependencies. This parallel architecture maintains efficient collaboration between the two branches, improving the ability to capture fine-grained semantic dependencies. The gated branch flexibly balances the results of the local attention branch with the output of the SSM branch, enhancing the ability to capture spatially related features.

[0192] The data processing method provided in this application can improve the ability to capture local fine-grained semantic dependencies and spatial correlation features.

[0193] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0194] In some embodiments, the data processing apparatus may also be integrated into multiple electronic devices, such as multiple servers, with the data processing method of this application being implemented by the multiple servers.

[0195] In this embodiment, a server will be used as an example for detailed description. For example, ... Figure 4 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically: The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402. In some embodiments, the processor 401 may include one or more processing cores; in some embodiments, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 401.

[0196] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0197] The electronic device also includes a power supply 403 that supplies power to the various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0198] The electronic device may also include an input module 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0199] The electronic device may also include a communication module 405. In some embodiments, the communication module 405 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 405 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0200] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402, thereby realizing the various functions in the various method embodiments of this application, as follows: The process involves: acquiring the sequence data to be processed and projecting it onto a first subspace to generate first branch sequence data; projecting the sequence data onto a second subspace to generate second branch sequence data, which reflects state space characteristics and includes a state space backbone subsequence and a state space gating subsequence; projecting the sequence data onto a third subspace to generate third branch sequence data; performing local attention processing on the first branch sequence data to obtain a local attention result; determining a state space adjustment output based on the state space backbone subsequence and the state space gating subsequence; generating fusion weight information based on the third branch sequence data; determining a fusion output result based on the fusion weight information, the local attention result, and the state space adjustment output; and fusing the fusion output result with the sequence data to be processed to obtain output features.

[0201] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0202] Therefore, embodiments of this application provide a computer-readable storage medium storing instructions that can be loaded by a processor to perform steps in any of the data processing methods provided in embodiments of this application. For example, the instructions can perform the following steps: The process involves: acquiring the sequence data to be processed and projecting it onto a first subspace to generate first branch sequence data; projecting the sequence data onto a second subspace to generate second branch sequence data, which reflects state space characteristics and includes a state space backbone subsequence and a state space gating subsequence; projecting the sequence data onto a third subspace to generate third branch sequence data; performing local attention processing on the first branch sequence data to obtain a local attention result; determining a state space adjustment output based on the state space backbone subsequence and the state space gating subsequence; generating fusion weight information based on the third branch sequence data; determining a fusion output result based on the fusion weight information, the local attention result, and the state space adjustment output; and fusing the fusion output result with the sequence data to be processed to obtain output features.

[0203] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0204] According to one aspect of this application, a computer program product or computer program is provided, comprising instructions stored in a computer-readable storage medium. A processor of a computer device reads the instructions from the computer-readable storage medium and executes the instructions, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.

[0205] Since the instructions stored in the storage medium can execute the steps of any of the data processing methods provided in the embodiments of this application, the beneficial effects that any of the data processing methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0206] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0207] The foregoing has provided a detailed description of a data processing method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, The method includes: The process involves acquiring the sequence data to be processed and projecting it onto a first subspace to generate first branch sequence data; projecting the sequence data to be processed onto a second subspace to generate second branch sequence data, wherein the second branch sequence data is sequence data reflecting the characteristics of the state space, and the second branch sequence data includes a state space backbone subsequence and a state space gated subsequence; and projecting the sequence data to be processed onto a third subspace to generate third branch sequence data. Local attention processing is applied to the first branch sequence data to obtain the local attention result; Based on the state space backbone subsequence and the state space gating subsequence, the state space adjustment output is determined, including: The state space backbone subsequence is decomposed in the frequency domain to obtain frequency domain features; the energy distribution entropy value of the frequency domain features is calculated; a state matrix is ​​generated based on the energy distribution entropy value; and the state space regulation output is determined based on the state matrix, the state space backbone subsequence, and the state space gating subsequence. Based on the third branch sequence data, fusion weight information is generated; and based on the fusion weight information, the local attention result, and the state space adjustment output, the fusion output result is determined. The fusion output result is fused with the sequence data to be processed to obtain the output features.

2. The method as described in claim 1, characterized in that, The frequency domain features include Fourier transform results and wavelet transform results; The step of performing frequency domain decomposition on the state space backbone subsequence to obtain frequency domain features includes: Perform a discrete Fourier transform on the main subsequence of the state space to obtain the Fourier transform result; The discrete wavelet transform is performed on the main subsequence of the state space to obtain the wavelet transform result; The calculation of the energy distribution entropy value of the frequency domain feature includes: Calculate the energy distribution entropy value of the Fourier transform result; and Calculate the energy distribution entropy value of the wavelet transform result.

3. The method as described in claim 2, characterized in that, The process of generating the state matrix based on the energy distribution entropy value includes: The first component is calculated based on the energy distribution entropy value of the Fourier transform result, the energy distribution entropy value of the wavelet transform result, and the diagonal state matrix; The second component is calculated based on the energy distribution entropy value of the Fourier transform result, the energy distribution entropy value of the wavelet transform result, and the discrete wavelet matrix; Calculate the component sum of the first component and the second component, and the component sum is the state matrix.

4. The method as described in claim 1, characterized in that, The step of determining the state-space adjustment output based on the state matrix, the state-space backbone subsequence, and the state-space gating subsequence includes: Based on the state matrix and the state space backbone subsequence, determine the state space backbone output; Based on the state space gating subsequence, the state space backbone output is adjusted to obtain the state space adjusted output.

5. The method as described in claim 4, characterized in that, The step of determining the state space backbone output based on the state matrix and the state space backbone subsequence includes: The input features at time t are extracted from the main subsequence of the state space, where t is an integer greater than 1; The hidden state at time t is calculated based on the hidden state at time t-1, the input features at time t, and the state matrix. Based on the hidden state at time t and the input features at time t, calculate the output features at time t; The output features at multiple time points are combined to obtain the state space backbone output.

6. The method as described in claim 4, characterized in that, The step of adjusting the state space backbone output based on the state space gating subsequence to obtain the state space adjusted output includes: The state-space gated subsequence is processed by an activation function to obtain the activation result; The activation processing result is multiplied element-wise with the state space backbone output to obtain the state space adjustment output.

7. The method as described in claim 1, characterized in that, The step of generating fusion weight information based on the third branch sequence data includes: The third branch sequence data is normalized to obtain the fusion weight information; The step of determining the fusion output result based on the fusion weight information, the local attention result, and the state space adjustment output includes: Based on the fusion weight information, the local attention result and the state space adjustment output are weighted and calculated to obtain the fusion output result.

8. The method as described in claim 1, characterized in that, The step of fusing the fusion output result with the sequence data to be processed to obtain output features includes: The fusion output is linearly projected to obtain a linear projection result, the dimension of which is the same as the dimension of the sequence data to be processed. The linear projection result is residually concatenated with the sequence data to be processed to obtain the output feature.

9. The method as described in claim 1, characterized in that, The step of projecting the sequence data to be processed onto the second subspace to generate second branch sequence data, the second branch sequence data including state space backbone subsequence and state space gated subsequence, including: The sequence data to be processed is projected onto the second subspace to generate a state space projection result; The state space projection result is split to obtain the state space backbone subsequence and the state space gating subsequence.

10. The method as described in claim 1, characterized in that, The step of performing local attention processing on the first branch sequence data to obtain local attention results includes: Based on the preset window information, the first branch sequence data is segmented to obtain multiple segmentation results; Each of the segmentation results is subjected to self-attention processing to obtain the corresponding self-attention result; The multiple self-attention results are concatenated according to the order of the multiple segmentation results in the first branch sequence data to obtain the concatenation result; The splicing result is processed by an activation function to obtain the local attention result.

11. The method as described in claim 1, characterized in that, The method is implemented by a trained data processing model; Before acquiring the sequence data to be processed and projecting the sequence data to be processed into the first subspace, the second subspace, and the third subspace to generate the first branch sequence data, the second branch sequence data, and the third branch sequence data, respectively, the method further includes: Acquire training sequence data and project the training sequence data into a first subspace to generate training first branch sequence data; project the training sequence data into a second subspace to generate training second branch sequence data, the training second branch sequence data including training state space backbone subsequence and training state space gating subsequence; project the training sequence data into a third subspace to generate training third branch sequence data; Local attention processing is performed on the training first branch sequence data to obtain the training local attention result; Based on the main subsequences of the training state space, a training state matrix is ​​generated; Based on the training state matrix, the training state space backbone subsequence, and the training state space gating subsequence, the training state space adjustment output is determined; Based on the training third branch sequence data, training fusion weight information is generated; and based on the training fusion weight information, the training local attention result, and the training state space adjustment output, the training fusion output result is determined. The training fusion output result is fused with the training sequence data to obtain the training output features; Based on the training output features and target output features, the parameters of the data processing model to be trained are backpropagated and updated until the preset training termination condition is met, thus obtaining the trained data processing model.

12. A data processing apparatus, characterized in that, The device includes: A sequence projection unit is used to acquire the sequence data to be processed, project the sequence data to be processed onto a first subspace to generate first branch sequence data; project the sequence data to be processed onto a second subspace to generate second branch sequence data, the second branch sequence data being sequence data reflecting state space characteristics, the second branch sequence data including state space backbone subsequences and state space gated subsequences; and project the sequence data to be processed onto a third subspace to generate third branch sequence data. A local attention unit is used to perform local attention processing on the first branch sequence data to obtain a local attention result. A state-space adjustment unit is used to determine a state-space adjustment output based on the state-space backbone subsequence and the state-space gated subsequence. Specifically, the state-space adjustment unit is used to perform frequency domain decomposition on the state-space backbone subsequence to obtain frequency domain features; calculate the energy distribution entropy value of the frequency domain features; generate a state matrix based on the energy distribution entropy value; and determine the state-space adjustment output based on the state matrix, the state-space backbone subsequence, and the state-space gated subsequence. The fusion weighting unit is used to generate fusion weight information based on the third branch sequence data; and to determine the fusion output result based on the fusion weight information, the local attention result, and the state space adjustment output. The data fusion unit is used to fuse the fusion output result with the sequence data to be processed to obtain output features.

13. An electronic device, characterized in that, It includes a processor and a memory, the memory storing instructions; the processor loads instructions from the memory to perform the steps of the data processing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions adapted for loading by a processor to perform the steps of the data processing method according to any one of claims 1 to 11.

15. A computer program product, characterized in that, It includes instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 11.

Images