Data processing method, device and system

Abstract

The invention discloses a data processing method. The method comprises the following steps: acquiring first feature representations and second feature representations of a plurality of tokens; the first feature representation and the second feature representation are feature representations which are obtained through different mapping modes for the plurality of tokens and are represented in a time domain; wherein each token in the plurality of tokens is at least one of a text, an image or an audio; converting the first feature representation into a feature representation represented in a frequency domain to obtain a third feature representation; converting the second feature representation into a feature representation represented in a frequency domain to obtain a fourth feature representation; and fusing the third feature representation and the fourth feature representation to obtain a fifth feature representation, and converting the fifth feature representation into a feature representation represented in a time domain to obtain a sixth feature representation of the plurality of tokens. According to the invention, the operation complexity can be reduced.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a data processing method, apparatus and system. Background Technology

[0002] Generative artificial intelligence technology has made groundbreaking progress in recent years. Currently, large models based on the Transformer architecture have become the de facto standard in this field, demonstrating powerful capabilities in interdisciplinary scenarios such as natural language processing, computer vision, and even biomedicine.

[0003] To improve the model's ability to understand and predict text, images, or other sequential information, the industry commonly adopts the Transformer architecture based on attention mechanisms as the core of the basic model. This method captures long-distance dependencies by calculating the relevance scores between all tokens in the input sequence, thereby achieving a deep understanding and generation of sequence data. Simultaneously, to enable the model to perceive the sequential information of the sequence, positional encoding is typically introduced to assign each token its absolute or relative position within the sequence.

[0004] However, when dealing with long sequences, the computational complexity of the attention-based Transformer model is proportional to the square of the input sequence length, causing the training and inference costs to rise sharply as the sequence length increases. Summary of the Invention

[0005] This application provides a data processing method for reducing computational complexity.

[0006] In a first aspect, this application provides a data processing method, the method comprising: obtaining a first feature representation and a second feature representation of a plurality of tokens; the first feature representation and the second feature representation are feature representations in the time domain obtained by mapping the plurality of tokens through different methods; wherein each of the plurality of tokens is at least one of text, image or audio; converting the first feature representation into a feature representation in the frequency domain to obtain a third feature representation; converting the second feature representation into a feature representation in the frequency domain to obtain a fourth feature representation; fusing the third feature representation and the fourth feature representation to obtain a fifth feature representation, and converting the fifth feature representation into a feature representation in the time domain to obtain a sixth feature representation of the plurality of tokens.

[0007] In this embodiment, the time-domain to frequency-domain conversion itself has near-linear computational complexity, and the fusion of data in the frequency domain also has linear computational complexity. Especially when N is extremely large, the computational complexity is linearly related to the token length N (O(NlogN)). By doing so, the computational complexity can be reduced.

[0008] Furthermore, in the time domain, the positional information of tokens is reflected in the changing pattern of feature differences between tokens at different positions according to their order. The time-to-frequency domain transformation converts this "time-domain sequential change pattern" into "frequency" and "phase" features in the frequency domain. High-frequency components correspond to "rapidly changing feature patterns in the time domain" (usually the positional associations of adjacent or near-neighbor tokens), while low-frequency components correspond to "slowly changing feature patterns in the time domain" (usually the global positional associations of distant tokens). The phase further records the temporal starting point of feature changes (ensuring that the positional order of tokens can be restored during the inverse transformation). The frequency domain attributes of frequency combined with phase are essentially an indirect but complete representation of token positional information. Without the need for additional positional encoding, position-related global information can be integrated through frequency domain interactions (such as element-wise multiplication), thereby further reducing the computational complexity of the model.

[0009] Furthermore, since the embodiments of this application do not require a traditional position encoding module, the rigid limitation on the length of the input sequence caused by the size of the position encoding table can be removed.

[0010] In one possible implementation, the first feature representation includes a first sub-representation of each of the plurality of tokens, and the third feature representation includes a plurality of second sub-representations, each of which is obtained by converting elements at the same position in different first sub-representations from the time domain to the frequency domain.

[0011] The first sub-representation is the temporal feature representation of a "single token" among multiple tokens. It is a component of the first feature representation, which serves as the overall temporal feature of multiple tokens. It is composed of the first sub-representations corresponding to each token in sequence (e.g., text tokens in chronological order, image patches in spatial order). For example, the first sub-representation can be obtained by mapping the original features of the token through linear transformation or other methods, preserving the positional association information of the token in the temporal domain.

[0012] The second sub-representation is the frequency domain feature representation of the first sub-representation after time-domain to frequency-domain conversion. It is a component of the third feature representation. The third feature representation serves as the overall frequency domain feature of the first feature representation. Its generation logic can be as follows: with a fixed batch size and feature dimensions, only elements located at the same position in the first sub-representations of different tokens (such as the first feature dimension element and the second feature dimension element of all tokens' first sub-representations) are processed separately using time-domain to frequency-domain conversion algorithms such as Fourier transform. Each processing result is a second sub-representation. All second sub-representations are combined to form the third feature representation.

[0013] In one possible implementation, the second feature representation includes a third sub-representation of each of the plurality of tokens, and the fourth feature representation includes a plurality of fourth sub-representations, each of which is obtained by converting elements at the same position on different third sub-representations from the time domain to the frequency domain.

[0014] The third sub-representation is another type of temporal feature representation for a "single token" among multiple tokens. It is a component of the second feature representation. While both the second and first feature representations are temporal features of the token, their mapping methods differ (the first feature representation often uses linear transformations, while the second feature representation often uses multilayer perceptron (MLP) processing). Therefore, the third sub-representation is a representation of the token's original features through...

[0015] The temporal features of a single token obtained by mapping methods such as MLP are combined with multiple third sub-representations in sequence to form a second feature representation.

[0016] The fourth sub-representation is the frequency domain feature representation of the third sub-representation after time-domain to frequency-domain transformation. It is a component of the fourth feature representation. The fourth feature representation serves as the overall frequency domain feature of the second feature representation, and its generation logic is consistent with that of the second sub-representation: for "elements located at the same position in the third sub-representations of different tokens", each fourth sub-representation is obtained by processing them separately through Fourier transform. All fourth sub-representations are combined to form the fourth feature representation, and the dimensions of the fourth sub-representation match those of the second sub-representation, laying the foundation for subsequent frequency domain fusion (such as element-wise multiplication).

[0017] In one possible implementation, the third feature is represented by a Fourier transform of the first feature representation, and the fourth feature is represented by a Fourier transform of the second feature representation.

[0018] In one possible implementation, the third feature representation and the fourth feature representation have the same size, and the fusion of the third feature representation and the fourth feature representation includes: performing a product operation on the elements located at the same position between the third feature representation and the fourth feature representation.

[0019] In one possible implementation, the first feature is represented as a time-domain representation of the feature representations of the plurality of tokens obtained by linear transformation of the feature representations of the plurality of tokens.

[0020] In one possible implementation, the second feature is represented as a time-domain representation of the feature representations of the plurality of tokens obtained by a multilayer perceptron (MLP) module.

[0021] In one possible implementation, the first feature representation and the second feature representation are time-domain representations obtained by mapping the fifth feature representations of the plurality of tokens through different methods; the plurality of tokens includes a target token; the fifth feature representation includes a fifth sub-representation of the target token, which is obtained by convolution operation on the sub-representations of the tokens preceding the target token among the plurality of tokens and the sub-representation of the target token.

[0022] By fusing information from the target token and its predecessors through convolution operations, the local correlation of features can be enhanced. Furthermore, the feature representation of a token will only access information from its predecessors and will not access information from its successors, thus ensuring its causality.

[0023] Secondly, this application provides a data processing apparatus, the apparatus comprising:

[0024] The acquisition module is used to acquire a first feature representation and a second feature representation of multiple tokens; the first feature representation and the second feature representation are feature representations in the time domain obtained by mapping the multiple tokens through different methods; wherein, each of the multiple tokens is at least one of text, image or audio;

[0025] The conversion module is used to convert the first feature representation into a feature representation in the frequency domain to obtain a third feature representation; convert the second feature representation into a feature representation in the frequency domain to obtain a fourth feature representation; fuse the third feature representation and the fourth feature representation to obtain a fifth feature representation; and convert the fifth feature representation into a feature representation in the time domain to obtain a sixth feature representation of the plurality of tokens.

[0026] In one possible implementation, the first feature representation includes a first sub-representation of each of the plurality of tokens, and the third feature representation includes a plurality of second sub-representations, each of which is obtained by converting elements at the same position in different first sub-representations from the time domain to the frequency domain.

[0027] In one possible implementation, the second feature representation includes a third sub-representation of each of the plurality of tokens, and the fourth feature representation includes a plurality of fourth sub-representations, each of which is obtained by converting elements at the same position on different third sub-representations from the time domain to the frequency domain.

[0028] In one possible implementation, the third feature is represented by a Fourier transform of the first feature representation, and the fourth feature is represented by a Fourier transform of the second feature representation.

[0029] In one possible implementation, the third feature representation and the fourth feature representation have the same size, and the conversion module is configured to:

[0030] Perform a product operation on the elements located in the same position between the third feature representation and the fourth feature representation.

[0031] In one possible implementation, the first feature is represented as a time-domain representation of the feature representations of the plurality of tokens obtained by linear transformation of the feature representations of the plurality of tokens.

[0032] In one possible implementation, the second feature is represented as a time-domain representation of the feature representations of the plurality of tokens obtained by a multilayer perceptron (MLP) module.

[0033] In one possible implementation, the first feature representation and the second feature representation are time-domain representations of the fifth feature representations of the plurality of tokens obtained by different mapping methods.

[0034] The plurality of tokens includes a target token; the fifth feature representation includes a fifth sub-representation of the target token, which is obtained by convolution operation on the sub-representations of the tokens preceding the target token and the sub-representation of the target token.

[0035] Thirdly, this application provides a computing device cluster. The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is used to execute instructions stored in the at least one memory to cause the computing device or the computing device cluster to perform a method as described in the first aspect or any implementation thereof.

[0036] Fourthly, this application provides a computer-readable storage medium storing instructions that instruct a computing device or a cluster of computing devices to execute the method of the database system described in the first aspect or any implementation thereof.

[0037] Fifthly, this application provides a computer program product containing instructions that, when run on a computing device or a cluster of computing devices, causes the computing device or cluster of computing devices to execute the database system execution method described in the first aspect or any implementation thereof.

[0038] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0039] Figure 1 A flowchart illustrating a data processing method provided in this application;

[0040] Figure 2a and Figure 2b A flowchart illustrating a data processing method provided in this application;

[0041] Figure 3 A schematic diagram of the architecture of a data processing device provided in this application;

[0042] Figure 4 A schematic diagram of the architecture of a computing device provided in this application;

[0043] Figure 5 A schematic diagram of the architecture of a computing device cluster provided in this application;

[0044] Figure 6 This is a schematic diagram of the architecture of a computing device cluster provided in this application. Detailed Implementation

[0045] First, some expressions that may appear in this application will be explained.

[0046] "First" and "second" are used to distinguish different objects or to differentiate different treatments of the same object, rather than to describe a specific order of objects.

[0047] "At least one" means one or more, while "more" means two or more.

[0048] "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone. A and B can be singular or plural.

[0049] The character " / " generally indicates that the objects before and after it are in an "or" relationship. For example, A / B can mean A or B.

[0050] Furthermore, the terms "comprising," "including," and "having" used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0051] It should be noted that in this application, the terms "exemplary" or "for example" are used to indicate that something is being described or illustrated. Any implementation or design scheme described as "exemplary" or "for example" (such as the embodiments in this application) should not be construed as being more preferred or advantageous than other implementations or design schemes. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0052] In the specification and drawings of this application, the terms "of", "relevant", and "corresponding" may sometimes be used interchangeably. It should be noted that when the distinction is not emphasized, they have the same meaning.

[0053] Below is a brief description of some of the terms used in this application.

[0054] (1) Neural Network

[0055] A neural network can be composed of neural units, which can be operational units that take xs and an intercept of 1 as inputs, and whose output can be:

[0056]

[0057] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0058] (2) Deep Neural Networks

[0059] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0060] (3) Transformer: A deep learning model architecture based on self-attention mechanism, specifically designed for processing sequential data. It abandons the traditional recurrent and convolutional structures and improves training efficiency and model performance by parallel computing of global dependencies, and has become a basic model in various fields.

[0061] (4) Fourier Transform: A mathematical tool that transforms a function or signal from its original domain (such as the time domain) to the frequency domain. It reveals the frequency components of a signal and enables operations such as filtering, denoising, and feature extraction in fields such as digital signal processing and image analysis.

[0062] (5) Attention mechanism: An information processing method in neural networks that mimics human cognition. It dynamically assigns weights to different parts of the input data, enabling the model to focus on processing the information most relevant to the current task, thereby improving the efficiency and accuracy when processing complex inputs.

[0063] (6) Context window: The maximum length of the input sequence that a natural language processing model can process in a single computation. It defines the range of adjacent contextual information that the model can refer to when making predictions or understanding. The model's performance will significantly decrease when the input exceeds this range.

[0064] (7) Computational complexity: A method for measuring the relationship between the computational resources (such as time) required to execute an algorithm and the size of its input. It is often used to analyze and compare the efficiency of different algorithms and is a core indicator for evaluating the scalability of algorithms.

[0065] (8) Positional encoding extrapolation capability: The ability of a model to maintain good performance when dealing with inputs that are longer than the sequences it has seen during training. This capability depends primarily on the design of the positional encoding, whether it can effectively generalize to larger positional indices beyond the training range without causing a sharp drop in model performance.

[0066] (9) Causal Convolution: A special type of one-dimensional convolution operation that ensures that when computing the output at any time step t, it depends only on the input at the current time step t and the inputs before it (t-1, t-2, ...), and cannot "see" any future inputs (t+1, t+2, ...). This strict adherence to time order makes it very suitable for tasks such as autoregressive models and time series forecasting.

[0067] The above are explanations of some of the terms used in this application. The following section introduces the application scenarios of the embodiments of this application:

[0068] The embodiments of this application can serve as the execution actions of a transformer network. The transformer network can be specifically applied in, but is not limited to, the following scenarios:

[0069] 1. Natural Language Processing (NLP) Scenarios

[0070] Natural Language Processing (NLP) is the most common application area for Transformer models, covering the entire process from basic text understanding to complex generation tasks. In text understanding tasks, pre-trained Transformer-based models such as BERT and RoBERTa can capture text contextual relationships through bidirectional attention, accurately completing tasks such as text classification (e.g., sentiment analysis, news category division), named entity recognition (e.g., extracting company names and monetary entities from financial reports), and text similarity calculation (e.g., matching user questions with candidate answers in question-answering systems). In text generation tasks, models such as GPT and T5, employing Decoder or Encoder-Decoder architectures, can achieve coherent and semantically accurate generation, such as machine translation (e.g., Chinese-English bilingual translation, preserving the original semantics and sentence style), dialogue systems (e.g., intelligent customer service generating responses appropriate to the context), text summarization (e.g., compressing long reports into key information summaries), code generation (e.g., automatically generating Python, Java, and other code snippets based on natural language requirements), and even supporting literary creation (e.g., generating novel excerpts and poems). In addition, Transformer can handle low-resource language tasks, achieving efficient understanding and generation of texts in less commonly spoken languages ​​through cross-language pre-training.

[0071] 2. Computer Vision (CV) Scene

[0072] In image understanding tasks, ViT (Vision Transformer) segments images into fixed-size patches, treating each patch as an "image token." By capturing global pixel relationships through the Transformer Encoder, it can efficiently perform image classification (e.g., identifying object categories in images with accuracy surpassing traditional CNNs), object detection (e.g., locating and classifying multiple objects in an image, such as pedestrian and vehicle detection in autonomous driving scenarios), and image segmentation (e.g., segmenting tumor regions in medical images, segmenting defective parts in industrial quality inspection). In image generation tasks, Transformer-based models such as Stable Diffusion and DALL-E can generate high-resolution, detailed images based on natural language descriptions (e.g., generating a corresponding visual image from the input "a seaside castle in the sunlight"), and can also perform image editing (e.g., locally modifying image content, style transfer, converting photos to an oil painting style). Furthermore, Transformers are widely used in video understanding. For example, the Video Swin Transformer captures dynamic relationships between video frames through spatiotemporal attention, performing video classification (e.g., identifying whether a video belongs to sports, animation, or documentary) and action detection (e.g., identifying abnormal pedestrian movements in a video).

[0073] 3. Voice processing scenarios

[0074] In speech recognition tasks, Whisper and Conformer models (which combine the advantages of CNN and Transformer) based on Transformer can convert speech signals into text, supporting multilingual and multi-scene recognition (such as speech transcription in noisy environments and real-time transcript generation of meeting recordings), and are more adaptable to differences in accent and speech rate. In text-to-speech (TTS) tasks, Transformer-TTS, VITS, and other models can generate natural and fluent speech based on text content, simulate the timbre and intonation of different speakers (such as generating speech with specific emotions for audiobook recording and intelligent assistant voice output), and can also achieve zero-sample speech cloning (generating the speaker's speech with only a small amount of audio from the target speaker). In speech understanding tasks, Transformer can combine speech signals and text information to complete speech emotion analysis (such as identifying whether a customer's emotion is satisfied, angry, or neutral from customer service call recordings), speech command recognition (such as recognizing commands such as "turn on the lights" and "adjust the air conditioner temperature" in smart homes), and even support cross-modal speech-text association tasks (such as speech-text alignment for speech caption generation).

[0075] 4. Biomedical and Life Science Scenarios

[0076] For example, in gene sequence analysis tasks, Transformer-based models such as DNABERT and Enformer can treat base pairs (A, T, C, G) in DNA sequences as "biological tokens," capturing long-distance dependencies in gene sequences through attention mechanisms. This enables gene function prediction (e.g., determining whether a gene sequence belongs to a protein-coding region), gene variation analysis (e.g., identifying gene mutation sites that may lead to disease), and inference of species evolutionary relationships (e.g., analyzing the phylogenetic relationships between different species through gene sequence alignment). In drug development tasks, Transformer can model molecular structures (treating atoms and chemical bonds in molecules as tokens) to complete drug molecule design (e.g., generating small molecule compounds with specific pharmacological activities for anticancer and antiviral drug development), predict drug molecule properties (e.g., predicting molecule toxicity and solubility to screen candidate drugs that meet clinical needs), and predict drug-target interactions (e.g., determining whether drug molecules can bind to disease-related proteins to reduce experimental screening costs). Furthermore, Transformer can be applied to cross-modal analysis of medical imaging and gene data, such as combining CT images with gene data to predict disease risk (e.g., the probability of lung cancer incidence).

[0077] 5. Recommendation System Scenarios

[0078] In recommendation systems, Transformer significantly improves recommendation accuracy and user experience by capturing the complex relationships between users and items, and between items themselves, and is especially suitable for processing multimodal, long-sequence interaction data. In e-commerce recommendation scenarios, Transformer can treat a user's browsing, purchasing, and collection history as "behavioral sequence tokens." Combined with product text descriptions (such as product titles and details) and image features (such as product images), it uses attention mechanisms to uncover users' potential interests (e.g., if a user has recently been frequently browsing outdoor products, it infers their camping needs and recommends tents and sleeping bags), enabling personalized product recommendations. In content recommendation scenarios (such as short video, news, and music recommendations), Transformer can process long-sequence user interaction data (such as a user's viewing history, dwell time, and likes on short videos), capturing dynamic changes in interests (e.g., if a user recently switched from liking food videos to fitness videos, it adjusts recommendations accordingly). Simultaneously, it combines semantic features of the content (such as video titles and news articles) to achieve cross-content type association recommendations (e.g., recommending other information related to the topics the user has already viewed). Furthermore, Transformer can solve the "cold start" problem in recommendation systems by using cross-domain attention (e.g., using user behavior on platform A to predict their interests on platform B) to generate reasonable recommendations for new users and items, reducing the initial predicament of having no recommendation data for users or items.

[0079] 6. Multimodal interaction scenarios

[0080] In multimodal understanding tasks, models like CLIP and ALBEF based on Transformer can align text and image features through cross-modal attention, enabling image-text retrieval (e.g., inputting the text "red cat" to retrieve corresponding images from an image library; or inputting an image to retrieve text describing that image), cross-modal content matching (e.g., determining whether an advertisement image and its text are semantically consistent), and multimodal sentiment analysis (e.g., combining video footage, speech tone, and subtitle text to comprehensively determine the emotional tendency in a video). In multimodal generation tasks, Transformer can achieve the effect of "generating multiple modalities from one modality," such as generating images from text (e.g., Stable Diffusion), generating text from images (e.g., image captioning to generate descriptive text for images), generating images from speech (e.g., converting a scene described in speech into a visual image), and even supporting multimodal dialogue (e.g., a user asks a question via voice + image, and the model combines information from both modalities to generate a comprehensive response; for example, a user uploads a picture of a malfunctioning phone and describes "screen flickering" via voice, and the model generates the cause of the malfunction and repair suggestions). These types of scenarios are widely used in fields such as intelligent assistants, content creation, and accessibility assistance (such as describing image content for visually impaired people).

[0081] Next, we will discuss the shortcomings of the Transformer model:

[0082] When processing long sequences, the computational complexity of the attention-based Transformer model is proportional to the square of the input sequence length, leading to a sharp increase in training and inference costs as the sequence length grows. Secondly, its context window is limited. The model's ability to handle sequence length is limited by the positional encoding extrapolation capability, making it difficult to effectively handle extremely long text inputs far exceeding the pre-training length, thus limiting its application in scenarios such as long document understanding and long-range dialogue. The reasons for these shortcomings are as follows: The fundamental reason for the low computational efficiency is that the standard self-attention mechanism requires calculating an attention score for each pair of tokens in the sequence, a pairwise interaction process that leads to a quadratic increase in computational cost. The limited context window stems from existing positional encoding schemes; neither absolute nor relative positional encoding can generalize to sequences much longer than those seen during training. On extremely long sequences, positional information becomes confused or invalidated, leading to a collapse in model performance. To address these issues, [further research is needed]. Figure 1 , Figure 1 A flowchart illustrating a data processing method provided in this application embodiment includes:

[0083] 101. Obtain a first feature representation and a second feature representation of multiple tokens; the first feature representation and the second feature representation are feature representations in the time domain obtained by mapping the multiple tokens through different methods; wherein, each of the multiple tokens is at least one of text, image or audio;

[0084] The embodiments corresponding to steps 101 to 104 can be directly applied to a network layer of a machine learning model (of course, one or more network layers of a machine learning model can be based on the execution ideas of steps 101 to 104).

[0085] Optionally, step 101 can obtain two types of temporal feature representations of multiple tokens, steps 102-103 convert them to the frequency domain, and step 104 fuse them and convert them back to the temporal domain. The features output by this process can match the input and output dimensions of existing network layers and can be directly embedded into the network layer structure of the transformer without adjusting upstream and downstream modules (such as feedforward networks and output prediction layers).

[0086] This embodiment can also be applied to a head in a network layer. For example, each head independently executes the "local coding-frequency domain interaction-temporal domain recovery" process of steps 101-104.

[0087] The embodiments corresponding to steps 101 to 104 can be used to replace the attention layer of the prior art.

[0088] In deep learning (especially sequence processing tasks), a "token" is the smallest semantic or feature unit after the model has broken down the input data. Data can be transformed into an "ordered sequence," and its smallest unit after decomposition can be called a token. For example, when processing text, a token can be a single Chinese character, an English word, or a sub-word (such as "un-" or "happy"); when processing an image, a token can be a fixed-size image patch (such as a 16×16 pixel patch); when processing audio, a token can be an audio segment extracted according to a time window (such as a 20ms audio frame).

[0089] In one possible implementation, the first feature representation and the second feature representation are time-domain representations of the fifth feature representations of the plurality of tokens obtained by different mapping methods.

[0090] The fifth feature representation is fed into two parallel paths to generate the first and second feature representations, respectively. The output first and second feature representations always retain the attributes of the time-domain features. Both retain the spatial features of the token along the sequence length dimension without undergoing frequency domain transformation. This "local coding to dual-path mapping" design not only ensures the local correlation of features but also achieves differentiated information extraction through different mappings, adapting to subsequent frequency domain modulation requirements.

[0091] In one possible implementation, the plurality of tokens includes a target token; the fifth feature representation includes a fifth sub-representation of the target token, which is obtained by convolution operation on the sub-representations of the tokens preceding the target token among the plurality of tokens and the sub-representation of the target token.

[0092] "Multiple tokens" refers to the original sequence units processed by the model (such as text words, image patches, and audio frames), while "target token" refers to any token in the sequence for which local related features are to be generated. The "fifth feature representation" is the overall feature tensor after all tokens are encoded by local context. The part corresponding to a single target token is the "fifth sub-representation of the target token"—the two have a "global-local" relationship, and the fifth feature representation is composed of the fifth sub-representations of all tokens in sequence.

[0093] With the target token as the current position, the convolutional kernel only covers its own sub-representations and those of preceding tokens. For example, when the target token is "d" (sequence position 4), the convolutional kernel only operates on the sub-representations of "b", "c", and "d" (corresponding to positions 2, 3, and 4). The fifth sub-representation of "d" is calculated using "b×k1+c×k2+d×k3" (k1, k2, and k3 are the convolutional kernel parameters), completely ignoring the sub-representations of subsequent tokens such as "e" and "f". In other words, it fuses the local context information of the preceding adjacent positions into the vector representation of each token in the sequence. The output of this step is a sequence tensor with the same dimension as the input, enhanced with local information. Furthermore, the vector representation of each token only accesses information from the tokens preceding it, ensuring causality.

[0094] By fusing information from the target token and its previous tokens through convolution operations, the local correlation of features can be enhanced. When generating the first and second feature representations subsequently, differentiated mapping can be performed based on these locally enhanced features. Ultimately, in frequency domain interaction, global information integration is achieved without losing local details, solving the problem of fragmented local information in existing sparse attention methods.

[0095] For example, the convolution operation can be performed using "depth-separable causal convolution", such as a one-dimensional convolution with a kernel size of 3. To keep the sequence length unchanged, left padding can be performed at the beginning of the input sequence (e.g., padding the sequence "abcdef" with two 0s to get "00abcdef").

[0096] As shown in Table 1, a sequence 'abcdef' is padded with two leading zeros. A convolutional kernel slides from the beginning to the end of the sequence, and the output is the sum of the product of the kernel and the elements of the sequence it covers. For example, a' = 0*k1 + 0*k2 + a*k3, d' = b*k1 + c*k2 + d*k3. The output only touches the inputs preceding it, not the inputs following it. For example, d' will not touch outputs like e or f to ensure causality.

[0097] Table 1

[0098] input 0 0 a b c d e f kernel k1 k2 k3 kernel k1 k2 k3 kernel k1 k2 k3 kernel k1 k2 k3 kernel k1 k2 k3 kernel k1 k2 k3 output a' b' c' d' e' f'

[0099] In one possible implementation, the first feature is represented as a time-domain representation of the feature representations of the plurality of tokens obtained by linear transformation of the feature representations of the plurality of tokens.

[0100] In one possible implementation, the second feature is represented as a time-domain representation of the feature representations of the plurality of tokens obtained by a multilayer perceptron (MLP) module.

[0101] Optionally, the mapping method for generating the first feature representation is a linear transformation. For example, the fifth feature representation can be directly mapped to the "value (V) information flow," i.e., the first feature representation, through a linear transformation layer (W_V). This process involves no nonlinear operations and only adapts the feature dimensions. Optionally, the mapping method for generating the second feature representation is multilayer perceptron (MLP) processing. For example, the fifth feature representation is first transformed by a linear transformation layer (W_G1), then nonlinear expressive power is introduced through the SiLU nonlinear activation function, and finally, the interaction between channels is simulated by a grouped convolutional layer (W_G2), ultimately obtaining the "gate (G) information flow," i.e., the second feature representation. See [link to details] for further information. Figure 2a As shown.

[0102] Optionally, after generation, the feature dimensions of both representations are split into n_heads heads (n_heads is the preset number of heads), so that the feature dimension of each head is d_model / n_heads, where d_model is the original feature dimension. At this time, the data format is transformed into [batch size, number of heads, sequence length, feature dimension of each head].

[0103] 102. Convert the first feature representation into a feature representation in the frequency domain to obtain the third feature representation;

[0104] 103. Convert the second feature representation into a feature representation in the frequency domain to obtain the fourth feature representation;

[0105] The first feature representation is the "value (V) information flow" (mapped from the fifth feature representation through a linear transformation), and the second feature representation is the "gate (G) information flow" (mapped from the fifth feature representation through an MLP module). Both are temporal features—preserving the temporal / spatial order of tokens along the sequence length dimension (such as the order of text tokens or the spatial order of image patches), and the dimensions are completely consistent (both are [batch size, number of heads (if it is a multi-head mechanism), sequence length, feature dimension]).

[0106] In one possible implementation, the third feature is represented by a Fourier transform of the first feature representation, and the fourth feature is represented by a Fourier transform of the second feature representation. For ease of description, the Fourier transform will be used as an example below. Other types of time-domain to frequency-domain conversion can also be used in the embodiments of this application, and the embodiments of this application are not limited thereto.

[0107] The value information stream tensor (i.e., the first feature representation) and the gate information stream tensor (i.e., the second feature representation) are transformed along their sequence length dimension, respectively. This operation converts the information originally represented in the time domain into a complex form represented in the frequency domain.

[0108] In one possible implementation, the first feature representation includes a first sub-representation of each of the plurality of tokens, and the third feature representation includes a plurality of second sub-representations, each of which is obtained by converting elements at the same position in different first sub-representations from the time domain to the frequency domain.

[0109] For example, the input sequence exists as a three-dimensional tensor with dimensions [batch size, sequence length, feature dimension]. Batch size is the number of samples processed at one time (e.g., if two texts are processed simultaneously, batch size = 2); feature dimension is the vector representation dimension of each token (e.g., if each text token is represented by a 512-dimensional vector, feature dimension = 512); sequence length dimension is the number of tokens in a single sample (e.g., if a text contains 10 tokens, sequence length = 10), and this dimension directly reflects the order information of the tokens (the order of text, the spatial order of image patches, the temporal order of audio frames).

[0110] With fixed batch size dimension (dimension 0 = 1) and feature dimension (dimension 2 = 512), operations are performed only on feature data in the sequence length dimension (dimension 1 = 10). That is, for each channel in the feature dimension, the feature value sequence of 10 tokens is processed separately. For example, take the feature value of channel 0 in the feature dimension: the temporal feature value of the 10 tokens in this channel is [2.1, 3.5, 1.8, 4.2, 2.9, 3.7, 1.5, 4.0, 2.3, 3.1] (simulating the temporal sequence, reflecting the feature changes of the tokens in channel 0). Similarly, each of channel 1, channel 2... channel 511 has a set of temporal feature value sequences of 10 tokens.

[0111] Perform an FFT on the sequence length dimension of each feature channel individually: Taking the feature sequence of channel 0 [2.1, 3.5, 1.8, 4.2, 2.9, 3.7, 1.5, 4.0, 2.3, 3.1] as an example, after performing the FFT, the frequency domain features (complex form) of that channel are obtained, for example:

[0112] [29.1+0j,-1.2+2.5j,0.8-1.9j,-0.5+1.3j,0.3-0.7j,2.9-0j,0.3+0.7j,-0.5-1.3j,0.8+1.9j,-1.2-2.5j]; The length of this frequency domain feature is still 10, and each complex component corresponds to a specific frequency component of the time domain sequence under this channel (e.g., the first component corresponds to the DC component, reflecting the overall level of the feature; subsequent components correspond to the AC component, reflecting the rhythm of the feature change with the token order).

[0113] The third feature representation is obtained by combining the frequency domain feature sequences of all 512 feature channels according to their original dimensions. The third feature representation is still [1, 10, 512] (the batch size, sequence length, and feature dimension remain unchanged), but the feature value at each position changes from a real number in the time domain to a complex number in the frequency domain.

[0114] In one possible implementation, FFT can be performed along the sequence length dimension on the first feature representation (V information flow) and the second feature representation (G information flow), respectively. This transforms the sequentially arranged feature sequence of tokens in the time domain into complex feature representations of different frequency components in the frequency domain, namely the third feature representation (frequency domain form of V) and the fourth feature representation (frequency domain form of G). Each complex component of the frequency domain feature corresponds to the amplitude and phase information of a specific frequency in the time domain sequence, which can capture global sequence trends (such as semantic rhythm and spatial texture periodicity of an image).

[0115] Optionally, if a multi-head mechanism is adopted (such as dividing the feature dimension into n_heads independent heads), then the first and second feature representations of each head are independently FFTed—the V / G information stream of each head is converted into the third / fourth feature representation of the corresponding head, ensuring that the subspace features captured by different heads can be independently interacted in the frequency domain, which is consistent with the multi-subspace parallel logic of existing multi-head attention.

[0116] Optionally, to avoid interference from the circular convolution boundary caused by FFT, the transform length of FFT is set to twice the length of the original sequence (achieved by padding with zeros) during the transformation to ensure that the frequency domain features can completely preserve the global information of the time domain sequence. When the inverse transform is performed back to the time domain, it is truncated to the original length.

[0117] The existing attention layer needs to be passed through Q·K T Calculating pairwise associations between tokens has a time complexity of O(N). 2 In this embodiment, since the time-domain to frequency-domain conversion and the fusion of information in the frequency domain are less complex, the overall computational complexity can be reduced, thereby improving efficiency.

[0118] In one possible implementation, the second feature representation includes a third sub-representation of each of the plurality of tokens, and the fourth feature representation includes a plurality of fourth sub-representations, each of which is obtained by converting elements at the same position on different third sub-representations from the time domain to the frequency domain.

[0119] 104. The third feature representation and the fourth feature representation are fused to obtain the fifth feature representation, and the fifth feature representation is converted into a feature representation in the time domain to obtain the sixth feature representation of the multiple tokens.

[0120] In one possible implementation, elements at the same position between the third feature representation and the fourth feature representation can be multiplied.

[0121] The existing attention layer needs to be passed through "Q·K" T "Calculating pairwise associations between tokens has a time complexity of O(N)." 2 In this scheme, the third and fourth feature representations (in frequency domain form) can be used to achieve global information fusion through element-wise multiplication, an operation with a complexity of only O(N). Combined with the inherent O(N·logN) complexity of FFT, the overall processing cost for long sequences is reduced from O(N·logN) to O(N·logN). 2 The time complexity is reduced to O(N·logN), significantly improving efficiency.

[0122] Specifically, refer to Figure 2b This allows for element-wise complex multiplication of the frequency domain representations of the value information stream and the gate information stream. Through this operation, the model can achieve information interaction between any two positions in the sequence with a near-linear computational complexity of O(N·logN), effectively solving the complexity bottleneck of existing self-attention mechanisms (O(N²)).

[0123] In the frequency domain, the Fourier coefficients of the transformed gated representation are multiplied element-wise with the Fourier coefficients of the value representation. This operation is mathematically equivalent to performing a circular convolution in the time domain. Compared to the complex dot product calculations and Softmax normalization in self-attention mechanisms, the computational cost of element-wise multiplication in the frequency domain is extremely small. This operation can be understood as the gated representation acting as a dynamic filter in the frequency domain, weighting or attenuating the frequency components of the value representation, thereby achieving selective aggregation of global information.

[0124] After the frequency domain information interaction is completed, the Inverse Fast Fourier Transform (iFFT) algorithm can be invoked to convert the frequency domain result obtained from the multiplication in the previous step back to the time domain, resulting in a sequence tensor containing global context information. Subsequently, this tensor undergoes necessary dimensionality adjustment and a linear transformation, and is then passed to the next layer of the model as the final output of the entire module.

[0125] The product obtained in the previous step is then subjected to an inverse fast fourier transform (IFFT) to convert it back from the frequency domain to the time domain. The length of the inverse transform should be the same as the length set during the forward transform. After transforming back to the time domain, due to the padding previously applied, the sequence length needs to be truncated from the transform length back to the original sequence length.

[0126] The n_heads heads processed as described above are concatenated along the feature dimension to restore their original feature dimension d_model. Finally, the concatenated sequence is passed through a final linear transformation layer to obtain the final output of this processing module. This output can be used as input to the next layer or for final task prediction.

[0127] Furthermore, the embodiments of this application abandon the traditional positional encoding module, removing the rigid limitation on the length of the input sequence caused by the size of the positional encoding table. This design allows the model to theoretically handle context windows of infinite length.

[0128] The implementation of the embodiments of this application will be described next from the perspective of the device, referring to... Figure 3 This application provides a data processing apparatus 300, which includes:

[0129] The acquisition module 301 is used to acquire a first feature representation and a second feature representation of multiple tokens; the first feature representation and the second feature representation are feature representations in the time domain obtained by mapping the multiple tokens through different methods; wherein, each of the multiple tokens is at least one of text, image or audio;

[0130] The specific description of the acquisition module 301 can be referred to the description of step 101 in the above embodiment, and the similarities will not be repeated.

[0131] The conversion module 302 is used to convert the first feature representation into a feature representation in the frequency domain to obtain a third feature representation; convert the second feature representation into a feature representation in the frequency domain to obtain a fourth feature representation; fuse the third feature representation and the fourth feature representation to obtain a fifth feature representation; and convert the fifth feature representation into a feature representation in the time domain to obtain a sixth feature representation of the plurality of tokens.

[0132] The specific description of the conversion module 302 can be referred to the description of steps 102 to 104 in the above embodiments, and the similarities will not be repeated.

[0133] In one possible implementation, the first feature representation includes a first sub-representation of each of the plurality of tokens, and the third feature representation includes a plurality of second sub-representations, each of which is obtained by converting elements at the same position in different first sub-representations from the time domain to the frequency domain.

[0134] In one possible implementation, the second feature representation includes a third sub-representation of each of the plurality of tokens, and the fourth feature representation includes a plurality of fourth sub-representations, each of which is obtained by converting elements at the same position on different third sub-representations from the time domain to the frequency domain.

[0135] In one possible implementation, the third feature is represented by a Fourier transform of the first feature representation, and the fourth feature is represented by a Fourier transform of the second feature representation.

[0136] In one possible implementation, the third feature representation and the fourth feature representation have the same size, and the conversion module is configured to:

[0137] Perform a product operation on the elements located in the same position between the third feature representation and the fourth feature representation.

[0138] In one possible implementation, the first feature is represented as a time-domain representation of the feature representations of the plurality of tokens obtained by linear transformation of the feature representations of the plurality of tokens.

[0139] In one possible implementation, the second feature is represented as a time-domain representation of the feature representations of the plurality of tokens obtained by a multilayer perceptron (MLP) module.

[0140] In one possible implementation, the first feature representation and the second feature representation are time-domain representations of the fifth feature representations of the plurality of tokens obtained by different mapping methods.

[0141] The plurality of tokens includes a target token; the fifth feature representation includes a fifth sub-representation of the target token, which is obtained by convolution operation on the sub-representations of the tokens preceding the target token and the sub-representation of the target token.

[0142] This application also provides a computing device 1000. For example... Figure 4 As shown, the computing device 1000 includes a bus 1003, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1003. The computing device 1000 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1000.

[0143] Bus 1003 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus 1003 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1003 may include a path for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0144] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0145] The memory 1006 may include volatile memory, such as random access memory (RAM). The memory 1006 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD). The memory 1006 stores executable program code, which the processor 1004 executes to implement the method performed by the aforementioned data processing device. Specifically, the memory 1006 stores instructions for the method executed by the data processing device.

[0146] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0147] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0148] like Figure 5 As shown, the computing device cluster includes at least one computing device 1000. The memory 1006 of one or more computing devices 1000 in the computing device cluster may store instructions for executing methods performed by a data processing apparatus (e.g., including an acquisition module 301 and a conversion module 302).

[0149] In some possible implementations, one or more computing devices 1000 in the computing device cluster can also be used to execute partial instructions of the method executed by the data processing device. In other words, a combination of one or more computing devices 1000 can jointly execute the instructions of the method executed by the data processing device.

[0150] It should be noted that the memory 1006 in different computing devices 1000 in the computing device cluster can store different instructions for executing some functions of the data processing device.

[0151] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 6 One possible implementation is shown. For example... Figure 6 As shown, the two computing devices 120A and 120B are connected via a network. Specifically, they are connected to the network through the communication interface in each computing device. In this type of possible implementation, the memory 116 in computing device 120A may store instructions for executing the data processing method functions in this embodiment. Simultaneously, the memory 116 in computing device 120B may store instructions for implementing the data processing method functions. Alternatively, the memory 116 in computing device 120A may store instructions for implementing part of the data processing method functions. Simultaneously, the memory 116 in computing device 120B may store instructions for implementing another part of the data processing method functions. For example, the memory 116 in computing device 120A may store instructions for implementing the functions of the acquisition module 301. The memory 116 in computing device 120B may store instructions for implementing the conversion module 302.

[0152] It should be understood that Figure 6 The functions of computing device 120A shown can also be performed by multiple computing devices. Similarly, the functions of computing device 120B can also be performed by multiple computing devices.

[0153] The computing device 120A may further include a processor 114, a communication interface 118, and a bus 112; the computing device 120B may further include a processor 114, a communication interface 118, and a bus 112.

[0154] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 5 and Figure 6 The connection method of the computing device cluster. The difference is that the memory 116 of one or more computing devices in the computing device cluster can store the same instructions for executing data processing methods.

[0155] In some possible implementations, the memory 116 of one or more computing devices in the computing device cluster may also each store a portion of the instructions for executing the data processing method. In other words, a combination of one or more computing devices can jointly execute the instructions for executing the data processing method.

[0156] It should be noted that the memory 116 in different computing devices within the computing device cluster can store different instructions for executing parts of the data processing method. That is, the instructions stored in the memory 116 of different computing devices can implement the functions of one or more modules in the data processing device.

[0157] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the method executed by the aforementioned data processing apparatus.

[0158] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the method executed by the data processing apparatus described above.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain a first feature representation and a second feature representation of multiple tokens; the first feature representation and the second feature representation are feature representations in the time domain obtained by mapping the multiple tokens through different methods; wherein, each of the multiple tokens is at least one of text, image or audio; The first feature representation is converted into a feature representation in the frequency domain to obtain the third feature representation; The second feature representation is converted into a feature representation in the frequency domain to obtain the fourth feature representation; The third feature representation and the fourth feature representation are fused to obtain the fifth feature representation, and the fifth feature representation is converted into a feature representation in the time domain to obtain the sixth feature representation of the multiple tokens.

2. The method according to claim 1, characterized in that, The first feature representation includes a first sub-representation of each of the plurality of tokens, and the third feature representation includes a plurality of second sub-representations, each of which is obtained by converting elements at the same position in different first sub-representations from the time domain to the frequency domain.

3. The method according to claim 1 or 2, characterized in that, The second feature representation includes a third sub-representation of each of the plurality of tokens, and the fourth feature representation includes a plurality of fourth sub-representations, each of which is obtained by converting elements at the same position in different third sub-representations from the time domain to the frequency domain.

4. The method according to any one of claims 1 to 3, characterized in that, The third feature is obtained by Fourier transforming the first feature, and the fourth feature is obtained by Fourier transforming the second feature.

5. The method according to any one of claims 1 to 4, characterized in that, The third feature representation and the fourth feature representation have the same size, and the fusion of the third feature representation and the fourth feature representation includes: Perform a product operation on the elements located in the same position between the third feature representation and the fourth feature representation.

6. The method according to any one of claims 1 to 5, characterized in that, The first feature representation is a time-domain representation obtained by linearly transforming the feature representations of the plurality of tokens.

7. The method according to any one of claims 1 to 6, characterized in that, The second feature representation is the time-domain representation of the feature representation of the multiple tokens obtained by the multilayer perceptron (MLP) module.

8. The method according to any one of claims 1 to 7, characterized in that, The first feature representation and the second feature representation are feature representations in the time domain obtained by mapping the fifth feature representation of the plurality of tokens through different methods; The plurality of tokens includes a target token; the fifth feature representation includes a fifth sub-representation of the target token, which is obtained by convolution operation on the sub-representations of the tokens preceding the target token and the sub-representation of the target token.

9. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire a first feature representation and a second feature representation of multiple tokens; the first feature representation and the second feature representation are feature representations in the time domain obtained by mapping the multiple tokens through different methods; wherein, each of the multiple tokens is at least one of text, image or audio; The conversion module is used to convert the first feature representation into a feature representation in the frequency domain to obtain a third feature representation; convert the second feature representation into a feature representation in the frequency domain to obtain a fourth feature representation; fuse the third feature representation and the fourth feature representation to obtain a fifth feature representation; and convert the fifth feature representation into a feature representation in the time domain to obtain a sixth feature representation of the plurality of tokens.

10. The apparatus according to claim 9, characterized in that, The first feature representation includes a first sub-representation of each of the plurality of tokens, and the third feature representation includes a plurality of second sub-representations, each of which is obtained by converting elements at the same position in different first sub-representations from the time domain to the frequency domain.

11. The apparatus according to claim 9 or 10, characterized in that, The second feature representation includes a third sub-representation of each of the plurality of tokens, and the fourth feature representation includes a plurality of fourth sub-representations, each of which is obtained by converting elements at the same position in different third sub-representations from the time domain to the frequency domain.

12. The apparatus according to any one of claims 9 to 11, characterized in that, The third feature is obtained by Fourier transforming the first feature, and the fourth feature is obtained by Fourier transforming the second feature.

13. The apparatus according to any one of claims 9 to 12, characterized in that, The third feature representation and the fourth feature representation have the same size, and the conversion module is used for: Perform a product operation on the elements located in the same position between the third feature representation and the fourth feature representation.

14. The apparatus according to any one of claims 9 to 13, characterized in that, The first feature representation is a time-domain representation obtained by linearly transforming the feature representations of the plurality of tokens.

15. The apparatus according to any one of claims 9 to 14, characterized in that, The second feature representation is the time-domain representation of the feature representation of the multiple tokens obtained by the multilayer perceptron (MLP) module.

16. The apparatus according to any one of claims 9 to 15, characterized in that, The first feature representation and the second feature representation are feature representations in the time domain obtained by mapping the fifth feature representation of the plurality of tokens through different methods; The plurality of tokens includes a target token; the fifth feature representation includes a fifth sub-representation of the target token, which is obtained by convolution operation on the sub-representations of the tokens preceding the target token and the sub-representation of the target token.

17. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 8.

18. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 8.

19. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 8.

Images