Data processing method and apparatus, and related device
By converting the input data of AI models from real numbers to hypercomplex numbers, and utilizing hypercomplex numbers to retain more dimensional feature information, the problem of low training and inference accuracy of AI models is solved, achieving higher training accuracy and lower resource consumption.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAWEI TECH CO LTD
- Filing Date
- 2025-03-31
- Publication Date
- 2026-06-23
Smart Images
Figure CN122263958A_ABST
Abstract
Description
[0001] This application claims priority to Russian patent application filed on December 20, 2024, with application number RU2024138649 and entitled “One Architecture Solution to real domain based on Hypercomplex Attention”, the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus and related equipment. Background Technology
[0003] With the development of artificial intelligence (AI) technology, AI models are typically designed using the Transformer architecture. The core of the Transformer architecture is the attention mechanism, which evaluates the interactions between different components of the input data and emphasizes the components most relevant to the current deep learning task.
[0004] like Figure 1 As shown, the Transformer architecture can include an encoder and a decoder, and both the encoder and decoder can include N stacked layers, where N is a positive integer. Each stacked layer in the encoder typically includes, in sequence, an attention layer, a residual connection and normalization (add & normalize) layer, a feed-forward layer, and another residual connection and normalization layer, as shown below. Figure 1 As shown. In this diagram, the input to each stack layer is tensor data, where each element is a real number. The result calculated by the stack layer based on the tensor data can be passed to the next stack layer in the encoder for further calculation (as the input to that next stack layer). The input data to the stack layers in the decoder can include not only the output of the previous stack layer but also the final calculation result from the encoder, such as... Figure 1 As shown.
[0005] However, in real-world applications, the training or inference accuracy of AI models employing attention mechanisms often falls short of expectations. For instance, when using AI models to train or infer from audio data, the training or inference accuracy is typically at a low level. Summary of the Invention
[0006] This application provides a data processing method designed to maintain a high level of inference accuracy in quantized AI models. Furthermore, this application also provides a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product.
[0007] Firstly, this application provides a data processing method, which can be executed by a corresponding data processing device. Specifically, the data processing device acquires raw data, which serves as input to an AI (Artificial Intelligence) model, such as audio or video data. The AI model includes an attention layer and a feedforward layer, and may employ a Transform architecture. Then, the data processing device generates input data for the attention layer based on the raw data. The values of the elements in this input data are hypercomplex numbers, including real and imaginary parts. Next, the data processing device performs calculations on the input data in the attention layer to obtain a first calculation result, where the values of the elements in the first calculation result are hypercomplex numbers. Finally, based on the first calculation result, the data processing device executes the feedforward layer in the AI model.
[0008] Since each element in the input data of the attention layer is a hypercomplex number, and the real part of the hypercomplex number indicates the fundamental features of the original data, while the imaginary part indicates features such as signal amplitude, phase, or frequency, hypercomplex numbers retain more dimensional features than real numbers. Therefore, computations based on hypercomplex numbers with more dimensional features in the attention layer can improve the accuracy of the output results, thereby enhancing the overall training or inference accuracy of the AI model. Furthermore, in practical applications, even with a reduction in the number of attention layers, the training or inference accuracy of the AI model can still reach a high level. This allows for an effective reduction in the number of parameters, thus reducing both the model size and resource consumption during training and inference.
[0009] In one possible implementation, the original data may specifically be audio data. In this case, the original data may be data in hypercomplex form, including a real part and an imaginary part.
[0010] In one possible implementation, when the data processing device generates input data for the attention layer based on the original data, it may specifically encode the original data to obtain tensor data corresponding to the original data, wherein the values of the elements in the tensor data are real numbers. Then, the data processing device transforms the tensor data to obtain the input data (hypercomplex numbers) for the attention layer. In this way, by converting the real-number tensor data into hypercomplex number input data, the data processing device can utilize the imaginary part of the hypercomplex number to carry more dimensional information, thereby improving the accuracy of the computational results output by the attention layer, and thus improving the overall training or inference accuracy of the AI model.
[0011] In one possible implementation, the raw data used as input to the AI model includes image data or video data. In this case, for image or video data, by ultimately converting the raw data into hypercomplex number form as input data, the imaginary part of the hypercomplex number can carry more dimensional information, thereby improving the accuracy of the AI model in training or inference based on the image or video data.
[0012] In one possible implementation, the AI model further includes multiple network layers, including a first residual connection layer, a first normalization layer, a linear layer, a modified linear unit layer, a random deactivation layer, a second residual connection layer, and a second normalization layer. When executing the feedforward layer based on the first calculation result, specifically, the multiple network layers can be executed sequentially based on the first calculation result and the input data to obtain a second calculation result. The values of the elements in the second calculation result are hypercomplex numbers, and the feedforward layer is then executed based on this second calculation result. Thus, by continuing with the aforementioned multiple network layers based on the first calculation result in hypercomplex number format and the input data, the accuracy of the generated calculation result can be further improved, thereby further enhancing the accuracy of the AI model's training or inference.
[0013] In one possible implementation, the AI model further includes a transformation layer. Specifically, when the data processing device executes the feedforward layer based on the first calculation result, it may first transform the first calculation result into a third calculation result in the transformation layer, where the elements in the third calculation result are real numbers. Then, the third calculation result is used as the input data for the feedforward layer, and the feedforward layer is executed. Thus, by converting the calculation result in hypercomplex number format into a real number format before executing the feedforward layer, the data processing device can adapt to the feedforward layer's requirement of performing calculations based on real numbers, thereby reducing the need for modifications to the feedforward layer in practical applications.
[0014] In one possible implementation, hypercomplex numbers include complex numbers or dual numbers.
[0015] Secondly, this application provides a data processing apparatus, characterized in that the apparatus comprises: an acquisition module for acquiring raw data, the raw data serving as input to an artificial intelligence (AI) model, the AI model including an attention layer and a feedforward layer; a generation module for generating input data for the attention layer based on the raw data, wherein the values of the elements in the input data are hypercomplex numbers, the hypercomplex numbers including real and imaginary parts; a calculation module for performing calculations on the input data in the attention layer to obtain a first calculation result, wherein the values of the elements in the first calculation result are hypercomplex numbers; and an execution module for executing the feedforward layer based on the first calculation result.
[0016] In one possible implementation, the generation module is configured to: encode the original data to obtain tensor data corresponding to the original data, wherein the values of the elements in the tensor data are real numbers; and transform the tensor data to obtain the input data for the attention layer.
[0017] In one possible implementation, the raw data includes image data or video data.
[0018] In one possible implementation, the AI model further includes multiple network layers, including a first residual connection layer, a first normalization layer, a linear layer, a modified linear unit layer, a random deactivation layer, a second residual connection layer, and a second normalization layer; the execution module is used to: execute the multiple network layers sequentially according to the first calculation result and the input data to obtain a second calculation result, wherein the values of the elements in the second calculation result are hypercomplex numbers; and execute a feedforward layer based on the second calculation result.
[0019] In one possible implementation, the AI model further includes a transformation layer; and an execution module configured to: transform a first calculation result into a third calculation result in the transformation layer, wherein the values of the elements in the third calculation result are real numbers; and use the third calculation result as input data for the feedforward layer to execute the feedforward layer.
[0020] In one possible implementation, hypercomplex numbers include complex numbers or dual numbers.
[0021] The data processing apparatus provided in the second aspect corresponds to the data processing method provided in the first aspect. Therefore, the technical effects of the second aspect and any implementation thereof can be found in the description of the relevant aspects of the technical effects of the first aspect and the corresponding implementation thereof, and will not be repeated here.
[0022] Thirdly, this application provides a cluster including one or more computing devices, each computing device including a processor and a memory; wherein the memory is used to store instructions, and the processor executes the instructions stored in the memory to perform the operation steps of the data processing method described in the first aspect and any implementation thereof.
[0023] When a cluster comprises multiple computing devices, the cluster can be used to execute the operational steps of the data processing method described in the first aspect and any implementation thereof. In this case, the attention layer (and feedforward layer) can be deployed on multiple computing devices in the cluster, so that the attention layer (and feedforward layer) are executed jointly by multiple computing devices.
[0024] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on at least one computing device, cause the at least one computing device to perform the operational steps of the data processing method described in the first aspect or any implementation thereof.
[0025] Fifthly, this application provides a computer program product containing instructions that, when run on at least one computing device, causes the at least one computing device to perform the operational steps of the data processing method described in the first aspect or any implementation thereof.
[0026] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description
[0027] Figure 1 This is a schematic diagram of the Transformer architecture;
[0028] Figure 2 A schematic diagram of the structure of an exemplary data processing system provided in this application;
[0029] Figure 3 A flowchart illustrating a data processing method provided in this application;
[0030] Figure 4 This is a schematic diagram of multiple operators that need to be executed during the execution of the attention layer by an exemplary computing node 101;
[0031] Figure 5 A schematic diagram of multiple operators required to be executed during the execution of the attention layer for another exemplary computing node 101;
[0032] Figure 6 This is a schematic diagram showing the transition layer between the attention layer and the feedforward layer.
[0033] Figure 7 This is a schematic diagram showing a network layer consisting of multiple network layers between an attention layer and a feedforward layer.
[0034] Figure 8 This is a schematic diagram showing another type of attention layer and feedforward layer that includes multiple network layers;
[0035] Figure 9 This is a schematic diagram showing the results of testing the two BERT models separately;
[0036] Figure 10 A schematic diagram of the structure of a data processing device provided in this application;
[0037] Figure 11 This is a schematic diagram of the hardware structure of the computing devices in the cluster provided in this application. Detailed Implementation
[0038] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, various non-limiting embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained based on the embodiments in this application and based on the above content are within the scope of protection of this application.
[0039] See Figure 2 The diagram illustrates the structure of a data processing system. Figure 2 As shown, the data processing system 20 includes at least one computing node and at least one data collector. For ease of understanding, Figure 2 The following description uses a compute node 101 and data collectors 201 to 204 as an example. The compute node and each data collector can be interconnected via a bus, such as a peripheral component interconnect express (PCIe) bus or an InfiniBand bus. Alternatively, the compute node and each data collector can be interconnected via a network, such as a communication network including at least one switch.
[0040] The computing node 101 can be implemented using a processor or a computing device including a processor, such as a server. The processor can be any type of processor or any combination thereof, including a central processing unit (CPU), an accelerator, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system-on-chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, and a data processing unit (DPU). The accelerator can be, for example, a graphics processing unit (GPU), a neural network processing unit (NPU), or a tensor processing unit (TPU). This application does not limit the specific implementation method of the computing node.
[0041] Furthermore, computing node 101 can run an AI model, which can employ a Transformer architecture or other similar architectures. This AI model may include one or more attention layers, which can be network layers employing attention mechanisms, self-attention mechanisms, or other mechanisms derived from attention mechanisms. The AI model may also include feedforward layers. In practical applications, stacked layers can be constructed based on attention layers and feedforward layers, such as... Figure 1 As shown.
[0042] For example, it could be a large language model (LLM), a large language model meta AI (LLaMA) model, a bidirectional encoder representations from transformers (BERT) model, or a generative pre-trained Transformer 3 (GPT-3) model, or other types of models such as GPT-4, etc., without limitation.
[0043] It should be noted that, Figure 2 The example given is a single computing node. When the data processing system 20 includes multiple computing nodes, the AI model can be deployed in a distributed manner across multiple computing nodes.
[0044] A data collector is a device capable of collecting data, which can be implemented through hardware or software. When implemented in hardware, a data collector can be, for example, a sensor, such as an audio sensor for collecting audio data, a video sensor for collecting video data, or an image sensor for collecting image data, or other types of devices. When implemented in software, a data collector can be, for example, a client application capable of collecting text-type data input by the user. The data collected by the data collector can serve as input data for an AI model. This input data can be one or more of the following: audio data, video data, image data, and text data. Figure 2 As shown.
[0045] During the process of running the AI model and providing inference services (such as providing machine translation, text generation, or natural language processing services) on compute node 101, compute node 101 can typically execute the embedding layer in the AI model to convert the input data acquired by the data collector into tensor data. At this point, if the attention layer in the AI model uses real tensors (i.e., each element in the tensor data has a real value) for computation, compute node 101 will typically convert the input data into a real tensor. Alternatively, when the tensor data obtained from the input data has a complex value for each element (this tensor data can be called a complex tensor), compute node 101 will convert the complex tensor into a real tensor. Then, compute node 101 uses the finally converted real tensor as the input data for the attention layer and performs corresponding computations on this real tensor within the attention layer.
[0046] In practical applications, the input data for AI models may be audio, video, or image data. In this case, the tensor data obtained by computing node 101 through the embedding layer contains complex numbers for each element. Typically, the real part of this complex number indicates the basic features (or original features) of the input data, while the imaginary part indicates features such as signal amplitude, phase, or frequency. However, after computing node 101 converts the complex tensor into a real tensor, the signal amplitude, phase, or frequency features corresponding to the input data are lost. This leads to lower accuracy in the results calculated by computing node 101 based on the real tensor, which in turn can result in lower inference accuracy for the AI model.
[0047] Similarly, when computing node 101 trains the AI model using training samples, during the forward computation phase, computing node 101 performs the attention layer based on the real tensor corresponding to the training samples, which can easily lead to low training accuracy of the AI model.
[0048] Based on this, in the data processing system 20 provided in this application, the computing node 101 performs an attention layer based on tensor data with element values of hypercomplex, thereby improving the training accuracy or inference accuracy of the AI model.
[0049] In specific implementation, during AI model training or inference, computation node 101 transforms the input data A of the AI model into input data B for the attention layer. This transformation can be achieved by executing an embedding layer. At this point, the values of the elements in the transformed input data B for the attention layer are hypercomplex numbers, which can be complex numbers or dual numbers, including real and imaginary parts. Then, computation node 101 performs calculations on the input data B in the attention layer to obtain the corresponding results. The values of the elements in this result are still hypercomplex numbers. That is, computation node 101 executes the attention layer based on input data B carrying features such as amplitude, phase, or frequency. Therefore, based on the calculation results obtained from executing the attention layer, computation node 101 continues to execute the feedforward layer and other network layers in the AI model, thereby realizing the training or inference process for the AI model.
[0050] Since each element in the input data of the attention layer is a hypercomplex number, and the real part of the hypercomplex number can indicate the basic features of the input data A of the AI model, while the imaginary part of the hypercomplex number can indicate the amplitude, phase, or frequency of the corresponding input data A, the hypercomplex number can retain more dimensional features of the input data A of the AI model compared to real numbers. This makes it possible to improve the accuracy of the calculation results output by the attention layer by performing calculations based on hypercomplex numbers that retain more dimensional features, thereby improving the overall training accuracy or inference accuracy of the AI model.
[0051] Furthermore, in practical applications, even with a reduction in the number of attention layers in an AI model, the training or inference accuracy of the AI model can still reach a high level. This effectively reduces the number of parameters in the AI model, thereby not only reducing the size of the AI model but also reducing the resource consumption generated during the training or inference process.
[0052] It is worth noting that the above Figure 2 The data processing system 20 shown is merely an illustrative example and is not intended to limit the scope of the system. For instance, other possible data processing systems may include more or fewer computing nodes, or nodes with other functionalities, such as nodes for storing data or nodes for managing multiple computing nodes. Similarly, other possible communication systems may include a client, which could be program code deployed on a user-side terminal device or a web browser, used for human-computer interaction, such as presenting the inference results of an AI model to the user or obtaining training samples provided by the user through the client.
[0053] For ease of understanding, the embodiments for quantizing AI models provided in this application are described below with reference to the accompanying drawings.
[0054] See Figure 3 , Figure 3 This is a flowchart illustrating an exemplary data processing method provided in an embodiment of this application. Figure 3 The data processing method shown can be applied to Figure 2 The data processing system 20 shown can be applied to other possible data processing systems. For ease of understanding and description, it will be used in the following example. Figure 2 The data processing system 20 shown is used as an example for explanation. Furthermore, to distinguish it from the input data of the attention layer, the input data of the AI model will be referred to as raw data below.
[0055] like Figure 3 As shown, the data processing method may specifically include the following steps.
[0056] S301: Compute node 101 acquires raw data, which is used as input to the AI model.
[0057] The original data can be either unimodal or multimodal. When the original data is unimodal, it can be any one of the following types of data: audio, video, image, and text. When the original data is multimodal, it can be any combination of audio, video, image, and text types.
[0058] In practical applications, compute node 101 can utilize a data collector to acquire raw data. For example, in an audio recognition scenario, compute node 101 can use a data collector 201 (such as a microphone) to collect audio data and use this audio data as input data for an AI model, enabling compute node 101 to use the AI model to identify whether the speech belongs to a legitimate user based on the audio data. Similarly, in an image recognition scenario, compute node 101 can use a data collector 202 (such as a camera) to collect image data and use this image data as input data for an AI model, enabling compute node 101 to use the AI model to identify the user based on the image data.
[0059] S302: Computation node 101 generates input data for the attention layer in the AI model based on the original data. The values of the elements in the input data are hypercomplex numbers, and the hypercomplex numbers include real and imaginary parts.
[0060] As a first example, hypercomplex numbers can specifically be complex numbers. Generally, complex numbers can be represented as A + iB, where A and B are both real numbers, and i is the imaginary unit. 2 = -1; and A is the real part, iB is the imaginary part.
[0061] As a second example, hypercomplex numbers can specifically be dual numbers. Typically, a dual number can be represented as x + εy, where x and y are both real numbers, and ε is a null terminator. 2 =0, and ε is not a real number; and x is the real part, and εy is the imaginary part.
[0062] In practical applications, hypercomplex numbers can also be represented in other ways, and there are no restrictions on this.
[0063] In one possible implementation, computing node 101 can encode the original data to convert it into tensor data. In this case, the values of the elements in the tensor data obtained after converting the original data may be hypercomplex numbers or real numbers. For example, when the original data is audio data, the values of each element in the tensor data obtained by computing node 101 can all be hypercomplex numbers. In this case, the imaginary part of the hypercomplex number can be used to indicate the signal amplitude and phase characteristics corresponding to the audio data. When the original data is text data (such as user-input keywords or sentences), the values of each element in the tensor data obtained by computing node 101 can all be real numbers.
[0064] Specifically, when the values of the elements in the tensor data are hypercomplex numbers, the computation node 101 can determine the tensor data as the input data for the attention layer. When the values of the elements in the tensor data are real numbers, the computation node 101 can also convert the values of the elements in the tensor data into hypercomplex numbers and use the new tensor data obtained after the conversion as the input data for the attention layer.
[0065] In practical applications, when computing node 101 caches the input data of the attention layer, it can store it in the form of a real number tensor. At this time, as shown in the following formula (1), the value of each element in the input data can be represented by a two-dimensional array.
[0066]
[0067] Where Z is the hypercomplex number, X is the real part of the hypercomplex number, τY is the imaginary part of the hypercomplex number, and τ = ε or τ = i; H is the field of the hypercomplex number; and R is the field of the real number.
[0068] Furthermore, when the raw data acquired by computing node 101 includes data from different modalities fed back by multiple data collectors, computing node 101 can align the data from multiple modalities during the encoding process. For example, it can map the multimodal data to the same embedding space to obtain tensor data corresponding to the multimodal raw data. In this way, computing node 101 can use the aligned multimodal data to train or infer AI models. For instance, computing node 101 can use AI models to simultaneously authenticate users based on their voice (audio data) and photos (image data).
[0069] S303: Computation node 101 performs calculations on the input data in the attention layer to obtain a first calculation result, wherein the values of the elements in the first calculation result are hypercomplex numbers.
[0070] In this embodiment, computing node 101 can perform the corresponding calculation process in the attention layer based on input data with element values of hypercomplex numbers. Since the real part of the hypercomplex number in the input data can indicate the basic features of the original data, and the imaginary part of the hypercomplex number can carry features such as signal amplitude, phase, or frequency corresponding to the original data, computing node 101 can effectively improve the accuracy of the calculation results in the attention layer by performing calculations based on the multi-dimensional features of the original data.
[0071] As an implementation example, the attention layer may include, for example: Figure 4 The operators shown are linear operators, matrix multiplication operators, scaling operators, and softmax operators. Furthermore, computation node 101 can execute these operators sequentially based on the input data (X+τY). Figure 4 The multiple operators shown complete the calculation process of the attention layer.
[0072] Specifically, such as Figure 4 As shown, when computing node 101 executes linear operators based on input data, it can compare the input data with the query weight W respectively. Q Key weight W K and value weight W V Performing multiplication yields the query tensor Q, key tensor K, and value tensor V corresponding to the input data. Among these, W... Q W K and W V For each parameter tensor, the value of each element can be a hypercomplex number or a real number, without any restriction.
[0073] Then, as Figure 4 As shown, computation node 101 can perform a matrix multiplication operator based on the query tensor Q and the key tensor K. Specifically, it can perform matrix multiplication of the query tensor Q and the key tensor K to obtain a first tensor, in which the values of the elements are still hypercomplex numbers. Typically, the values of the elements in this first tensor can be used to measure the degree of correlation / influence between different parts of the input data.
[0074] Next, as Figure 4 As shown, computation node 101 can execute a scaling operator based on the first tensor to scale the values of the elements in the first tensor to obtain the second tensor. In practical applications, the scaling operator in the attention layer can make the gradient changes of the parameters in the attention layer smoother during the training process of the AI model, thus accelerating the training efficiency of the AI model.
[0075] Then, as Figure 4As shown, computation node 101 can perform the softmax operator on the scaled second tensor to normalize the second tensor and obtain the third tensor. For example, computation node 101 can determine the value of each element in the second tensor after normalization based on the following formula (2) to complete the normalization process for the second tensor.
[0076]
[0077] The value of the k-th element after normalization.
[0078] In practical applications, the third tensor obtained after executing the softmax operator can also be called the importance tensor. The value range of each element in the importance tensor is [0, 1], which can be used to indicate the importance share of the part of the input data corresponding to that element relative to the rest of the input data.
[0079] Finally, as Figure 4 As shown, computation node 101 can perform a matrix multiplication operator based on the third tensor and the value tensor V. Specifically, it can perform matrix multiplication of the third tensor and the value tensor V to obtain a fourth tensor. This fourth tensor is the first calculation result output by the attention layer. The values of the elements in this fourth tensor are still hypercomplex numbers.
[0080] In practical applications, the computation performed by computing node 101 in the attention layer can be represented by the following formula (3).
[0081] Attention(XX) is the first computation result obtained by executing the attention layer; X is the input data of the attention layer; A is the query tensor, based on the input data and the query weight W. Q The result is obtained by multiplication; K is the key tensor, based on the input data and the key weights W. K The result is obtained by multiplication; V is a value tensor, based on the input data and value weights W. V The result is obtained by performing a multiplication operation; d is the tensor dimension. The tensor dimension of Q, K, and V can all be d, and d can be the embedding size of each part in the original data. is the scaling factor; T refers to the transpose.
[0082] For example, when calculating input data with elements taking the value of hypercomplex numbers based on formula (3), formula (3) can be specifically extended.
[0083] W Q The real part value in; To query weight W QThe imaginary part of the value; Key weight W K The real part value in; Key weight W K The imaginary part of the value; Value weight W V The real part value in; Value weight W V The imaginary part of the value.
[0084] As two implementation examples, the attention layer can include, for instance, ... Figure 5 The operators shown are linear operators, matrix multiplication operators, scaling operators, and polarization operators. Furthermore, computation node 101 can execute these operators sequentially based on the input data (X+τY). Figure 5 The multiple operators shown complete the calculation process of the attention layer.
[0085] Specifically, such as Figure 5 As shown, when computing node 101 executes linear operators based on input data, it can compare the input data with the query weight W respectively. Q Key weight W K and value weight W V Perform multiplication to obtain the query tensor Q, key tensor K, and value tensor V corresponding to the input data. Then, compute node 101 can perform matrix multiplication based on the query tensor Q and key tensor K to obtain the first tensor, and then perform scaling on the first tensor obtained by performing matrix multiplication to obtain the second tensor.
[0086] At this point, when the hypercomplex number is specifically a complex number, the value of the denominator calculated based on the above formula (2) may be close to 0. For example, when the values of the two elements in the second tensor are e iπ e 0 When the sum of the values of these two elements is 0, that is, e iπ +e 0 =-1+1=0. Thus, based on... Summing the values of multiple elements in the second tensor may result in a sum close to 0.
[0087] Based on this, computing node 101 can perform polarization operations on the second tensor. For example, it can use the following formula (5) to polarize the value of each element in the second tensor to obtain the third tensor.
[0088] The real part of the values (complex numbers) of the elements in the second tensor; min(Re(X)) is the minimum value among the real parts of the values of all elements included in the second tensor; max(Re(X)) is the maximum value among the real parts of the values of all elements included in the second tensor; Im(X) is the real value of the imaginary part of the values (complex numbers) of the elements in the second tensor; min(Im(X)) is the minimum value among the real values of the imaginary part of the values of all elements included in the second tensor; max(Im(X)) is the maximum value among the real values of the imaginary part of the values of all elements included in the second tensor.
[0089] When the hypercomplex number is specifically a dual number, if the calculation is performed based on the above formula (2), the value of the element in the second tensor can be represented by x + εy. At this time, the Taylor series formula can be used to express e. x+εy Expanded into the following formula (6).
[0090]
[0091] That is, e x+εy The real part of the data is emphasized, while the value of the imaginary part is negligible because the ε value is close to 0. Based on this, when the computing node 101 performs the polarization operator on the second tensor, it can specifically combine the above formula (2) and use the following formula (7) to polarize the value of each element in the second tensor.
[0092] Polarization=SoftMax(Re(X))+τ*SoftMax(Im(X)) Formula (7)
[0093] Where Polarization is the result of polarization processing of the values of the elements in the second tensor; Re(X) is the real part of the values of the elements (pairwise duals) in the second tensor; Im(X) is the real value of the imaginary part of the values of the elements (pairwise duals) in the second tensor; and τ is ε.
[0094] That is, the computation node 101 can calculate the real part and imaginary part (real value part) of the dual number of the elements in the second tensor using the above formula (2) to obtain two tensors. Then, the computation node 101 can associate the elements in the two tensors obtained by calculation to obtain tensor data in dual number format, thereby realizing the polarization processing of the second tensor to obtain the third tensor.
[0095] Finally, as Figure 5 As shown, computation node 101 can perform a matrix multiplication operator based on the third tensor and the value tensor V. Specifically, it can perform matrix multiplication of the third tensor and the value tensor V to obtain a fourth tensor. This fourth tensor is the first calculation result output by the attention layer. The values of the elements in this fourth tensor are still hypercomplex numbers.
[0096] It should be noted that the above implementation of performing computation on hypercomplex number input data in the attention layer is only an example. In actual applications, computation node 101 can also use other methods to perform computation on the input data. For example, the attention layer can include more or fewer operators, or other operators can be used to perform computation on the above-mentioned input data. Figure 4 or Figure 5 The operators shown are replaced, without any restrictions.
[0097] S304: Computation node 101 executes the feedforward layer based on the first calculation result.
[0098] In this embodiment, the AI model may include at least one attention layer and at least one feedforward layer, and the input to each feedforward layer may be obtained based on the output of the attention layer. For example, the AI model may employ... Figure 1 The architecture shown includes multiple attention layers and multiple feedforward layers.
[0099] This embodiment provides the following implementation examples.
[0100] In the first implementation example, the computation node 101 can use the first calculation result as the input data (tensor) of the feedforward layer. At this time, the value of each element in the input data of the feedforward layer is a hypercomplex number, and the computation node 101 can perform corresponding calculations in the feedforward layer based on the input data in hypercomplex number format.
[0101] In the second implementation example, such as Figure 6 As shown, a transformation layer may be included between the attention layer and the feedforward layer. This transformation layer can be used to convert hypercomplex numbers into real numbers. Then, computation node 101 can use the first calculation result as input data to the transformation layer and execute the transformation layer based on the input data to convert the first calculation result into a third calculation result, which is a tensor, and each element in the third calculation result has a real number value. Then, computation node 101 can use the third calculation result as input data to the feedforward layer and execute the feedforward layer based on the input data.
[0102] In the third implementation instance, such as Figure 7As shown, the attention layer and the feedforward layer may include multiple network layers, namely, residual connection (add) layer 1, normalization layer 1, linear layer, rectified linear unit (ReLU) layer, random dropout layer, residual connection layer 2, and normalization layer 2. Then, after the computing node 101 completes the execution of the attention layer and obtains the first calculation result, it can sequentially execute these multiple network layers based on the first calculation result and the input data of the attention layer.
[0103] Specifically, computation node 101 can use the first calculation result and the input data of the attention layer as the input data of the residual connection layer 1, and execute the residual connection layer 1. Specifically, it can add the first calculation result and the input data of the attention layer to obtain intermediate result 1. In this way, the input data of the attention layer is passed to the subsequent network layers of the attention layer through the residual connection, thereby alleviating problems such as gradient vanishing and gradient exploding during AI model training.
[0104] Then, computing node 101 can use intermediate result 1 as input data for normalization layer 1, and perform normalization layer 1 based on intermediate result 1 to obtain intermediate result 2, thereby achieving the standardization processing of intermediate result 1. This enables intermediate result 2 to have a stable mean and variance, which helps to accelerate the overall training process of AI model and improve the stability and generalization performance of AI model.
[0105] Specifically, when executing Residual Connection Layer 1 and Normalization Layer 1, computation node 101 can perform Residual Connection Layer 1 and Normalization Layer 1 based on the hypercomplex number to obtain intermediate result 2. Alternatively, computation node 101 can perform Residual Connection Layer 1 and Normalization Layer 1 based on the real part and imaginary part of the hypercomplex number respectively, and then concatenate the calculated real part result with the calculated imaginary part result to obtain intermediate result 2. Figure 7 As shown.
[0106] Next, as Figure 7 As shown, computing node 101 uses intermediate result 2 as input data for the linear layer and executes the linear layer based on intermediate result 2 to obtain intermediate result 3, so as to learn the linear relationship between different parts of the original data.
[0107] Then, as Figure 7 As shown, computing node 101 can use intermediate result 3 as input data for the modified linear unit layer, and execute the modified linear unit layer based on intermediate result 3 to obtain intermediate result 4, so as to introduce nonlinear factors into the AI model and learn the nonlinear relationship between the input data and the output results of the AI model.
[0108] Next, as Figure 7 As shown, computing node 101 can use intermediate result 4 as data input for the random deactivation layer, and execute the random deactivation layer based on intermediate result 4 to obtain intermediate result 5. This achieves the random discarding (deactivation) of some less important features in the random deactivation layer while retaining more important features, thereby reducing the risk of overfitting in the AI model and improving the generalization ability of the AI model.
[0109] Then, as Figure 7 As shown, computation node 101 can execute residual connection layer 2 and normalization layer 2 based on intermediate result 5 and the output data of residual connection layer 1 to obtain a second computation result, where the value of each element in the second computation result is a hypercomplex number. The implementation method of computation node 101 executing residual connection layer 2 and normalization layer 2 based on intermediate result 5 and the output data of residual connection layer 1 is similar to the implementation method of executing residual connection layer 1 and normalization layer 1 based on the first computation result and the input data of the attention layer, as described above. For details, please refer to the relevant descriptions above; further elaboration is not required here.
[0110] Finally, compute node 101 can use the second calculation result as input data for the feedforward layer. In this case, the value of each element in the input data of the feedforward layer is a hypercomplex number, and compute node 101 can perform corresponding calculations in the feedforward layer based on the hypercomplex number format input data. Alternatively, a layer such as... can be deployed before the feedforward layer. Figure 8 The transformation layer shown allows computation node 101 to use the second calculation result as input data for the transformation layer after obtaining it, and to execute the transformation layer to convert the second calculation result into a third calculation result, where each element in the third calculation result is a real number. Therefore, computation node 101 can use the third calculation result as input data for the feedforward layer and execute the feedforward layer based on this input data, such as... Figure 8 As shown.
[0111] It should be noted that the process of computation node 101 executing the attention layer to the feedforward layer can be considered part of the training phase of the AI model. After completing the feedforward layer, computation node 101 can continue training the remaining layers of the AI model based on the calculated results, such as backpropagation and gradient updates. Alternatively, the process of computation node 101 executing the attention layer to the feedforward layer can be considered part of the AI model's inference process. After completing the feedforward layer, computation node 101 can continue executing the remaining network layers in the AI model until the final output of the AI model is obtained.
[0112] Thus, during the training or inference process of the AI model, the computing node 101 executes the attention layer (and feedforward layer) based on the input data in the hypercomplex number format. It can retain the features of the original data such as signal amplitude, phase or frequency based on the imaginary part in the hypercomplex number. This makes it possible to improve the accuracy of the calculation results output by the attention layer by performing calculations based on the hypercomplex number that retains more features in the attention layer. In this way, the overall training accuracy or inference accuracy of the AI model can be improved.
[0113] Furthermore, in actual testing scenarios, when computation is performed on input data based on hypercomplex number format in the attention layer, the training accuracy or inference accuracy of the AI model can still reach a high level when the number of attention layers in the AI model is reduced.
[0114] Specifically, for the BERT-1 and BERT-2 models that employ an attention mechanism, the values of each element in the input data of the attention layer in the BERT-1 model are real numbers, while the values of each element in the input data of the attention layer in the BERT-2 model are dual numbers. Furthermore, the number of attention layers in the BERT-2 model is less than that in the BERT-1 model; specifically, the number of attention layers in the BERT-2 model is approximately 3 / 5 of the number of attention layers in the BERT-1 model.
[0115] By performing the same training process on the IMDB, QNLI, and SST-2 datasets for both BERT-1 and BERT-2 models—including using the same training samples, the same training algorithm, the same number of iterations, and the same number of epochs (the process of completely traversing the entire dataset)—and then testing the trained BERT-1 and BERT-2 models using test data from these datasets, the following results can be obtained: Figure 9 The test results are shown.
[0116] like Figure 9 As shown, with the number of parameters of the BERT-2 model reduced by 39%, the model accuracy of the BERT-2 model tested on the IMDB and QNLI datasets is higher than that of the BERT-1 model tested on the IMDB and QNLI datasets. The model accuracy of the BERT-2 model tested on the SST-2 dataset is the same as that of the BERT-1 model tested on the SST-2 dataset.
[0117] Therefore, performing computations based on hypercomplex numbers in the attention layer can help reduce the number of parameters in the AI model, thereby effectively reducing the size of the AI model, such as... Figure 9As shown, the size of the BERT-2 model can be reduced by 40%, thereby reducing the resource consumption of AI models during training or inference, including reducing the consumption of computing resources and storage resources.
[0118] It is worth noting that other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.
[0119] The above combination Figures 1 to 9 The data processing method provided in the embodiments of this application will be introduced. Next, the structure of the data processing device and computing device provided in the embodiments of this application will be described with reference to the accompanying drawings.
[0120] See Figure 10 A schematic diagram of the structure of a data processing device is shown. Figure 10 The data processing apparatus 1000 shown includes:
[0121] The acquisition module 1001 is used to acquire raw data, which serves as the input to the artificial intelligence (AI) model. The AI model includes an attention layer and a feedforward layer.
[0122] The generation module 1002 is used to generate input data for the attention layer based on the original data. The values of the elements in the input data are hypercomplex numbers, which include real and imaginary parts.
[0123] The calculation module 1003 is used to perform calculations on the input data in the attention layer to obtain a first calculation result, wherein the values of the elements in the first calculation result are hypercomplex numbers.
[0124] Execution module 1004 is used to execute the feedforward layer based on the first calculation result.
[0125] In one possible implementation, the generation module 1002 is configured to:
[0126] The original data is encoded to obtain the corresponding tensor data, where the values of the elements in the tensor data are real numbers.
[0127] The tensor data is transformed to obtain the input data for the attention layer.
[0128] In one possible implementation, the raw data includes image data or video data.
[0129] In one possible implementation, the AI model further includes multiple network layers, including a first residual connection layer, a first normalization layer, a linear layer, a modified linear unit layer, a random deactivation layer, a second residual connection layer, and a second normalization layer.
[0130] Execution module 1004 is used for:
[0131] Based on the first calculation result and the input data, multiple network layers are executed sequentially to obtain the second calculation result, where the values of the elements in the second calculation result are hypercomplex numbers.
[0132] Based on the second calculation result, the feedforward layer is executed.
[0133] In one possible implementation, the AI model also includes a conversion layer;
[0134] Execution module 1004 is used for:
[0135] In the transformation layer, the first calculation result is transformed into the third calculation result, and the values of the elements in the third calculation result are real numbers;
[0136] The third calculation result is used as the input data for the feedforward layer, and the feedforward layer is executed.
[0137] In one possible implementation, hypercomplex numbers include complex numbers or dual numbers.
[0138] because Figure 10 The data processing device 1000 shown corresponds to the above. Figure 3 The computing node 101 in the illustrated embodiment, therefore Figure 10 For the specific implementation of the data processing device 1000 and its technical effects, please refer to the above. Figure 3 The relevant details in the illustrated embodiments are described in detail here, and will not be repeated here.
[0139] Exemplarily, the data processing device 1000 described above can be implemented in software or hardware. In the first example, the data processing device 1000 can be implemented in software. In this case, the data processing device 1000 can specifically be code running on a computing instance, such as code running in a physical device, virtual machine, or container. In the second example, when implemented in hardware, the data processing device 1000 can be implemented using a processor, or using a physical device including a processor, such as a server. The processor can be a CPU, ASIC, PLD, CPLD, FPGA, GAL, SoC, SDI chip, AI chip, or DPU, or any combination of the above processors. Furthermore, the number of processors included in the data processing device 1000 can be one or more, and the types of processors can be one or more. The specific number and types of processors can be set according to the actual business requirements of the application, and this embodiment does not limit this.
[0140] In addition, this application also provides a cluster that includes at least one computing device. Figure 11 This application provides a schematic diagram of the hardware structure of a computing device 1100, which, for example, can implement the above-described... Figure 3 The computing node 101 in the illustrated embodiment, etc.
[0141] like Figure 11 As shown, the computing device 1100 includes a processor 1101, a memory 1102, and a communication interface 1103. The processor 1101, memory 1102, and communication interface 1103 communicate via a bus 1104, or via wireless transmission or other means. The memory 1102 stores instructions, and the processor 1101 executes the instructions stored in the memory 1102. Further, the computing device 1100 may also include a memory unit 1105, which is connected to the processor 1101, the storage medium 1102, and the communication interface 1103 via the bus 1104. The memory 1102 stores program code, and the processor 1101 can read the program code stored in the memory 1102 into the memory unit 1105 and execute the program code in the memory unit 1105 to perform the following operations:
[0142] Obtain raw data, which serves as the input to the artificial intelligence (AI) model. The AI model includes an attention layer and a feedforward layer.
[0143] The input data for the attention layer is generated based on the original data. The values of the elements in the input data are hypercomplex numbers, which include real and imaginary parts.
[0144] The input data is calculated in the attention layer to obtain the first calculation result, and the values of the elements in the first calculation result are hypercomplex numbers.
[0145] Based on the first calculation result, the feedforward layer is executed.
[0146] It should be understood that in this embodiment, the processor 1101 can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete device assemblies, etc. A general-purpose processor can be a microprocessor or any conventional processor.
[0147] The memory 1102 may include read-only memory and random access memory, and provides instructions and data to the processor 1101. The memory 1102 may also include non-volatile random access memory.
[0148] The memory 1102 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).
[0149] The communication interface 1103 is used to communicate with other devices connected to the computing device 1100. The bus 1104 may include a data bus, as well as a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus 1104 in the figure.
[0150] It should be understood that the computing device 1100 in this application embodiment can correspond to executing the embodiments of this application. Figure 3 The method executed by computing node 101 in the illustrated method, and the above and other operations and / or functions implemented by computing device 1100, are respectively for the purpose of implementing... Figure 3 The process of the corresponding methods in [the document] will not be elaborated here for the sake of brevity.
[0151] Alternatively, multiple computing devices in the cluster can jointly execute the embodiments of this application. Figure 3 The method executed by compute node 101 in the illustrated method. In this case, the attention layer can be distributed across multiple compute devices in the cluster.
[0152] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the aforementioned data processing method.
[0153] This application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.
[0154] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.
[0155] The computer program product can be a software installation package. When any of the aforementioned data processing methods is required, the computer program product can be downloaded and executed on a computing device.
[0156] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.
[0157] The terminology used in the above embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, “one or more” refers to one, two, or more; the character “ / ” generally indicates that the preceding and following objects are in an “or” relationship. In the embodiments of this application, “simultaneously” means within the same time period, including situations where they are at the same moment. The terms “first,” “second,” etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate, and this is merely a way of distinguishing objects with the same attributes in the embodiments of this application.
[0158] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0159] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A data processing method, characterized in that, The method includes: Obtain raw data, which serves as input to an artificial intelligence (AI) model, which includes an attention layer and a feedforward layer; The input data for the attention layer is generated based on the original data, wherein the values of the elements in the input data are hypercomplex numbers, and the hypercomplex numbers include real and imaginary parts; The input data is calculated in the attention layer to obtain a first calculation result, wherein the values of the elements in the first calculation result are hypercomplex numbers; Based on the first calculation result, the feedforward layer is executed.
2. The method according to claim 1, characterized in that, The step of generating the input data for the attention layer based on the original data includes: The original data is encoded to obtain tensor data corresponding to the original data, wherein the values of the elements in the tensor data are real numbers; The tensor data is transformed to obtain the input data for the attention layer.
3. The method according to claim 2, characterized in that, The raw data includes image data or video data.
4. The method according to any one of claims 1 to 3, characterized in that, The AI model also includes multiple network layers, which include a first residual connection layer, a first normalization layer, a linear layer, a modified linear unit layer, a random deactivation layer, a second residual connection layer, and a second normalization layer. The step of executing the feedforward layer based on the first calculation result includes: Based on the first calculation result and the input data, the plurality of network layers are executed sequentially to obtain a second calculation result, wherein the values of the elements in the second calculation result are hypercomplex numbers; Based on the second calculation result, the feedforward layer is executed.
5. The method according to any one of claims 1 to 4, characterized in that, The AI model also includes a transformation layer; The step of executing the feedforward layer based on the first calculation result includes: In the conversion layer, the first calculation result is converted into a third calculation result, wherein the values of the elements in the third calculation result are real numbers; The third calculation result is used as the input data for the feedforward layer, and the feedforward layer is executed.
6. The method according to any one of claims 1 to 5, characterized in that, The hypercomplex number includes complex numbers or dual numbers.
7. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire raw data, which serves as the input to an artificial intelligence (AI) model, which includes an attention layer and a feedforward layer. A generation module is used to generate input data for the attention layer based on the original data, wherein the values of the elements in the input data are hypercomplex numbers, and the hypercomplex numbers include real and imaginary parts; A calculation module is used to perform calculations on the input data in the attention layer to obtain a first calculation result, wherein the values of the elements in the first calculation result are hypercomplex numbers. An execution module is used to execute the feedforward layer based on the first calculation result.
8. The apparatus according to claim 7, characterized in that, The generation module is used for: The original data is encoded to obtain tensor data corresponding to the original data, wherein the values of the elements in the tensor data are real numbers; The tensor data is transformed to obtain the input data for the attention layer.
9. The apparatus according to claim 8, characterized in that, The raw data includes image data or video data.
10. The apparatus according to any one of claims 7 to 9, characterized in that, The AI model also includes multiple network layers, which include a first residual connection layer, a first normalization layer, a linear layer, a modified linear unit layer, a random deactivation layer, a second residual connection layer, and a second normalization layer. The execution module is used for: Based on the first calculation result and the input data, the plurality of network layers are executed sequentially to obtain a second calculation result, wherein the values of the elements in the second calculation result are hypercomplex numbers; Based on the second calculation result, the feedforward layer is executed.
11. The apparatus according to any one of claims 7 to 10, characterized in that, The AI model also includes a transformation layer; The execution module is used for: In the conversion layer, the first calculation result is converted into a third calculation result, wherein the values of the elements in the third calculation result are real numbers; The third calculation result is used as the input data for the feedforward layer, and the feedforward layer is executed.
12. The apparatus according to any one of claims 7 to 11, characterized in that, The hypercomplex number includes complex numbers or dual numbers.
13. A cluster, characterized in that, The cluster includes at least one computing device, and each computing device includes a processor and memory; The memory is used to store instructions, and the processor executes the instructions stored in the memory to cause the cluster to perform the method as described in any one of claims 1 to 6.
14. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on at least one computing device, cause the at least one computing device to perform the method as described in any one of claims 1 to 6.
15. A computer program product containing instructions, characterized in that, When it is run on at least one computing device, it causes the at least one computing device to perform the method as described in any one of claims 1 to 6.