Inference result acquisition method, electronic device, and computer-readable storage medium

Abstract

The embodiments of the present application provide an inference result acquisition method, an electronic device, and a computer-readable storage medium. The inference result acquisition method comprises: after acquiring query text, an electronic device converting the query text into a token sequence, then acquiring, according to the token sequence, a first query matrix, a first key matrix, and a first value matrix in a prefill stage, and performing in parallel a dot product operation, a softmax normalization operation, and a weighted summation operation on the first query matrix, the first key matrix, and the first value matrix; next, in a decode stage, performing in parallel a matrix-vector operation, a softmax normalization operation, and a weighted summation operation on a current result vector, a second key matrix, and a second value matrix; and finally, obtaining a next token of an inference result according to a result of the weighted summation operation in the decode stage. The method uses parallel computing as a means to reduce computational latency and mitigate computational bottlenecks in prefill stages, and to increase bandwidth utilization and mitigate memory bottlenecks in decode stages.

Description

Methods for obtaining inference results, electronic devices, and computer-readable storage media

[0001] This application claims priority to Chinese Patent Application No. 202411670087.3, filed on November 20, 2024, entitled "Method for Obtaining Reasoning Results, Electronic Device and Computer-Readable Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of smart terminal technology, and in particular to a method for obtaining reasoning results, an electronic device, and a computer-readable storage medium. Background Technology

[0003] In the field of artificial intelligence, especially in natural language processing and machine learning tasks, the Transformer module has become a core technology. The core of the Transformer architecture is the self-attention mechanism, which enables the Transformer architecture to dynamically focus on different parts of the sequence when processing long sequences of data, thereby improving the efficiency and accuracy of information processing.

[0004] The computation process of self-attention can be divided into three main steps: First, the dot product between the query matrix and the key matrix is ​​calculated to obtain an initial attention score. Next, this score is added to a mask to allow certain information to be "seen" or ignored, thereby controlling the model's learning and prediction process. Then, these scores are normalized using a softmax function to obtain the final attention weights. Finally, these weights are multiplied by the value matrix to generate the output. Typically, the first and third steps are executed by a matrix computation unit, while the second step is executed by a vector computation unit.

[0005] Large-scale model inference is generally divided into two stages: prefill and decoding. Each stage has different computational characteristics regarding self-attention. The prefill stage processes multiple tokens at once, and its challenge lies in handling large intermediate results, which is typically computationally intensive, making computation a bottleneck. The decoding stage, on the other hand, processes only one token at a time, making memory bandwidth a bottleneck. Summary of the Invention

[0006] This application provides a method for obtaining inference results, an electronic device, and a computer-readable storage medium to reduce computational latency in the pre-filling stage, alleviate computational bottlenecks, and improve bandwidth utilization to alleviate memory access bottlenecks in the decoding stage.

[0007] In a first aspect, embodiments of this application provide a method for obtaining inference results, comprising: acquiring a user's query text; converting the query text into a tag sequence; obtaining a first query matrix, a first key matrix, and a first value matrix for a pre-filling stage based on the tag sequence; performing dot product, normalization, and weighted summation operations in parallel among the first query matrix, the first key matrix, and the first value matrix; obtaining a result vector output by the pre-filling stage based on the result of the weighted summation operation in the pre-filling stage; and obtaining a second key matrix and a second value matrix for a decoding stage based on the hidden matrix in the pre-filling stage operation; wherein the result vector output by the pre-filling stage is the first tag of the inference result corresponding to the query text; performing matrix-vector operations, normalization, and weighted summation operations in parallel among the current result vector, the second key matrix, and the second value matrix; and obtaining the next tag of the inference result based on the result of the weighted summation operation in the decoding stage.

[0008] In one possible implementation, after obtaining the next tag of the inference result based on the result of the weighted summation operation in the decoding stage, the method further includes: repeatedly performing the matrix-vector operation, normalization operation, and weighted summation operation of the parallel computation of the current result vector, the second key matrix, and the second value matrix, and obtaining the next tag of the inference result based on the result of the weighted summation operation in the decoding stage, until the length of the obtained tag reaches a predetermined length or a terminator is generated.

[0009] In one possible implementation, the parallel execution of the dot product, normalization, and weighted summation operations between the first query matrix, the first key matrix, and the first value matrix includes: dividing the first query matrix into n query sub-matrices; where n is a positive integer; performing the dot product operation between the i-th query sub-matrix and the first key matrix to obtain the dot product result of the i-th query sub-matrix; where i is a positive integer, i≤n; performing the weighted summation operation between the normalization result of the (i-1)-th query sub-matrix and the first value matrix, and the normalization operation of the dot product result of the i-th query sub-matrix; after the weighted summation operation between the normalization result of the (i-1)-th query sub-matrix and the first value matrix is ​​completed, performing the dot product operation between the (i+1)-th query sub-matrix and the first key matrix.

[0010] In one possible implementation, the dot product operation between the i-th query submatrix and the first key matrix includes: performing the dot product operation between the i-th query submatrix and the first key matrix through a matrix operation unit.

[0011] In one possible implementation, the parallel execution of the normalization operation result of the (i-1)th query submatrix and the weighted summation operation of the first value matrix, and the normalization operation of the dot product operation result of the ith query submatrix, includes: after the matrix operation unit completes the dot product operation of the ith query submatrix and the first key matrix to obtain the dot product operation result of the ith query submatrix, the dot product operation result of the ith query submatrix is ​​passed to the vector operation unit; during the process of performing the weighted summation operation of the normalization operation result of the (i-1)th query submatrix and the first value matrix through the matrix operation unit, the normalization operation of the dot product operation result of the ith query submatrix is ​​performed through the vector operation unit.

[0012] In one possible implementation, the parallel computation of the current result vector, the second key matrix, and the second value matrix, including matrix-vector operations, normalization operations, and weighted summation operations, comprises: dividing the current result vector into m sub-vectors; where m is a positive integer; performing in parallel matrix-vector operations and normalization operations between the j-th sub-vector and the second key matrix, and performing a weighted summation operation between the normalization result of the (j-1)-th sub-vector and the second value matrix; where j is a positive integer, j≤m.

[0013] In one possible implementation, the parallel execution of matrix-vector operations and normalization operations between the j-th sub-vector and the second key matrix, and the weighted summation operation of the normalization result of the (j-1)-th sub-vector and the second value matrix, includes: merging the matrix-vector operations and normalization operations into a single operator; and, while using the merging operator to perform the matrix-vector operations and normalization operations between the j-th sub-vector and the second key matrix through the vector operation component, performing the weighted summation operation of the normalization result of the (j-1)-th sub-vector and the second value matrix through the matrix operation component.

[0014] In one possible implementation, the step of performing the dot product operation between the i-th query submatrix and the first key matrix to obtain the dot product result of the i-th query submatrix includes: dividing the first key matrix into n target submatrixes; performing the dot product operation between the i-th query submatrix and the i target submatrixes to obtain the dot product result of the i-th query submatrix.

[0015] In one possible implementation, the normalization operation of the dot product result of the i-th query submatrix includes: obtaining the s-th row vector of the dot product result of the i-th query submatrix; where s is a positive integer, s≤ the number of rows of the dot product result of the i-th query submatrix; and performing a normalization operation on the s-th row vector when s is divisible by the computational granularity of the vector operation unit.

[0016] In one possible implementation, after obtaining the s-th row vector of the dot product operation result of the i-th query submatrix, the method further includes: when s cannot be divided by the calculation granularity of the vector operation component, generating a mask based on the number of valid data in the s-th row vector to cover the data in the s-th row vector other than the valid data; and performing a normalization operation on the s-th row vector after the masking operation.

[0017] Secondly, embodiments of this application provide an electronic device, including: one or more processors; a memory; multiple application programs; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method provided in the first aspect.

[0018] It should be understood that the second aspect of the embodiments of this application is consistent with the technical solution of the first aspect of the embodiments of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be described again.

[0019] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method provided in the first aspect.

[0020] Fourthly, embodiments of this application provide a computer program product, which includes a computer program that, when run on a computer, causes the computer to perform the method provided in the first aspect.

[0021] In one possible design, the program in the fourth aspect can be stored wholly or partially on a storage medium packaged with the processor, or it can be stored wholly or partially on a memory not packaged with the processor. Attached Figure Description

[0022] Figure 1 is a schematic diagram of an AI accelerator;

[0023] Figure 2 is a schematic diagram of adding a mask in attention calculation;

[0024] Figure 3 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0025] Figure 4 is a schematic diagram of the system architecture provided in an embodiment of this application;

[0026] Figure 5 is a flowchart of a method for obtaining reasoning results provided in an embodiment of this application;

[0027] Figure 6 is a flowchart of a method for obtaining reasoning results provided in another embodiment of this application;

[0028] Figure 7 is a schematic diagram of parallel computing in the pre-filling stage provided in an embodiment of this application;

[0029] Figure 8 is a flowchart of a method for obtaining reasoning results provided in another embodiment of this application;

[0030] Figure 9 is a schematic diagram of parallel computation in the decoding stage provided in an embodiment of this application;

[0031] Figure 10 is a flowchart of mask-aware computation provided in an embodiment of this application;

[0032] Figure 11 is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0033] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0034] In existing related technologies, artificial intelligence (AI) accelerators are generally used to accelerate self-attention calculations. Figure 1 is a schematic diagram of an AI accelerator. As shown in Figure 1, an AI accelerator is usually composed of scalar components, vector operation components, and cube operation components. The following is a brief introduction to each component.

[0035] 1) Scalar component: Performs scalar calculations and basic control logic. Scalar calculations refer to operations on individual data points.

[0036] Implementation: In most processors, such as the central processing unit (CPU) or graphics processing unit (GPU), the standard computing core is capable of handling scalar operations.

[0037] 2) Vector component: Implements vector operations, that is, operations on one-dimensional arrays. This operation is very common in many applications such as image processing and / or physical simulation.

[0038] Implementation: In GPUs, vector operations can typically be handled by a single instruction multiple data unit in the computing core.

[0039] 3) Cube component: Implements matrix operations, which is especially important for deep learning and complex scientific computing.

[0040] The various components mentioned above are connected via high-speed data paths, ensuring efficient data transfer between different computing units and thus optimizing overall computing performance. For example, GPUs incorporate high-bandwidth memory access designs, such as L1 / L2 caches and shared memory, all optimized to support high-speed data exchange.

[0041] Large language models play a crucial role in modern technology, providing powerful intelligent support for a wide range of online services. Whether delivering natural-flowing conversational experiences, assisting users in coding, or even generating innovative videos and images, these models demonstrate exceptional capabilities. In these applications, attention mechanisms are key, enabling large language models to focus on the most important parts of the input data, thus providing more accurate and personalized services. This capability not only enhances user experience but also revolutionizes various industries, driving the widespread application and development of artificial intelligence technology.

[0042] To address the computational bottleneck in the pre-filling stage of large model inference, existing technologies often employ FlashAttention to optimize it. FlashAttention is a method that partitions attention computation along the Reduce dimension. Specifically, this method first breaks down the originally continuous computation into smaller blocks. Then, it adjusts the block size and modifies the iteration methods of the QK, softmax, and SV operations, allowing intermediate values ​​from each computation to be stored entirely within the chip, thus reducing access to off-chip memory. In particular, the memory usage of some intermediate values ​​is related to the square of the sequence length, making FlashAttention more effective for long sequences. It should be noted that in this paper, QK refers to the step in self-attention computation where the dot product between the query matrix and the key matrix is ​​calculated to obtain the initial attention score; softmax is the step in self-attention computation where the attention score is normalized using a normalization function to obtain the final attention weights; and SV is the step in self-attention computation where the attention weights and value matrix are weighted and summed.

[0043] To address the memory access bottleneck in the inference and decoding stages of large models, existing technologies include flash decoding, which increases parallelism and improves GPU computing resource utilization by partitioning the sequence. Furthermore, to reduce the overhead of generating duplicate keys and values, historical keys and values ​​can be cached in memory. Since the keys and values ​​accessed during the decoding stage are historical information, they are cached in memory. For memory management, page attention is typically used. This technique borrows from operating system memory management strategies, dividing memory into fixed-size pages. For keys and values ​​of uncertain length, if their length exceeds the allocated memory size, an additional fixed-size page is allocated. Page attention optimizes memory usage, reduces idle memory, and allows for more available memory to support larger batch sizes, thus optimizing latency in the decoding stage.

[0044] Furthermore, due to the causal nature of attention calculations, the current input can only be associated with previous inputs, not subsequent inputs. Therefore, a mask needs to be added to the attention calculation to mask some of the correlations between inputs, thus ensuring causality. Since the length of each user input and the length of the system prefix are uncertain, the mask shape is a trapezoid with uncertain length and width. The introduction of speculative inference optimization techniques further contributes to the irregularity of the mask. Generating an irregular mask directly using an AI accelerator is inefficient, so it is typically generated using a CPU. Positions where inputs cannot be correlated are set to negative infinity, and other positions are set to zero. Finally, the generated mask is transferred to the AI ​​accelerator's memory. Figure 2 illustrates the addition of a mask in attention calculations. As shown in Figure 2, the mask is added to the attention score to achieve the masking effect.

[0045] In existing related technologies, when using FlashAttention technology in the pre-filling stage, QK and SV operations are standard general matrix multiplications, performed on the matrix operation unit. The softmax operation is a vector computation, performed on the vector operation unit. Although the computational cost of vector operations is less than that of general matrix multiplication, the computational power of the vector operation unit is usually much lower than that of the matrix operation unit, so the softmax operation still takes up a significant amount of time. Furthermore, because the pre-filling stage is limited by computational bottlenecks, the softmax operation further exacerbates the computational bottleneck problem.

[0046] FlashDecoding optimizes the utilization of computing resources, but it doesn't solve the memory access bottleneck during the decoding stage. Furthermore, while PageAttention's management of key and value caches improves memory utilization, it also fragments this memory into blocks. With all keys and values ​​cached in memory during decoding, the bandwidth utilization for moving these memory blocks is insufficient, further exacerbating the memory access bottleneck during decoding.

[0047] Furthermore, while generating masks externally and feeding them into AI accelerators is a common practice, this approach has several drawbacks. First, during the inference phase, especially in real-time applications, each new input may require a custom mask, and dynamically generating masks is extremely time-consuming. It's also impossible to generate a fixed mask by padding to the maximum length. Even if mask generation could be performed on a GPU / neural-network processing unit (NPU), the overhead of generating and feeding it into the AI ​​accelerator is still significant. Second, since the mask size is quadratically related to the input, the resulting storage overhead and additional scoring operations become bottlenecks for long inputs. Finally, because many mask-generated elements are zero (especially when using inverted triangle and sliding window masks), this means that computation on these elements is useless, yet computational resources are still allocated to processing them, leading to resource waste.

[0048] Based on the above problems, this application provides a method for obtaining inference results. This method can be applied to electronic devices to improve the efficiency of electronic devices in processing attention calculations, enhance service response speed and resource utilization efficiency, and ensure that users obtain a smooth and accurate experience in various application scenarios.

[0049] The aforementioned electronic device can be a server, on which a large language model is deployed. This server can be a cloud server or a terminal server. This application does not impose any restrictions on the specific type of electronic device.

[0050] For example, Figure 3 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. As shown in Figure 3, the electronic device 300 includes a processor 310 and a communication interface 320. Optionally, the electronic device 300 may also include a memory 330. The processor 310, the communication interface 320, and the memory 330 can communicate with each other through internal connection paths to transmit control and / or data signals. The memory 330 is used to store computer programs, and the processor 310 is used to call and run the computer programs from the memory 330.

[0051] The processor 310 and memory 330 can be combined into a single processing device, but more commonly they are independent components. The processor 310 executes the program code stored in the memory 330. In specific implementations, the memory 330 can be integrated into the processor 310, or it can be independent of the processor 310.

[0052] In addition, to further enhance the functionality of the electronic device 300, the electronic device 300 may also include one or more of an input unit 360 and a display unit 370.

[0053] Optionally, the electronic device 300 may also include a power supply 350 for providing power to various devices or circuits in the electronic device 300.

[0054] It should be understood that the processor 310 in the electronic device 300 shown in Figure 3 can be a system-on-a-chip (SoC). The processor 310 may include a CPU and may further include other types of processors, such as GPUs and / or NPUs.

[0055] For ease of understanding, the following embodiments of this application will take an electronic device with the structure shown in FIG3 as an example, and in conjunction with the accompanying drawings and application scenarios, specifically illustrate the method for obtaining the reasoning results provided by the embodiments of this application.

[0056] Figure 4 is a schematic diagram of the system architecture provided in one embodiment of this application. As shown in Figure 4, after the user's query text is input into the large language model system, it first undergoes a pre-filling stage to generate the first tag of the inference result, followed by a decoding stage with an unknown number of autoregressive iterations to generate subsequent outputs. Regarding the masks introduced in the QK, Softmax, and SV stages during attention computation to ensure causality, this embodiment proposes mask-aware computation to eliminate the overhead caused by mask generation, transmission, and / or storage. Furthermore, since the pre-filling stage and the decoding stage have different problems—one is a computational bottleneck, and the other is a memory access bottleneck—this embodiment proposes a triple pipeline and computational load redistribution scheme to optimize computational and memory access bottlenecks and accelerate the speed at which the large language model responds to users.

[0057] The method for obtaining inference results provided in this application embodiment will be described below with reference to the system architecture shown in Figure 4.

[0058] Figure 5 is a flowchart of a method for obtaining inference results according to an embodiment of this application. The method for obtaining inference results can be executed by an electronic device 300. As shown in Figure 5, the method for obtaining inference results may include:

[0059] Step 501: Obtain the user's query text.

[0060] Referring to Figure 4, the user's query text can be: "Help me analyze this C++ code.", "Translate the sentence above.", and "Where is the capital city of France?". In practice, users can input query requests in the form of text, voice, and / or images. This embodiment does not limit the way users input query requests.

[0061] When a user enters a query request in text form, the electronic device 300 can directly obtain the query text from the query request; when a user enters a query request in voice and / or image form, the electronic device 300 can first recognize the query request and obtain the query text from the recognition result.

[0062] Step 502: Convert the above query text into a token sequence.

[0063] Specifically, in Figure 4, ABCD is the marker sequence of the above query text transformation.

[0064] Step 503: Obtain the first query matrix, the first key matrix, and the first value matrix for the pre-filling stage based on the above-mentioned tag sequence.

[0065] In Figure 4, Q represents the first query matrix, K represents the first key matrix, and V represents the first value matrix.

[0066] Step 504: Perform the dot product operation, normalization operation, and weighted summation operation between the first query matrix, the first key matrix, and the first value matrix in parallel.

[0067] In the pre-filling stage, the dot product operation, or QK operation, is used to calculate the dot product between the first query matrix and the first key matrix to obtain the initial attention score; the normalization operation, or softmax operation, is used to normalize the above attention score through a normalization function to obtain the final attention weight; the weighted summation operation, or SV operation, is a weighted summation operation performed on the above attention weight and the first value matrix in the attention calculation.

[0068] In this step, the specific process of performing the dot product operation, normalization operation, and weighted summation operation between the first query matrix, the first key matrix, and the first value matrix in parallel can be found in the description of the embodiment shown in Figure 6 of this application, and will not be repeated here.

[0069] Step 505: Based on the result of the weighted summation operation in the pre-filling stage, obtain the result vector output by the pre-filling stage; and based on the hidden matrix in the operation process of the pre-filling stage, obtain the second key matrix and the second value matrix for the decoding stage.

[0070] The result vector output by the pre-filling stage is the first label of the reasoning result corresponding to the query text, as shown in Figure 4. The result vector output by the pre-filling stage is vector E.

[0071] Step 506: Perform matrix-vector operations, normalization operations, and weighted summation operations on the current result vector, the second key matrix, and the second value matrix in parallel.

[0072] In the decoding stage, matrix-vector operations, namely QK operations, calculate the product between the current result vector and the second key matrix to obtain the attention score; normalization operations, namely softmax operations, also use a normalization function to normalize the above attention scores to obtain the final attention weights; weighted summation operations, namely SV operations, are performed in the decoding stage based on the above attention weights and the second key matrix.

[0073] Step 507: Based on the result of the weighted summation operation in the above decoding stage, obtain the next tag of the above reasoning result.

[0074] In some examples, after step 507, the electronic device 300 may repeat steps 506 to 507 until the length of the obtained marker reaches a predetermined length or a terminator is generated. The predetermined length can be set by the specific implementation; this embodiment does not limit the size of the predetermined length.

[0075] Referring to Figure 4, after the output result vector E in the pre-filling stage, the electronic device 100 repeats steps 506 to 507. After multiple autoregressive iterations, the labels of the inference results after vector E can be obtained sequentially, namely vector F, vector G, and vector H, until a terminator is generated. <eos>.

[0076] In the above method for obtaining the inference result, after the electronic device 300 obtains the user's query text, it converts the query text into a token sequence. Then, based on the token sequence, it obtains the first query matrix, the first key matrix, and the first value matrix for the pre-filling stage. It then performs dot product, normalization, and weighted summation operations between the first query matrix, the first key matrix, and the first value matrix in parallel. Next, based on the result of the weighted summation operation in the pre-filling stage, it obtains the result vector output from the pre-filling stage. Furthermore, based on the hidden matrix obtained during the pre-filling stage, it obtains the second key matrix and the second value matrix for the decoding stage. It then performs matrix-vector operations, normalization, and weighted summation operations between the current result vector, the second key matrix, and the second value matrix in parallel. Finally, based on the result of the weighted summation operation in the decoding stage, it obtains the next token for the inference result. This allows for reduced computational latency and alleviated computational bottlenecks during the pre-filling stage through parallel computation; and improved bandwidth utilization and alleviated memory access bottlenecks during the decoding stage through parallel computation.

[0077] Figure 6 is a flowchart of a method for obtaining reasoning results provided in another embodiment of this application. As shown in Figure 6, in the embodiment shown in Figure 5 of this application, step 504 may include:

[0078] Step 601: Divide the first query matrix into n query submatrices.

[0079] Where n is a positive integer.

[0080] Step 602: Perform the dot product operation between the i-th query submatrix and the first key matrix to obtain the dot product result of the i-th query submatrix.

[0081] Where i is a positive integer, i≤n.

[0082] Specifically, performing the dot product operation between the i-th query submatrix and the first key matrix can be done by: performing the dot product operation between the i-th query submatrix and the first key matrix through the matrix operation unit.

[0083] Step 603: Perform the weighted summation of the normalized result of the (i-1)th query submatrix and the first value matrix, and the normalization of the dot product result of the ith query submatrix in parallel.

[0084] Specifically, the parallel execution of the normalization operation result of the (i-1)th query submatrix and the weighted summation operation of the first value matrix, as well as the normalization operation of the dot product operation result of the ith query submatrix, can be performed as follows: after the matrix operation unit completes the dot product operation of the ith query submatrix and the first key matrix and obtains the dot product operation result of the ith query submatrix, the dot product operation result of the ith query submatrix is ​​passed to the vector operation unit; during the process of performing the weighted summation operation of the normalization operation result of the (i-1)th query submatrix and the first value matrix through the matrix operation unit, the normalization operation of the dot product operation result of the ith query submatrix is ​​performed through the vector operation unit.

[0085] Step 604: After the normalization operation result of the (i-1)th query submatrix and the weighted summation operation of the first value matrix are completed, the dot product operation of the (i+1)th query submatrix and the first key matrix is ​​performed.

[0086] In this embodiment, the electronic device 300 employs a triple pipeline scheme in the pre-filling stage to alleviate the computational bottleneck problem. First, the first query matrix is ​​divided into n query sub-matrices, denoted by {Tiling1,Tiling2,…,Tiling n}; the first key matrix is ​​not divided, thus ensuring that the divided sub-matrices can fully utilize all cores of the AI ​​accelerator during QK operations, and that the intermediate data of the division can be completely stored in the chip's cache without accessing off-chip memory. At the same time, the integrity of the first key matrix is ​​preserved, allowing the divided tasks to run independently and avoiding synchronization waiting overhead.

[0087] Considering that the softmax operation in the pre-filling stage takes about the same amount of time as the combined QK and SV operations, and that the softmax operation is performed by the vector operation unit while the QK and SV operations are performed by the matrix operation unit, a parallel pipeline approach can be adopted to perform the softmax, QK, and SV operations, thus saving computation time. Specifically, after the matrix operation unit completes the QK operation for the i-th till, it passes the dot product result of the i-th till to the vector operation unit, which then performs the softmax operation on the dot product result of the i-th till. Simultaneously, the matrix operation unit does not wait for the vector operation unit to complete its operation but immediately begins executing the SV operation for the (i-1)-th till. Furthermore, even after the matrix operation unit completes the SV operation for the (i-1)-th till, the vector operation unit continues to perform the softmax operation. Therefore, the matrix operation unit then immediately executes the QK operation for the (i+1)-th till.

[0088] Here, the QK operation of the i-th Tilling refers to the dot product operation of the i-th Tilling and the first key matrix, and the SV operation of the (i-1)-th Tilling refers to the weighted summation operation of the normalized result of the (i-1)-th Tilling and the first value matrix.

[0089] In this embodiment, the electronic device 300 reduces the idle time of the computing unit and alleviates the computing bottleneck problem in the pre-filling stage by overlapping the softmax operation performed by the vector operation unit with the QK and SV operations performed by the matrix operation unit in time.

[0090] Figure 7 is a schematic diagram of parallel computing in the pre-filling stage according to an embodiment of this application. In Figure 7, gray squares represent operations performed by the matrix operation unit, white squares represent operations performed by the vector operation unit, and the numbers on the squares indicate the tilling number. As can be seen from box 71 in Figure 7, the matrix operation unit first performs the QK operation for the first tilling. After the operation is completed, the matrix operation unit passes the dot product result of the first tilling to the vector operation unit, which then performs the softmax operation on the dot product result of the first tilling. At the same time, the matrix operation unit does not wait for the vector operation unit to complete its operation, but immediately begins to execute the QK operation for the second tilling.

[0091] After completing the QK operation of the second tilling, the matrix operation unit passes the dot product result of the second tilling to the vector operation unit, which then performs the softmax operation on the dot product result of the second tilling. Simultaneously, the matrix operation unit does not wait for the vector operation unit to complete its operation but immediately begins executing the SV operation of the first tilling. Furthermore, even after the matrix operation unit completes the SV operation of the first tilling, the vector operation unit is still performing the softmax operation. Therefore, the matrix operation unit then immediately executes the QK operation of the third tilling. The subsequent tilling operations follow the same process and will not be described further here.

[0092] In addition, as can be seen from box 72 in Figure 7, the electronic device 300 sets up two buffers in the cache for parallel processing in the pipeline. With the design of two buffers, only the space of two tilling data is needed to satisfy the buffer flow of intermediate data of the square of the sequence length (sel_len^2).

[0093] It should be noted that the content expressed by box 71 in Figure 7 is the same as that expressed by box 41 in Figure 4, so box 41 in Figure 4 will not be explained again.

[0094] In this embodiment, the electronic device 300 overlaps the QK operation, SV operation and softmax operation in time to form a triple pipeline, thereby improving the utilization rate of the vector operation unit and the matrix operation unit, reducing the computational delay, and alleviating the computational bottleneck in the pre-filling stage.

[0095] Figure 8 is a flowchart of a method for obtaining reasoning results provided in another embodiment of this application. As shown in Figure 8, in the embodiment shown in Figure 5 of this application, step 506 may include:

[0096] Step 801: Divide the current result vector into m sub-vectors.

[0097] Where m is a positive integer.

[0098] Step 802: Perform matrix-vector operations and normalization operations between the j-th sub-vector and the second key matrix in parallel, and perform a weighted summation operation between the normalization result of the (j-1)-th sub-vector and the second value matrix.

[0099] Where j is a positive integer, j≤m.

[0100] Specifically, the parallel execution of matrix-vector operations and normalization operations between the j-th sub-vector and the second key matrix, as well as the weighted summation operation of the normalization result of the (j-1)-th sub-vector and the second value matrix, can be achieved by: merging the matrix-vector operations and normalization operations into a single operator; using this merged operator, performing the matrix-vector operations and normalization operations between the j-th sub-vector and the second key matrix through the vector operation component, and performing the weighted summation operation of the normalization result of the (j-1)-th sub-vector and the second value matrix through the matrix operation component.

[0101] In this embodiment, the electronic device 300 alleviates the memory access bottleneck problem during the decoding phase by redistributing computational load and employing a triple pipeline scheme. During the decoding phase, since there is only one Query, Q changes from a matrix to a vector (vector E in Figure 4). Thus, the QK and SV operations during decoding change from general matrix operations to matrix-vector operations. If these two operations are still deployed on the matrix operation unit, the computational efficiency is not high, and it will also increase the computation time. However, the softmax operation is still a vector operation. Therefore, the time taken by the vector operation unit to perform the softmax operation is much less than the sum of the times taken by the matrix operation unit to perform the QK and SV operations. Thus, the optimization method of overlapping the softmax operation with the QK and SV operations in time is no longer feasible. Therefore, in this embodiment, the electronic device 300 first performs a computational load redistribution, moving the QK operation to the vector operation unit, which then performs the QK operation. Furthermore, the QK operation and the softmax operation are merged into a single operator, i.e., a single fusion operator is used to perform the QK operation and the softmax operation. This ensures that the time taken by the vector operation unit to perform the QK operation and the softmax operation is comparable to the time taken by the matrix operation unit to perform the SV operation.

[0102] Specifically, in the vector operation unit, during the QK and softmax operations of the j-th sub-vector using the fusion operator, the matrix operation unit performs the SV operation of the (j-1)-th sub-vector. The QK operation of the j-th sub-vector refers to the matrix-vector operation between the j-th sub-vector and the second key matrix; the softmax operation of the j-th sub-vector normalizes the result of the matrix-vector operation using the softmax function; and the SV operation of the (j-1)-th sub-vector is a weighted summation operation based on the normalized result of the (j-1)-th sub-vector and the second key matrix.

[0103] Furthermore, while the vector operation unit loads the key cache of the j-th sub-vector, the matrix operation unit loads the valve cache of the (j-1)-th sub-vector. This allows for parallel access to different memory blocks, improving memory access bandwidth utilization and alleviating the memory access bottleneck during the decoding stage. Here, loading the key cache of the j-th sub-vector by the vector operation unit can be understood as the vector operation unit retrieving the second key matrix corresponding to the j-th sub-vector from the key cache; loading the valve cache of the (j-1)-th sub-vector by the matrix operation unit can be understood as the matrix operation unit retrieving the second value matrix of the (j-1)-th sub-vector from the valve cache.

[0104] Figure 9 is a schematic diagram of parallel computation in the decoding stage according to an embodiment of this application. In Figure 9, gray squares represent operations performed by the matrix operation unit, white squares represent operations performed by the vector operation unit, and the numbers on the squares indicate the nth sub-vector. As can be seen from Figure 9, the QK operation and softmax operation are merged into one operation, which is performed by the vector operation unit, while the SV operation is performed by the matrix operation unit. In each pipeline, the QK and softmax fusion operator overlaps with the SV operation in time, forming a pipeline. Specifically, firstly, the vector operation unit uses the fusion operator to perform the QK and softmax operations on the first sub-vector, and then passes the normalized result of the first sub-vector to the matrix operation unit, which performs the SV operation on the first sub-vector. At the same time, the vector operation unit does not wait for the matrix operation unit to finish its operation, but immediately performs the QK and softmax operations on the second sub-vector. The subsequent sub-vector operations are performed in the same way and will not be described again here.

[0105] It should be noted that the content expressed in box 42 of Figure 9 is the same as that in Figure 4, so box 42 of Figure 4 will not be explained again.

[0106] In this embodiment, the electronic device 300 reduces data transmission overhead by merging QK and Softmax into a single operator. Pipeline technology enables the vector operation unit and matrix operation unit to access the Key cache and Value cache in parallel, thereby improving bandwidth utilization and alleviating the memory access bottleneck in the decoding stage.

[0107] In the embodiment shown in Figure 6 of this application, step 602 can be: dividing the first key matrix into n target sub-matrices, and then performing a dot product operation between the i-th query sub-matrix and the i-th target sub-matrix to obtain the dot product result of the i-th query sub-matrix.

[0108] Thus, the normalization operation of the dot product result of the i-th query submatrix in step 603 can be performed as follows: obtain the s-th row vector of the dot product result of the i-th query submatrix; where s is a positive integer, s ≤ the number of rows in the dot product result of the i-th query submatrix; when s is divisible by the calculation granularity of the vector operation component, perform normalization on the s-th row vector; when s is not divisible by the calculation granularity of the vector operation component, generate a mask based on the number of valid data in the s-th row vector to mask the data in the s-th row vector other than the valid data, and then perform normalization on the s-th row vector after the masking operation. The calculation granularity of the vector operation component can be set by the implementation itself; for example, the calculation granularity can be 128.

[0109] In other words, when normalizing the dot product result of the i-th query submatrix, for the portion of the dot product result whose length is divisible by the computational granularity of the vector operation unit, a strategy without physical mask generation is adopted. This is directly controlled by the instruction set to avoid mask generation and perform normalization directly. For the portion of the dot product result whose length is not divisible by the computational granularity of the vector operation unit, especially at the edges of the dot product result, a locally generated mask is used. In these areas, the data is directly modified to negative infinity, ensuring that data outside the edges does not affect the result when performing operations such as softmax. Specifically, for each row vector in the dot product result matrix, if the row number corresponding to this row vector is divisible by the computational granularity of the vector operation unit, then the strategy without physical mask generation is adopted for this row vector, and normalization is performed directly on this row vector. If the number of rows corresponding to this row vector cannot be divided by the calculation granularity of the vector operation unit, then a mask needs to be generated based on the number of valid data in the row vector to cover the data in the row vector other than the aforementioned valid data, and then the vector after the masking operation is normalized.

[0110] As can be seen from the above description, in this embodiment, masks are introduced in the QK, Softmax, and SV stages to ensure causality during attention calculation. This embodiment proposes mask-aware computation to eliminate the overhead caused by mask generation, transmission, and / or storage.

[0111] Figure 10 is a flowchart of mask-aware calculation provided in an embodiment of this application. As shown in Figure 10, it may include:

[0112] Step 1001: Process in batches.

[0113] In this step, batch processing means processing requests one by one. As shown in Figure 4, the user-input query text consists of three sentences, which can be considered three requests. In Figure 10, 1007 represents the mask matrix corresponding to these three requests. It can be seen that if calculated using the method in Figure 2, there are many black parts, indicating a significant amount of invalid computation. In this embodiment, the three requests are calculated separately during the attention mechanism calculation. Thus, the mask matrix corresponding to the three requests, as shown in 1008, has much less black content, resulting in significantly fewer invalid computations.

[0114] Step 1002, QK mask perception calculation.

[0115] Specifically, for each first query matrix, referring to the description in the embodiment shown in Figure 6 of this application, when performing QK operation, the first query matrix needs to be divided into n query sub-matrices. In this embodiment, when performing QK operation, mask-aware calculation is required, so the first key matrix needs to be further divided into n target sub-matrices. Then, the i-th query sub-matrix is ​​multiplied by the i-th target sub-matrix to obtain the dot product result of the i-th query sub-matrix. That is, the first query sub-matrix is ​​multiplied by one target sub-matrix divided by the first key matrix, the second query sub-matrix is ​​multiplied by two target sub-matrices divided by the first key matrix, and so on. The final dot product result of the first query matrix can be shown as 1009 in Figure 10.

[0116] In practical implementation, when dividing the first query matrix into n query submatrices, it can be done by row, meaning each row of the first query matrix can be divided into a query submatrix. Similarly, when dividing the first key matrix into n target submatrices, it can be done by column, meaning each column of the first key matrix can be divided into a target submatrix. Thus, during QK mask-aware calculation, multiplying the first query submatrix by one target submatrix derived from the first key matrix, and multiplying the second query submatrix by the two target submatrices derived from the first key matrix, can be done by multiplying the first row of the first query matrix by the first column of the first key matrix, multiplying the second row of the first query matrix by the first and second columns of the first key matrix, and so on.

[0117] Step 1003: Determine whether the length is divisible by the calculation granularity of the vector operation unit. If yes, proceed to step 1005; if the length is not divisible by the calculation granularity of the vector operation unit, proceed to step 1004.

[0118] Step 1004: In-place minimization of mask generation. Then proceed to step 1005.

[0119] Step 1005, normalization operation.

[0120] Specifically, for the dot product result of the first query matrix obtained in step 1002, the s-th row vector of the dot product result can be obtained; where s is a positive integer, s≤ the number of rows of the dot product result; when s is divisible by the calculation granularity of the vector operation component, the s-th row vector is directly normalized; when s is not divisible by the calculation granularity of the vector operation component, a mask is generated according to the number of valid data in the s-th row vector, for example, a fixed inverted triangle mask is generated to cover the data in the s-th row vector other than the valid data, and then the s-th row vector after the masking operation is normalized. In this way, accurate read and write operations can be achieved to filter out invalid calculations.

[0121] Step 1006, SV mask perception operation.

[0122] In this embodiment, after the normalization operation in step 1005, the obtained matrix can be as shown in 1010, and the first value matrix can be as shown in 1011. It can be seen that the first value matrix has also been divided into n sub-matrices, and the first value matrix has also masked the redundant parts according to the mask. Therefore, the first sub-matrix of the first value matrix is ​​smaller and gradually becomes larger.

[0123] Then, multiply the matrix 1010 obtained by the normalization operation with the submatrices divided by the first value matrix to obtain the result matrix 1012 of the SV operation.

[0124] In the above embodiments, when performing attention calculations, the electronic device 300 employs mask-aware computation in the QK, Softmax, and SV stages. For the portion whose length is divisible by the computational granularity of the vector operation unit, normalization is performed directly without generating a mask. For the portion whose length is not divisible by the computational granularity of the vector operation unit, an in-place mask generation method is used, eliminating the need to generate a mask from the CPU and then transmit it to the AI ​​accelerator. This reduces the computation, transmission, and storage overhead of generating a mask externally and then transmitting it internally. During attention calculations, the instructions are precisely controlled to complete the calculations for different mask conditions, reducing mask addition operations in attention calculations and eliminating redundant calculations for masked portions.

[0125] It is understood that some or all of the steps or operations in the above embodiments are merely examples, and other operations or variations thereof can be performed in the embodiments of this application. Furthermore, the steps may be performed in different orders as presented in the above embodiments, and it is not necessary to perform all the operations in the above embodiments.

[0126] It is understood that, in order to achieve the above-mentioned functions, electronic devices include hardware and / or software modules that perform the respective functions. Based on the algorithm steps of the examples described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0127] This embodiment can divide the electronic device into functional modules according to the above method embodiment. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0128] Figure 11 is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. When each functional module is divided according to its corresponding functions, Figure 11 shows a possible composition of the electronic device 1100 involved in the above embodiment. As shown in Figure 11, the electronic device 1100 may include: a receiving unit 1101, a processing unit 1102 and a transmitting unit 1103.

[0129] The receiving unit 1101 can be used to support the electronic device 1100 in performing step 501, and / or for other processes of the technical solutions described in the embodiments of this application;

[0130] The processing unit 1102 can be used to support the electronic device 1100 in executing steps 502 to 507, steps 601 to 604, steps 801 to 802, and steps 1001 to 1006, and / or other processes used in the technical solutions described in the embodiments of this application.

[0131] It should be noted that all relevant content of each step involved in the method embodiments shown in Figures 5 to 10 of this application can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0132] The electronic device 1100 provided in this embodiment is used to execute the above-described method for obtaining the reasoning result, and thus can achieve the same effect as the above-described method.

[0133] It should be understood that electronic device 1100 can correspond to electronic device 300 shown in FIG3. The functions of receiving unit 1101 and transmitting unit 1103 can be implemented by processor 310 and communication interface 320 in electronic device 300 shown in FIG3; the function of processing unit 1102 can be implemented by processor 310 in electronic device 100 shown in FIG3.

[0134] When using integrated units, the electronic device 1100 may include a processing module, a storage module, and a communication module.

[0135] The processing module can be used to control and manage the actions of the electronic device 1100. For example, it can support the electronic device 1100 in executing the steps performed by the receiving unit 1101, processing unit 1102, and sending unit 1103. The storage module can support the electronic device 1100 in storing program code and data. The communication module can support communication between the electronic device 1100 and other devices.

[0136] The processing module can be a processor or controller, which can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as radio frequency circuitry, a Bluetooth chip, and / or a Wi-Fi chip.

[0137] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device 1100 involved in this embodiment can be a device having the structure shown in FIG3.

[0138] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to execute the method provided in the embodiments shown in Figures 5 to 10 of this application.

[0139] This application also provides a computer program product, which includes a computer program that, when run on a computer, causes the computer to execute the method provided in the embodiments shown in Figures 5 to 10 of this application.

[0140] In this embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0141] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0142] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0143] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.< / eos>

Claims

1. A method for obtaining a reasoning result, characterized in that, include: Get the user's query text; Convert the query text into a sequence of tags; The first query matrix, the first key matrix, and the first value matrix for the pre-filling stage are obtained based on the marked sequence. The dot product, normalization, and weighted summation operations between the first query matrix, the first key matrix, and the first value matrix are performed in parallel. Based on the result of the weighted summation operation in the pre-filling stage, the result vector output by the pre-filling stage is obtained; and based on the hidden matrix in the operation process of the pre-filling stage, the second key matrix and the second value matrix of the decoding stage are obtained; wherein, the result vector output by the pre-filling stage is the first tag of the inference result corresponding to the query text; Parallel computation of the current result vector, the second key matrix, and the second value matrix; matrix-vector operations, normalization operations, and weighted summation operations; Based on the result of the weighted summation operation in the decoding stage, the next tag of the inference result is obtained.

2. The method according to claim 1, characterized in that, After obtaining the next tag of the inference result based on the result of the weighted summation operation in the decoding stage, the method further includes: Repeat the steps of performing matrix-vector operations, normalization operations, and weighted summation operations on the current result vector, the second key matrix, and the second value matrix in parallel computation, and obtaining the next tag of the inference result based on the result of the weighted summation operation in the decoding stage, until the length of the obtained tag reaches a predetermined length or a terminator is generated.

3. The method according to claim 1, characterized in that, The parallel execution of the dot product, normalization, and weighted summation operations among the first query matrix, the first key matrix, and the first value matrix includes: Divide the first query matrix into n query submatrices; where n is a positive integer; Perform the dot product operation between the i-th query submatrix and the first key matrix to obtain the dot product result of the i-th query submatrix; where i is a positive integer, i≤n; The normalization operation result of the (i-1)th query submatrix and the weighted summation operation of the first value matrix are performed in parallel, as well as the normalization operation of the dot product operation result of the ith query submatrix. After the normalization operation result of the (i-1)th query submatrix and the weighted summation operation of the first value matrix are completed, the dot product operation of the (i+1)th query submatrix and the first key matrix is ​​performed.

4. The method according to claim 3, characterized in that, The step of performing the dot product operation between the i-th query submatrix and the first key matrix includes: The matrix operation unit performs a dot product operation between the i-th query submatrix and the first key matrix.

5. The method according to claim 4, characterized in that, The parallel execution of the weighted summation of the normalized result of the (i-1)th query submatrix and the first value matrix, and the normalization of the dot product result of the ith query submatrix, includes: After the matrix operation unit performs the dot product operation between the i-th query submatrix and the first key matrix and obtains the dot product operation result of the i-th query submatrix, the dot product operation result of the i-th query submatrix is ​​passed to the vector operation unit. During the process of performing a weighted summation operation between the normalized result of the (i-1)th query submatrix and the first value matrix through the matrix operation unit, the normalization operation of the dot product result of the i-th query submatrix is ​​performed through the vector operation unit.

6. The method according to claim 1, characterized in that, The parallel computation of the current result vector, the second key matrix, and the second value matrix includes matrix-vector operations, normalization operations, and weighted summation operations: Divide the current result vector into m sub-vectors; where m is a positive integer; The matrix-vector operation and normalization operation between the j-th subvector and the second key matrix are performed in parallel, as well as the weighted summation operation between the normalization result of the (j-1)-th subvector and the second value matrix; where j is a positive integer and j≤m.

7. The method according to claim 6, characterized in that, The parallel execution of matrix-vector operations and normalization operations between the j-th sub-vector and the second key matrix, and the weighted summation operation between the normalization result of the (j-1)-th sub-vector and the second value matrix, includes: Integrate matrix-vector operations and normalization operations into a single operator; During the process of using the fusion operator to perform matrix-vector operations and normalization operations between the j-th sub-vector and the second key matrix through the vector operation component, the matrix operation component performs a weighted summation operation between the normalization result of the (j-1)-th sub-vector and the second value matrix.

8. The method according to claim 5, characterized in that, The step of performing the dot product operation between the i-th query submatrix and the first key matrix to obtain the dot product result of the i-th query submatrix includes: Divide the first key matrix into n target sub-matrices; Perform a dot product operation between the i-th query submatrix and the i-th target submatrix to obtain the dot product result of the i-th query submatrix.

9. The method according to claim 8, characterized in that, The normalization operation of the result of the dot product operation of the i-th query submatrix includes: Obtain the s-th row vector of the dot product result of the i-th query submatrix; where s is a positive integer, and s ≤ the number of rows of the dot product result of the i-th query submatrix; When s is divisible by the calculation granularity of the vector operation unit, the s-th row vector is normalized.

10. The method according to claim 9, characterized in that, After obtaining the s-th row vector of the dot product result of the i-th query submatrix, the process further includes: When s cannot be divided by the calculation granularity of the vector operation unit, a mask is generated based on the number of valid data in the s-th row vector to cover the data in the s-th row vector other than the valid data. Normalize the s-th row vector after the masking operation.

11. An electronic device, characterized in that, include: One or more processors; Memory; Multiple applications; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the electronic device, cause the electronic device to perform the method as described in any one of claims 1-10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-10.

Images