A data parallel processing method and device

Abstract

The present invention discloses a data parallel processing method and apparatus. The method comprises: determining at least two input sequences based on at least one piece of input data, and concatenating the input sequences according to their order to obtain a total sequence; wherein each input sequence includes at least one word; obtaining a parallel parameter, and dividing the total sequence based on the parallel parameter to obtain at least two processing sequences, wherein the parallel parameter is determined at least by the number of parallel devices; and performing parallel processing on the processing sequences via the at least two parallel devices to obtain a circular attention. The present invention can reduce the amount of padding in the input sequence, reduce the waste of computing resources, and improve computing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data parallel processing method and device. Background Art

[0002] With the development of natural language modeling technology, natural language models are finding increasingly widespread applications. When processing input data, natural language models typically segment the input data into sentences, convert the sentences into tokens, and then form sequences of varying lengths based on the token indexes. When processing mixed sequences of input data in parallel, padding is typically performed at the end of the remaining sequences, based on the length of the longest sequence. For example, if the mixed sequences are: [0,1,2,3,4,5,6,7,8,9,10], [0,1,2,3], [0,1,2], [0,1,2,3,4], and the longest sequence is used as the basis, the other sequences are padded to obtain: [0,1,2,3,4,5,6,7,8,9,10], [0,1,2,3,p,p,p,p,p,p,p], [0,1,2,p,p,p,p,p,p,p,p], [0,1,2,3,4,p,p,p,p,p,p,p], where p represents the padding bit. Finally, the padded sequences are processed separately by parallel devices.

[0003] The method of padding the other sequences based on the length of the longest sequence and then processing the mixed sequences in parallel wastes computing resources for processing the padded parts. This is especially true when the mixed input sequences vary greatly in length and there are many input sequences, resulting in a massive waste of computing resources. Summary of the Invention

[0004] The present invention provides a data parallel processing method and device to reduce the padding amount of input sequences, reduce the waste of computing resources, and improve computing efficiency.

[0005] In a first aspect, an embodiment of the present invention provides a data parallel processing method, the method comprising:

[0006] Determining at least two input sequences according to at least one input data, and concatenating the input sequences according to the order of the input sequences to obtain a total sequence;

[0007] Wherein, each input sequence includes at least one word unit;

[0008] Obtaining a parallel parameter, and dividing the total sequence based on the parallel parameter to obtain at least two processing sequences, wherein the parallel parameter is determined at least according to the number of parallel devices;

[0009] The processing sequence is processed in parallel by at least two parallel devices to obtain circular attention.

[0010] In a second aspect, an embodiment of the present invention further provides a data parallel processing device, the device comprising:

[0011] A sequence splicing module is used to determine at least two input sequences and splice the input sequences according to the order of the input sequences to obtain a total sequence;

[0012] Wherein, each input sequence includes at least one word unit;

[0013] a sequence splitting module, configured to obtain a parallel parameter, and split the total sequence based on the parallel parameter and the input sequence to obtain at least two processing sequences, wherein the parallel parameter is determined at least according to the number of parallel devices;

[0014] The sequence parallel processing module is used to process the processing sequence in parallel through at least two parallel devices to obtain ring attention.

[0015] The technical solution of the embodiment of the present invention converts input data into input sequences, and splices the input sequences according to the order of the input sequences to obtain a total sequence, divides the total sequence by parallel parameters to obtain at least two processing sequences, and processes the processing sequences in parallel by at least two parallel devices to obtain circular attention. This solves the problem in the prior art that when processing at least two sequences of different lengths, the other sequences need to be padded based on the length of the longest sequence, and then mixed sequences are processed in parallel, which wastes computing resources and reduces processing speed. It reduces the amount of padding in the input sequence, reduces the waste of computing resources, and improves computing efficiency. Compared with other circular attention calculation methods in the prior art, it also reduces the amount of data required to calculate the circular attention, further improving computing efficiency.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 This is a flow chart of a data parallel processing method provided by the first embodiment of the present invention;

[0019] Figure 2 This is a flow chart of a data parallel processing method provided by the second embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of masks of various processing sequences of a causal task provided by the second embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of masks of various processing sequences of a non-causal task provided by the second embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram of a pre-communication of a causal task provided by the second embodiment of the present invention;

[0023] Figure 6 Schematic diagram of pre-communication of a non-causal task provided by the second embodiment of the present invention;

[0024] Figure 7 1 is a schematic diagram of the structure of a data parallel processing device provided in Embodiment 3 of the present invention;

[0025] Figure 8 This is a structural diagram of an electronic device provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. In the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be considered as exemplary. Their purpose is merely to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0028] The acquisition, transmission, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0029] Example 1

[0030] Figure 1 A flowchart of a data parallel processing method is provided for the first embodiment of the present invention. This embodiment is applicable to the situation where input data is converted into an input sequence and then processed in parallel. The method can be executed by a data parallel processing device. The data parallel processing device can be implemented in the form of hardware and / or software. The data parallel processing device can be configured in an electronic device in which a natural language model is deployed.

[0031] like Figure 1 As shown, the method includes:

[0032] S110 , determining at least two input sequences according to at least one piece of input data, and concatenating the input sequences according to the order of the input sequences to obtain a total sequence.

[0033] Each input sequence includes at least one word.

[0034] The input data refers to the input data of the natural language model. The input data may be in various forms, including but not limited to text, images, video, and audio. Furthermore, the input data in this embodiment also supports multimodal mixing, such as combining text with images. Converting the input data into an input sequence, and converting multimodal input data into an input sequence, can be accomplished using conventional methods, which are not limited in this embodiment.

[0035] A token is a unit obtained by processing input data. Specifically, if the input data is text, the text can be segmented or tokenized to obtain tokens. For example, if the text is "I love you," the tokens obtained after segmentation are "I," "love," and "you." If the input data is an image, the image can be divided into small patches. Feature extraction and vectorization are performed on each patch. The resulting vector is then compared with a pre-set token vocabulary using a cosine distance or Euclidean distance calculation to find the token closest to each patch and use it as the token corresponding to that patch.

[0036] An input sequence is an ordered sequence consisting of at least one word unit. The applicable scenario of this embodiment is mainly used to process mixed input sequences, especially mixed input sequences of different lengths. Therefore, the input sequence in this embodiment is at least two.

[0037] According to the order of each input sequence, each input sequence is spliced to obtain a total sequence. Therefore, the total sequence contains all the tokens obtained after the input data conversion, and each token is arranged in order. For example, take the input sequence including the following four sequences as an example: [0,1,2,3,4,5,6,7,8,9,10], [0,1,2,3], [0,1,2], [0,1,2,3,4], and the total sequence is [0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,0,1,2,0,1,2,3,4]. It should be noted that 0-10 in the above input sequence and the total sequence are only used to refer to the initial position of each word in the original input sequence, and are not used to represent the actual content of the word.

[0038] In this embodiment, the input data for the natural language model is converted into an input sequence. Multiple input sequences of varying lengths are sequentially concatenated into a total sequence, facilitating subsequent sequence segmentation based on the total sequence. This approach, which segments the sequence based on the total sequence formed by sequentially concatenating the input sequences to generate multiple segmented sequences, reduces the amount of padding compared to padding each sequence by the length of the longest sequence, thereby saving computing resources and improving input data processing efficiency.

[0039] S120: Acquire a parallel parameter, and split the total sequence based on the parallel parameter to obtain at least two processing sequences.

[0040] The parallelism parameter is determined at least by the number of parallel devices. The number of parallel devices can be predetermined, for example, four. The parallelism parameter can be the same as the number of parallel devices, or an integer multiple of the number of parallel devices, for example, two. Furthermore, when the natural language model performs causal tasks on input data, pre-communication between the parallel devices is required. In this case, the parallelism parameter can be set to twice the number of parallel devices.

[0041] Processing sequences are sequences obtained by partitioning the total sequence based on the parallelism parameter. When the parallelism parameter is the same as the number of parallel devices, the number of processing sequences is the same as the number of input sequences, but the tokens corresponding to the same processing sequence and input sequence may be different. When the parallelism parameter is an integer multiple of the number of parallel devices, the number of processing sequences is also an integer multiple of the input sequence.

[0042] In the prior art, each sequence is padded according to the length of the longest sequence and then distributed to each parallel device for processing. Therefore, each parallel device needs to consume the computing resources required to process the longest sequence. Especially when the mixed input sequences are of different lengths and the length differences are large, it will cause a huge waste of computing resources and reduce the efficiency of input data processing.

[0043] In this embodiment, the total sequence is divided based on the parallel parameters. On the one hand, it can ensure less sequence filling and reduce the waste of computing resources; on the other hand, through token allocation and pre-communication, it can ensure that the number of tokens processed by each parallel device is balanced, thereby avoiding one or more parallel devices being idle or congested, ensuring load balance among parallel devices, and improving computing efficiency.

[0044] Furthermore, S120 may include:

[0045] S121, determining the processing sequence length according to the total number of word units in the total sequence and the parallel parameter;

[0046] S122. Split the total sequence according to the length of the processing sequence to obtain at least two processing sequences.

[0047] In this embodiment, to ensure that all word units in the total sequence can be processed, when the total sequence is segmented based on the parallel parameter, if the total number of word units in the total sequence cannot be divided by the parallel parameter, the end of the total sequence can be padded and then segmented based on the parallel parameter so that the total number of word units in the total sequence after padding can be divided by the parallel parameter; or the value after dividing the total number of word units by the parallel parameter can be rounded up, and the obtained value is used as the number of word units in the processing sequence, and the processing sequence is segmented, and the end of the last processing sequence is padded.

[0048] In this embodiment, padding and segmentation are performed based on the total sequence. This setting ensures that the number of padding does not exceed the value obtained by dividing the total number of words by the parallel parameter, thereby reducing the number of sequence padding; at the same time, the number of words in each processing sequence obtained after padding and segmentation is made the same, so that the load of each parallel device is balanced; and the number of words in each processing sequence, as well as the padding amount of sequence padding based on an integer multiple of the parallel parameter, are less than the number of words in the input sequence after padding each input sequence based on the length of the longest sequence in the prior art, thereby reducing the consumption of computing resources and improving the processing efficiency of input data.

[0049] Furthermore, S121 may include: if the total number of word elements in the total sequence is an integer multiple of the parallel parameter, then the value obtained by dividing the total number of word elements in the total sequence by the parallel parameter is used as the processing sequence length; otherwise, the processing sequence length is calculated by the following formula: len=┌a┐, a=T / D, where len represents the processing sequence length, ┌ ┐ represents rounding up, T represents the total number of word elements in the total sequence, and D represents the parallel parameter.

[0050] In this embodiment, if the total number of tokens in the total sequence is an integer multiple of the parallelism parameter, the total number of tokens in the total sequence divided by the parallelism parameter is directly used as the processing sequence length, without padding. Otherwise, to ensure that all tokens are processed evenly by the parallel devices, the total number of tokens divided by the parallelism parameter is rounded up to the integer value, which is used as the processing sequence length.

[0051] For example, taking the total sequence [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 0, 1, 2, 3, 0, 1, 2, 0, 1, 2, 3, 4], the total number of tokens is 23. If the parallelism parameter is 4, the total number of tokens in the total sequence is not an integer multiple of the parallelism parameter. Therefore, the value obtained by dividing the total number of tokens by the parallelism parameter (23 / 4 = 5.75) is rounded up to 6, which is used as the processing sequence length.

[0052] Furthermore, S122 may include: if the total number of word elements in the total sequence is an integer multiple of the parallel parameter, the total sequence is divided according to the processing sequence length; otherwise, the total sequence is divided according to the processing sequence length, and the end of the last processing sequence is padded to make the number of word elements in the last processing sequence consistent with the processing sequence length.

[0053] In this embodiment, if the total number of word units in the total sequence is an integer multiple of the parallel parameter, there is no need to pad the sequence, and the total sequence can be directly divided according to the length of the processing sequence.

[0054] For example, if the total number of dimensions of the entire sequence is 24 and the parallel parameter is 4, then according to the word order, 6 words can be directly used to form a processing sequence.

[0055] If the total number of tokens in the total sequence is not an integer multiple of the parallel parameter, the total sequence is segmented according to the processing sequence length obtained by rounding up the value obtained by dividing the total number of tokens by the parallel parameter. For the last processing sequence, the number of tokens is less than the processing sequence length. Therefore, the end of the last processing sequence is padded so that the number of tokens in the last processing sequence is consistent with the processing sequence length. For example, taking the above total sequence of [0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,0,1,2,0,1,2,3,4] as an example, the total number of tokens is 23, the parallel parameter is 4, and the processing sequence length is 6, then the processing sequences after segmentation are: [0,1,2,3,4,5], [6,7,8,9,10,0], [1,2,3,0,1,2] and [0,1,2,3,4]. Except for the last processing sequence, the number of tokens in the remaining processing sequences is 6. Therefore, the end of the last processing sequence is padded to obtain [0, 1, 2, 3, 4, p]. Here, p refers to padding. This embodiment does not limit the content of the padding. Preferably, the padding content is 0.

[0056] S130. Process the processing sequence in parallel through at least two parallel devices to obtain circular attention.

[0057] In this embodiment, each parallel device processes the processing sequence in parallel based on a ring attention mechanism, generating ring attention corresponding to the input data. Ring attention is a distributed optimization technique designed to overcome the memory and computational bottlenecks of traditional attention mechanisms when processing long sequences. Its core concept is to improve sequence parallel processing and memory efficiency based on a ring communication topology and a block computing strategy.

[0058] This embodiment uses a circular attention mechanism to calculate the circular attention corresponding to the input data. Because total sequence segmentation may split tokens originally belonging to the same input sequence into at least two processing sequences, the circular attention mechanism can focus on the correlation between contexts. Causal relationships still exist within the segmented processing sequences. Therefore, splitting an input sequence into multiple segments does not significantly reduce the quality of natural language model training or inference.

[0059] The technical solution of the embodiments of the present invention converts input data into input sequences, concatenates the input sequences according to their order, obtains a total sequence, partitions the total sequence using parallel parameters to obtain at least two processing sequences, and processes the processing sequences in parallel using at least two parallel devices to achieve circular attention. This solves the problem of the prior art of padding other sequences based on the length of the longest sequence before performing mixed sequence parallel processing, which wastes computing resources and reduces processing speed. It reduces the amount of padding in the input sequence, reduces computing resource waste, and improves computing efficiency.

[0060] Example 2

[0061] Figure 2 This is a flowchart of a data parallel processing method provided in the second embodiment of the present invention. Based on the above embodiments, this embodiment of the present invention further specifies the process of each parallel device performing parallel processing on each processing sequence to obtain circular attention.

[0062] like Figure 2 As shown, the method includes:

[0063] S210 . Determine at least two input sequences according to at least one piece of input data, and concatenate the input sequences according to the order of the input sequences to obtain a total sequence.

[0064] S220 , obtaining a parallel parameter, and dividing the total sequence based on the parallel parameter to obtain at least two processing sequences.

[0065] The method of converting input data into input sequences, concatenating the input sequences, and then segmenting them based on parallel parameters to obtain processing sequences has been described in the above embodiments and will not be repeated in this embodiment.

[0066] S230: Obtain a task type that matches the input data.

[0067] Task types include causal and non-causal tasks. Causal tasks rely on observing changes in input data over time and the order of input data. Causal tasks are implemented based on causal models. Non-causal tasks focus on correlations or patterns in input data. Non-causal tasks are implemented based on non-causal models.

[0068] S240: Construct a mask matching each processing sequence according to the task type and the input sequence.

[0069] The mask refers to the attention mask, which is used to block information that does not need to be paid attention to or highlight information that needs to be paid attention to when calculating the attention weight, so that the artificial intelligence model can focus on a specific part of the processing sequence.

[0070] The attention mask is typically a binary tensor of the same length as the processed sequence. When subsequently calculating the attention score, the mask is multiplied by the attention score tensor. Positions with a value of 0 in the mask have their corresponding attention scores set to extremely small values (usually negative infinity). After the softmax function, the attention weights for these positions approach 0, causing them to be ignored by the natural language model. Positions with a value of 1 in the mask, on the other hand, have their attention weights calculated normally, allowing the model to focus on the information corresponding to these positions.

[0071] In this embodiment, there are certain differences in the construction of masks depending on the task type. This is related to the principles of causal and non-causal tasks themselves. Causal models model and predict data based on causal relationships. When constructing masks, the purpose is to ensure that the model can only use information from the past and current moments when predicting the output at the current moment, and cannot use future information, in order to comply with the logic of causal relationships. Non-causal models do not rely on strict causal relationships. The purpose of their mask construction is usually to control the model's attention to different information, or to process some special structures in the data, such as blocking out some irrelevant areas or filling parts in images or text.

[0072] Furthermore, if the task type is a causal task, S240 may further include: constructing a mask matching each processing sequence based on a unidirectional attention mechanism;

[0073] If the task type is a non-causal task, S240 may further include: constructing a mask matching each processing sequence based on a bidirectional attention mechanism;

[0074] Unidirectional and bidirectional attention mechanisms are two different approaches for processing sequential data in fields such as natural language processing. Unidirectional attention, also known as forward attention, processes input data in a single direction, from the start to the end of the sequence. Consequently, when calculating the attention weight, only the sequence information preceding the current position is considered to calculate the attention level for each position. In bidirectional attention, however, information can flow in both directions when processing input data, from the start to the end of the sequence and vice versa. All sequence information preceding and following the current position is considered when calculating the attention weight.

[0075] Therefore, causal tasks conform to the unidirectional attention mechanism, and the masks of each processing sequence of causal tasks usually take the form of a lower triangular matrix. Figure 3 A schematic diagram of the mask of each processing sequence of a causal task is provided. Figure 3 Taking the processing sequences after segmentation as [0,1,2,3,4,5], [6,7,8,9,10,0], [1,2,3,0,1,2] and [0,1,2,3,4] as examples, the mask is explained. Specifically, in the second processing sequence, each token comes from two different input sequences. Therefore, the second processing sequence can be further decomposed into [6,7,8,9,10] and [0]. Similarly, for the third processing sequence, it can be further decomposed into [1,2,3] and [0,1,2]. Therefore, in this example, it is necessary to perform circular attention calculation on the six processing sequences. The mask of the causal task is as follows Figure 3 As shown in the red oblique line shaded part, it is in the form of six downward triangles, and the six downward triangles correspond to the six processing sequences after decomposition.

[0076] like Figure 3 As shown, the computing resources consumed by the technical solution of this embodiment can be compared with the existing technology by summing the areas of the lower triangles. It should be noted that the sum of the areas of the lower triangles in this embodiment has no practical meaning and is not used to represent the actual computing resources required. It is only used to compare the computing resource consumption with the existing technology. Figure 3 For example, the sum of the areas of the six lower triangles is 6×6 / 2+5×5 / 2+1×1 / 2+3×3 / 2+3×3 / 2+6×6 / 2=58. However, using existing techniques, the four input sequences after padding based on the longest input sequence are: [0,1,2,3,4,5,6,7,8,9,10], [0,1,2,3,p,p,p,p,p,p,p], [0,1,2,p,p,p,p,p,p,p,p], and [0,1,2,3,4,p,p,p,p,p,p,p]. The sum of the areas of the lower triangles is 11×11×4 / 2=242. This not only requires a large amount of computing resources, but also wastes a lot of computing resources by processing the invalid padding. Comparing the area of the lower triangle in this embodiment with that of the lower triangle using the prior art, it can be seen that this embodiment will significantly reduce the consumption and waste of computing resources and improve the efficiency of input data processing.

[0077] Non-causal tasks are consistent with the bidirectional attention mechanism, and the masks of each processing sequence of non-causal tasks are usually in the form of a two-dimensional matrix. Figure 4 A schematic diagram of the mask of each processing sequence of a non-causal task is provided. Figure 4The mask is explained by taking the processing sequences after segmentation as [0,1,2,3,4,5], [6,7,8,9,10,0], [1,2,3,0,1,2] and [0,1,2,3,4] as examples. Figure 4 In this example, when performing ring attention calculation on the six processing sequences, the mask of the non-causal task is as follows: Figure 4 As shown in the blue grid shaded part in , it is in the form of six squares, and the six squares correspond to the six processing sequences after decomposition.

[0078] like Figure 4 As shown, the computing resources required to adopt the technical solution of this embodiment can be compared with the prior art by the sum of the areas of the squares. It should be noted that the sum of the areas of the squares in this embodiment has no practical meaning and is not used to represent the actual computing resources required. It is only used to compare the computing resource consumption with the prior art. Using the technical solution of this embodiment, the sum of the areas of the squares is 6×6+5×5+1×1+3×3+3×3+6×6=116. If the prior art is used, the four input sequences after padding based on the longest input sequence are: [0,1,2,3,4,5,6,7,8,9,10], [0,1,2,3,p,p,p,p,p,p,p], [0,1,2,p,p,p,p,p,p,p,p], [0,1,2,3,4,p,p,p,p,p,p,p], and the sum of the areas of the squares at this time is 11×11×4=484. This not only requires more computing resources, but also wastes a lot of computing resources by processing the invalid fill portion. Comparing the square area in this embodiment with the square area using the prior art, it can be seen that this embodiment will significantly reduce the consumption and waste of computing resources and improve the efficiency of input data processing.

[0079] In this embodiment, masking is used to highlight the necessary parts for calculating the circular attention, reducing the amount of data and computational complexity, and improving computational efficiency. Combining masking with the sequence padding and segmentation methods in this embodiment not only significantly reduces computing resource usage and improves input data processing efficiency, but also reduces invalid padding and reduces the computational resource waste caused by processing invalid padding.

[0080] Furthermore, if the task type is a causal task, this embodiment also includes: performing pre-communication on each parallel device, and the pre-communication includes: allocating the query vector, key vector and value vector of each word required to process the causal task to each parallel device.

[0081] In this embodiment, the pre-communication is used to achieve load balancing among parallel devices in causal tasks, thereby improving computing efficiency.

[0082] Specifically, the tokens in each processing sequence are rearranged based on the number of parallel devices. Each token is divided into 2 parts (the number of parallel devices × 2) and represented by different Q (Queues). Figure 5 A schematic diagram of the pre-communication of a causal task is provided, such as Figure 5 As shown, taking four parallel devices as an example, the tokens of each processing sequence are divided into 8 parts, named Q0-Q7. Figure 5 As shown in the figure, Q0+Q7 (red oblique shaded part), Q1+Q6 (yellow oblique shaded part), Q2+Q5 (green oblique shaded part) and Q3+Q4 (blue oblique shaded part), the area represents the amount of data that needs to be processed. Through the above allocation, the area corresponding to each color is the same. Therefore, Q0+Q7, Q1+Q6, Q2+Q5 and Q3+Q4 are respectively allocated to four parallel devices for processing, which can ensure load balancing of each parallel device.

[0083] exist Figure 5 In the example, KV0-KV7 are the key-value pairs corresponding to Q0-Q7 respectively. Therefore, when distributing tokens to each parallel device during pre-communication, the query vector, key vector, and value vector corresponding to the token need to be pre-distributed to each parallel device. Figure 5 For example, KV0 and KV7 corresponding to Q0 and Q7 are allocated to the first parallel device, KV1 and KV6 corresponding to Q1 and Q6 are allocated to the second parallel device, and so on.

[0084] It should be noted that, due to the calculation of the circular attention of non-causal tasks, such as Figure 6 As shown in the figure, each parallel device processes the same number of tokens, and there is no load imbalance problem between parallel devices. Therefore, for non-causal tasks, no pre-communication is required.

[0085] S250. Perform parallel processing on each processing sequence through at least two parallel devices to determine an attention score for each word in each processing sequence.

[0086] The attention score of each token is used to indicate the relevance or importance of the token to other tokens in the sequence. Since other tokens are involved, different types of tokens are involved in calculating the attention score, which leads to different calculation methods for the attention score depending on the task type.

[0087] Furthermore, if the task type is a causal task, the attention score of each word in each processing sequence is determined, including: calculating the attention score of each word based on the query vector, key vector and value vector of each word corresponding to the unidirectional attention of each processing sequence.

[0088] In causal tasks, due to the use of a unidirectional attention mechanism, each position can only focus on the elements at that position and before it when calculating the attention score. Therefore, when calculating the attention score of a token, only the token and the tokens before it can be focused on. At this time, the query vector, key vector, and value vector of the token and the token before it are used to perform a correlation calculation to calculate the attention score of the current token. The calculation of the attention score can be done using the conventional attention score calculation method, which will not be described in detail in this embodiment.

[0089] Furthermore, if the task type is a non-causal task, the attention score of each word in each processing sequence is determined, including: calculating the attention score of each word based on the query vector, key vector and value vector of each word corresponding to the bidirectional attention of each processing sequence.

[0090] In non-causal tasks, due to the bidirectional attention mechanism, each position can pay attention to all positions in the sequence when calculating the attention score, including itself and all other elements in the ring structure. Therefore, for the current token, the attention score of the current token is obtained by calculating the correlation between the query vector, key vector, and value vector of all tokens from the start position to the end position in the sequence.

[0091] S260. Calculate the ring attention based on the mask matching each processing sequence and the attention score of each word in each processing sequence.

[0092] In this embodiment, a mask is applied to the calculation process of the ring attention, so that when the model outputs the ring attention, it does not pay attention to invalid positions, thereby reducing the complexity of the calculation and improving the calculation efficiency.

[0093] Furthermore, S260 may include:

[0094] S261, determining the attention weight of each word-unit based on the mask matching each processing sequence and the attention score of each word-unit;

[0095] S262. Based on the mask matching each processing sequence and the attention weight of each word, the attention weight of each word in each processing sequence is weighted and summed to obtain a circular attention.

[0096] Specifically, a softmax operation is performed on the masked attention scores to obtain the attention weights for each token. The attention scores corresponding to positions with a value of 0 in the mask (invalid padding positions) are set to a very small value (usually negative infinity). After the softmax function, the attention weights of these invalid padding positions approach 0, and are therefore ignored by the natural language model. Meanwhile, positions with a value of 1 in the mask are allowed to have their attention weights calculated normally, allowing the model to focus on the information corresponding to these positions.

[0097] After obtaining the attention weights corresponding to each position, the information corresponding to each word is integrated according to its attention weight to obtain ring attention. Ring attention is an attention representation that includes ring structure information. This representation comprehensively considers the degree of attention each position in the sequence pays to other positions, as well as the additional information flow brought by the ring structure.

[0098] The technical solution of this embodiment converts the input data into an input sequence, and splices the input sequences according to the order of the input sequences to obtain a total sequence. The total sequence is segmented by parallel parameters to obtain at least two processing sequences. Different types of mask constructions are performed on each processing sequence according to the task type. The mask is combined with the sequence filling and segmentation method. This can not only significantly reduce the occupation of computing resources and improve the processing efficiency of input data, but also reduce the invalid filling part and reduce the waste of computing resources caused by processing the invalid filling part. When the task type is a causal task, pre-communication is performed to ensure the load balancing of each parallel device. Each processing sequence is processed in parallel by each parallel device, and the ring attention is calculated based on the mask matching each processing sequence and the attention score of each word in each processing sequence. The technical solution of this embodiment is more suitable for processing mixed input sequences of different lengths. Compared with the conventional ring attention calculation method, it saves more computing power, reduces computing power waste, and can ensure load balancing between parallel devices, making the data processing steps of the artificial intelligence model more efficient.

[0099] Example 3

[0100] Figure 7 This is a structural diagram of a data parallel processing device provided by the third embodiment of the present invention. Figure 7 As shown, the device includes:

[0101] A sequence splicing module 310 is used to determine at least two input sequences and splice the input sequences according to the order of the input sequences to obtain a total sequence;

[0102] Wherein, each input sequence includes at least one word unit;

[0103] A sequence splitting module 320 is configured to obtain a parallel parameter, and split the total sequence based on the parallel parameter and the input sequence to obtain at least two processing sequences, wherein the parallel parameter is determined at least according to the number of parallel devices;

[0104] The sequence parallel processing module 330 is used to process the processing sequence in parallel through at least two parallel devices to obtain ring attention.

[0105] The technical solution of the embodiments of the present invention converts input data into input sequences, concatenates the input sequences according to their order, obtains a total sequence, partitions the total sequence using parallel parameters to obtain at least two processing sequences, and processes the processing sequences in parallel using at least two parallel devices to achieve circular attention. This solves the problem of the prior art of padding other sequences based on the length of the longest sequence before performing mixed sequence parallel processing, which wastes computing resources and reduces processing speed. It reduces the amount of padding in the input sequence, reduces computing resource waste, and improves computing efficiency.

[0106] Based on the above embodiment, optionally, the sequence segmentation module 320 includes:

[0107] a processing sequence length determining unit, configured to determine the processing sequence length according to the total number of word units in the total sequence and the parallel parameter;

[0108] The total sequence segmentation unit is used to segment the total sequence according to the length of the processing sequence to obtain at least two processing sequences.

[0109] Based on the above embodiment, optionally, the processing sequence length determination unit is specifically configured to:

[0110] If the total number of tokens in the total sequence is an integer multiple of the parallel parameter, the value obtained by dividing the total number of tokens in the total sequence by the parallel parameter is used as the processing sequence length;

[0111] Otherwise, the processing sequence length is calculated using the following formula: len = ┌a┐, a = T / D, where len represents the processing sequence length, ┌ ┐ represents rounding up, T represents the total number of tokens in the total sequence, and D represents the parallel parameter.

[0112] Based on the above embodiment, optionally, the total sequence segmentation unit is specifically configured to:

[0113] If the total number of tokens in the total sequence is an integer multiple of the parallel parameter, the total sequence is split according to the length of the processing sequence;

[0114] Otherwise, the total sequence is divided according to the length of the processing sequence, and the end of the last processing sequence is padded so that the number of tokens in the last processing sequence is consistent with the length of the processing sequence.

[0115] Based on the above embodiment, optionally, the sequence parallel processing module 330 includes:

[0116] a task type determining unit, configured to obtain a task type matching the input data, wherein the task type includes a causal task and a non-causal task;

[0117] A mask determination unit, configured to construct a mask matching each processing sequence according to the task type and the input sequence;

[0118] an attention score determination unit, configured to perform parallel processing on each processing sequence using at least two parallel devices to determine an attention score for each word in each processing sequence;

[0119] A ring attention determination unit is used to calculate the ring attention based on the mask matching each processing sequence and the attention score of each word in each processing sequence.

[0120] Based on the above embodiment, optionally, if the task type is a causal task, the mask determination unit is specifically configured to:

[0121] Based on the unidirectional attention mechanism, a mask matching each processing sequence is constructed;

[0122] The attention score determination unit is specifically used to:

[0123] Based on the query vector, key vector, and value vector of each word corresponding to the unidirectional attention of each processing sequence, the attention score of each word is calculated.

[0124] Based on the above embodiment, optionally, if the task type is a non-causal task, the mask determination unit is specifically configured to:

[0125] Based on the bidirectional attention mechanism, a mask matching each processing sequence is constructed;

[0126] The attention score determination unit is specifically used to:

[0127] Based on the query vector, key vector, and value vector of each word corresponding to the bidirectional attention of each processing sequence, the attention score of each word is calculated.

[0128] Based on the above embodiment, optionally, the ring attention determination unit is specifically configured to:

[0129] Determine the attention weight of each word based on the mask matching each processing sequence and the attention score of each word;

[0130] According to the mask matching each processing sequence and the attention weight of each word, the attention weight of each word in each processing sequence is weighted and summed to obtain the ring attention.

[0131] Based on the above embodiment, optionally, the device further includes:

[0132] The pre-communication module is used to perform pre-communication with each parallel device, wherein the pre-communication includes: distributing the query vector, key vector and value vector of each word required for processing the causal task to each parallel device.

[0133] The data parallel processing device provided in the embodiment of the present invention can execute the data parallel processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0134] Example 4

[0135] Figure 8 A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0136] like Figure 8As shown, electronic device 10 includes at least one processor 11 and memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by the at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer programs stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of electronic device 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. An input / output (I / O) interface 15 is also connected to bus 14.

[0137] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0138] Processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any other suitable processor, controller, microcontroller, etc. Processor 11 executes the various methods and processes described above, such as the data parallel processing method.

[0139] In some embodiments, the data parallel processing method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data parallel processing method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the data parallel processing method in any other suitable manner (e.g., via firmware).

[0140] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0144] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0145] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0146] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0147] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data parallel processing method, characterized in that: include: Determining at least two input sequences according to at least one input data, and concatenating the input sequences according to the order of the input sequences to obtain a total sequence; Wherein, each input sequence includes at least one word unit; Obtaining a parallel parameter, and dividing the total sequence based on the parallel parameter to obtain at least two processing sequences, wherein the parallel parameter is determined at least according to the number of parallel devices; The processing sequence is processed in parallel by at least two parallel devices to obtain a circular attention, including: Acquire a task type that matches the input data, where the task type includes a causal task and a non-causal task; According to the task type and input sequence, a mask matching each processing sequence is constructed; Processing each processing sequence in parallel using at least two parallel devices to determine an attention score for each word in each processing sequence; Compute ring attention based on the mask matching each processing sequence and the attention score of each token in each processing sequence; If the task type is a causal task, a mask matching each processing sequence is constructed according to the task type, including: Based on the unidirectional attention mechanism, a mask matching each processing sequence is constructed; Determine the attention score for each token in each processing sequence, including: Calculate the attention score of each word based on the query vector, key vector, and value vector of each word corresponding to the unidirectional attention of each processing sequence; If the task type is a non-causal task, a mask matching each processing sequence is constructed according to the task type, including: Based on the bidirectional attention mechanism, a mask matching each processing sequence is constructed; Determine the attention score for each token in each processing sequence, including: Based on the query vector, key vector, and value vector of each word corresponding to the bidirectional attention of each processing sequence, the attention score of each word is calculated.

2. The method according to claim 1, characterized in that The total sequence is divided based on a parallel parameter to obtain at least two processing sequences, including: Determining the processing sequence length according to the total number of word units in the total sequence and the parallel parameter; The total sequence is divided according to the length of the processing sequence to obtain at least two processing sequences.

3. The method according to claim 2, characterized in that Determining the processing sequence length according to the total number of word units in the total sequence and the parallel parameter includes: If the total number of tokens in the total sequence is an integer multiple of the parallel parameter, the value obtained by dividing the total number of tokens in the total sequence by the parallel parameter is used as the processing sequence length; Otherwise, the processing sequence length is calculated using the following formula: len = ┌a┐, a = T / D, where len represents the processing sequence length, ┌ ┐ represents rounding up, T represents the total number of tokens in the total sequence, and D represents the parallel parameter.

4. The method according to claim 2, characterized in that According to the length of the processing sequence, the total sequence is divided into at least two processing sequences, including: If the total number of tokens in the total sequence is an integer multiple of the parallel parameter, the total sequence is split according to the length of the processing sequence; Otherwise, the total sequence is divided according to the length of the processing sequence, and the end of the last processing sequence is padded so that the number of tokens in the last processing sequence is consistent with the length of the processing sequence.

5. The method according to claim 1, wherein Based on the mask matching each processing sequence and the attention score of each token in each processing sequence, a ring attention is calculated, including: Determine the attention weight of each word based on the mask matching each processing sequence and the attention score of each word; According to the mask matching each processing sequence and the attention weight of each word, the attention weight of each word in each processing sequence is weighted and summed to obtain the ring attention.

6. The method according to claim 1, characterized in that If the task type is a causal task, before processing each processing sequence in parallel by at least two parallel devices, the method further includes: Performing pre-communication on each parallel device, wherein the pre-communication includes: distributing the query vector, key vector, and value vector of each word required for processing the causal task to each parallel device.

7. A data parallel processing device, characterized in that: include: A sequence splicing module is used to determine at least two input sequences based on at least one input data, and splice the input sequences according to the order of the input sequences to obtain a total sequence; Wherein, each input sequence includes at least one word unit; a sequence splitting module, configured to obtain a parallel parameter, and split the total sequence based on the parallel parameter and the input sequence to obtain at least two processing sequences, wherein the parallel parameter is determined at least according to the number of parallel devices; A sequence parallel processing module, configured to process the processing sequence in parallel through at least two parallel devices to obtain a circular attention; Sequence parallel processing module, including: a task type determining unit, configured to obtain a task type matching the input data, wherein the task type includes a causal task and a non-causal task; A mask determination unit, configured to construct a mask matching each processing sequence according to the task type and the input sequence; an attention score determination unit, configured to perform parallel processing on each processing sequence using at least two parallel devices to determine an attention score for each word in each processing sequence; a ring attention determination unit, configured to calculate the ring attention based on the mask matched to each processing sequence and the attention score of each word in each processing sequence; If the task type is a causal task, the mask determination unit is specifically configured to: construct a mask matching each processing sequence based on a unidirectional attention mechanism; the attention score determination unit is specifically configured to: calculate the attention score of each word-unit based on the query vector, key vector, and value vector of each word-unit corresponding to the unidirectional attention of each processing sequence; If the task type is a non-causal task, the mask determination unit is specifically used to: construct a mask that matches each processing sequence based on the bidirectional attention mechanism; the attention score determination unit is specifically used to: calculate the attention score of each word based on the query vector, key vector and value vector of each word corresponding to the bidirectional attention of each processing sequence.

Images