Text data processing method and device, computer equipment and storage medium

Abstract

The invention relates to a text data processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a text data set, respectively distributing each piece of text data in the text data set to at least one natural language model expert network in a plurality of natural language model expert networks, determining the text data distributed to the natural language model expert network for each natural language model expert network, the natural language model expert network and the text data assigned to the natural language model expert network are used as training groups. And performing parallel operation on the training groups corresponding to the plurality of natural language model expert networks to obtain a data operation result corresponding to each training group, and determining a text data processing result of each piece of text data according to the data operation result corresponding to each training group. By adopting the method, the tedious operation steps in the data processing process and the operation waiting time for each piece of text data can be reduced, and the text data processing efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a text data processing method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the development of artificial intelligence technology and various neural network models, the promotion and application of neural network models in fields such as natural language processing, image or speech recognition, etc. in practical application process, in order to meet the actual needs in different fields or application scenarios, it is necessary to continuously train and optimize the model, and usually the more complex the model structure is, the higher the model prediction accuracy is, but the network parameter size and the amount of calculation it has also increases, therefore, a large amount of resources are needed for large model training to obtain an applicable network model.

[0003] In the traditional technology, in order to reduce the resource consumption in the process of large model training, a method of applying Moe model (Mixture Of Experts models) with sparsity for data processing is appeared. However, due to the complex structure of Moe model, there are many experts (Experts), in the process of data processing, each expert needs to be calculated according to the token (such as word or phrase) acting on the expert, and there is still a problem of large amount of calculation and long operation waiting time, and the corresponding data processing efficiency also needs to be improved. SUMMARY

[0004] Therefore, it is necessary to provide a text data processing method, device, computer equipment, computer readable storage medium and computer program product capable of reducing the operation amount in the process of data processing and the waiting time for different to-be-processed data, so as to improve the data processing efficiency.

[0005] In a first aspect, the present application provides a text data processing method, comprising:

[0006] obtaining a text data set, and respectively assigning each text data in the text data set to at least one natural language model expert network in a plurality of natural language model expert networks;

[0007] determining, for each natural language model expert network, the text data assigned to the natural language model expert network, taking the natural language model expert network and the text data assigned to the natural language model expert network as a training group;

[0008] performing parallel operation on the training group corresponding to each of the plurality of natural language model expert networks to obtain a data operation result corresponding to each training group;

[0009] determine a text data processing result of each of the text data according to the data operation result corresponding to each of the training groups.

[0010] In a second aspect, the present application provides a text data processing apparatus, comprising:

[0011] a text data distribution module configured to obtain a text data set and distribute each of the text data in the text data set to at least one natural language model expert network in a plurality of natural language model expert networks respectively;

[0012] a training group obtaining module configured to, for each of the natural language model expert networks, determine the text data distributed to the natural language model expert network, and take the natural language model expert network and the text data distributed to the natural language model expert network as a training group;

[0013] a parallel operation module configured to perform parallel operation on the training group corresponding to each of the natural language model expert networks respectively, to obtain a data operation result corresponding to each of the training groups;

[0014] a text data processing result obtaining module configured to determine a text data processing result of each of the text data according to the data operation result corresponding to each of the training groups.

[0015] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0016] obtain a text data set and distribute each of the text data in the text data set to at least one natural language model expert network in a plurality of natural language model expert networks respectively;

[0017] for each of the natural language model expert networks, determine the text data distributed to the natural language model expert network, and take the natural language model expert network and the text data distributed to the natural language model expert network as a training group;

[0018] perform parallel operation on the training group corresponding to each of the natural language model expert networks respectively, to obtain a data operation result corresponding to each of the training groups;

[0019] determine a text data processing result of each of the text data according to the data operation result corresponding to each of the training groups.

[0020] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0021] obtain a text data set, and assign each text data in the text data set to at least one natural language model expert network in a plurality of natural language model expert networks respectively;

[0022] For each natural language model expert network, determine the text data assigned to the natural language model expert network, and take the natural language model expert network and the text data assigned to the natural language model expert network as a training group;

[0023] perform parallel operation on the training group corresponding to each of the plurality of natural language model expert networks respectively to obtain a data operation result corresponding to each training group;

[0024] determine a text data processing result of each text data according to the data operation result corresponding to each training group.

[0025] In a fifth aspect, the present application further provides a computer program product comprising a computer program which, when executed by a processor, implements the following steps:

[0026] obtain a text data set, and assign each text data in the text data set to at least one natural language model expert network in a plurality of natural language model expert networks respectively;

[0027] For each natural language model expert network, determine the text data assigned to the natural language model expert network, and take the natural language model expert network and the text data assigned to the natural language model expert network as a training group;

[0028] perform parallel operation on the training group corresponding to each of the plurality of natural language model expert networks respectively to obtain a data operation result corresponding to each training group;

[0029] determine a text data processing result of each text data according to the data operation result corresponding to each training group.

[0030] In the text data processing method, the text data set is obtained, each text data in the text data set is respectively allocated to at least one natural language model expert network in the plurality of natural language model expert networks, and for each natural language model expert network, text data allocated to the corresponding natural language model expert network is further determined, so that the natural language model expert network and the text data allocated to the natural language model expert network are used as a training group, thereby subsequent parallel operation can be performed on the plurality of training groups, and the waiting time of serial operation in the data processing process is reduced. Further, by performing parallel operation on the training groups corresponding to the plurality of natural language model expert networks respectively, the data operation results corresponding to each training group can be obtained at the same time, so that the text data processing result of each text data is determined according to the data operation result corresponding to each training group. Since serial operation is not required for the text data according to each natural language model expert network selected by the text data, the cumbersome operation steps in the data processing process and the operation waiting time for each text data are reduced, and the text data processing efficiency is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other related drawings can be obtained by those skilled in the art without creative labor.

[0032] Figure 1 An application environment diagram of a text data processing method of an embodiment;

[0033] Figure 2 A flowchart of a text data processing method of an embodiment;

[0034] Figure 3 A process diagram of dividing text data to a natural language model expert network in an embodiment;

[0035] Figure 4 A process diagram of rearranging and grouping a natural language model expert network and text data in an embodiment;

[0036] Figure 5 A process diagram of rearranging and grouping a natural language model expert network and text data in another embodiment;

[0037] Figure 6 A diagram of performing parallel operation on training groups corresponding to a plurality of natural language model expert networks in an embodiment;

[0038] Figure 7 A flowchart of a process for determining a text data processing result of each text data in an embodiment;

[0039] Figure 8 A flowchart of a process for determining a text data processing result of each text data according to a data operation result of a plurality of training groups in an embodiment;

[0040] Figure 9 A flowchart of a process for performing an operation on each training group of a plurality of training groups in an embodiment;

[0041] Figure 10 A schematic diagram of a training text matrix and an expert network matrix constructed for each training group corresponding to each natural language model expert network in an embodiment;

[0042] Figure 11 A flowchart of a process for a text data processing method in another embodiment;

[0043] Figure 12 A block diagram of a structure of a text data processing device in an embodiment;

[0044] Figure 13 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0045] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0046] The text data processing method provided by the embodiments of the present application relates to artificial intelligence technology, and can be applied to intelligent transportation, map navigation, auxiliary driving, network media and various other scenarios, and can be specifically applied to, for example, Figure 1The application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, portable wearable devices, and aircrafts, etc. The Internet of Things device can be a smart speaker, a smart TV, a smart air conditioner, a smart vehicle device, and a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, and a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, and smart glasses, etc. The server 104 can be a standalone physical broadcasting server, or a broadcasting server cluster composed of multiple physical broadcasting servers, or a cloud broadcasting server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms, etc. Among them, the terminal 102 and the server 104 can be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiments of the present application.

[0047] Among them, the terminal 102 and the server 104 can be individually used to execute the text data processing method provided in the embodiments of the present application, and the terminal 102 and the server 104 can also cooperatively execute the text data processing method provided in the embodiments of the present application. For example, taking the terminal 102 and the server 104 cooperatively executing the text data processing method provided in the embodiments of the present application as an example, the server 104 can specifically obtain a text data set from the terminal 102, and respectively assign each text data in the obtained text data set to at least one natural language model expert network in a plurality of natural language model expert networks. Further, the server 104 determines, for each natural language model expert network, the text data assigned to the natural language model expert network, so as to take the natural language model expert network and the text data assigned to the natural language model expert network as a training group. Among them, the server 104 specifically performs parallel operation on the training groups respectively corresponding to the plurality of natural language model expert networks, so as to simultaneously obtain data operation results corresponding to each training group, thereby determining the text data processing result of each text data according to the data operation result corresponding to each training group. Among them, the server 104 can feed back the obtained text data processing result of each text data to the terminal 102 for display and further application.

[0048] In one exemplary embodiment, as Figure 2As shown, a text data processing method is provided, and the method is applied to Figure 1 The server 104 in the method is taken as an example for illustration, and the method includes the following steps S202 to S208. Wherein:

[0049] In step S202, a text data set is obtained, and each text data in the text data set is respectively assigned to at least one natural language model expert network in a plurality of natural language model expert networks.

[0050] Specifically, by obtaining a text data set for inputting a natural language model, and obtaining routing decision data corresponding to a natural language model gating network, each text data in the text data set is respectively assigned to at least one natural language model expert network in a plurality of natural language model expert networks according to the routing decision data.

[0051] Wherein, the text data set includes a plurality of text data, and the text data can be words, phrases, characters, or numbers, i.e. different types of text such as Chinese or English. The natural language model for processing text data can be a mixture of expert models (Moe model), which includes a gating network and a plurality of expert networks. The gating network is used to determine which token (token, which can be text data such as words or phrases) is assigned to which expert network. The expert network can perform actual data operation according to the token acting on the expert network to obtain the corresponding data operation result.

[0052] Optionally, the expert network can be a feedforward neural network (FFN), which can also be understood as a multilayer perceptron (MLP) that can map multiple input data sets to a single output data set. The FFN uses multiple nonlinear processing layers to extract and convert hierarchical information features of input data, and the output of each layer is used as the input of the next layer until the final output is obtained.

[0053] Exemplarily, the gating network can also be understood as a gated fusion network (i.e., Gated Fusion Network, GFN), which can fuse information from multiple sources by learning a gating mechanism that controls the way information is integrated. The gating network includes various paths that process different types of data (which can be understood as paths that allocate data to different expert networks), and the gating mechanism determines how much weight each path contributes to the final output. By learning the gating mechanism, the gating network can dynamically adjust the expert networks to which different data is allocated and the network weights occupied by the allocated expert networks.

[0054] Further, for a natural language model, specifically by obtaining routing decision data corresponding to the gating network of the natural language model (which can also be understood as the gating mechanism learned by the gating network), it is determined which expert network or combination of expert networks is suitable for processing the text data according to the routing decision data, such as specifically determining that the combination of natural language model expert network 1 and natural language model expert network 3 is suitable for processing text data token0 according to the routing decision data, and specifically allocating text data token0 to natural language model expert network 1 and natural language model expert network 3, and determining the respective network weights of natural language model expert network 1 and natural language model expert network 3.

[0055] Among them, since different expert networks can focus on different regions or aspects of data, and the gating mechanism learned by the gating network can be used to determine how to combine the outputs of different expert networks to produce the final output, so according to the routing decision data (i.e., the gating mechanism learned by the gating network), each text data in the text data set can be dynamically divided, and each text data in the text data set can be allocated to at least one natural language model expert network in the plurality of natural language model expert networks.

[0056] It can be understood that each text data in the text data set can be allocated to one natural language model expert network, or can be allocated to multiple natural language model expert networks at the same time. Among them, if a certain text data is allocated to multiple natural language model expert networks at the same time, the network weight of each natural language model expert network needs to be considered when determining the text data processing result of the text data, that is, the text data processing result corresponding to the text data can be obtained by weighted summation.

[0057] In an exemplary embodiment, as shown in Figure 3 , a process of dividing text data to natural language model expert networks is provided, referring to Figure 3It can be seen that in the natural language model (such as the hybrid expert model), a gating network and a plurality of expert networks are included. Among them, the text data token0 "AAA" is allocated to the expert network 1 of the natural language model, and the text data token1 "BBB" is allocated to the expert network 0 of the natural language model.

[0058] Among them, referring to Figure 3 It can be seen that the text data token0 "AAA" is allocated to the expert network 1 of the natural language model to obtain the corresponding output data y1, and the text data token1 "BBB" is allocated to the expert network 0 of the natural language model to obtain the corresponding output data y2.

[0059] In an exemplary embodiment, after obtaining the text data set, further comprising:

[0060] Extracting the respective state tensor data corresponding to each text data in the text data set, and performing dimension adjustment on the respective state tensor data corresponding to each text data to obtain dimension-adjusted text data. Among them, the dimension-adjusted text data is used for data operation with the allocated natural language model expert network.

[0061] Specifically, the respective state tensor data corresponding to each text data in the text data set, such as the hidden state tensor (i.e. hidden_states) of the text data, is extracted, and the shape of the hidden state tensor is obtained. For example, the shape of the hidden state tensor before adjustment is three-dimensional, and the dimension adjustment is performed on the respective state tensor data corresponding to each text data, such as adjusting the three-dimensional state tensor data (i.e. hidden state tensor) to two-dimensional, to obtain the dimension-adjusted text data. Among them, by adjusting the dimension of the state tensor data corresponding to the text data, such as adjusting from three-dimensional to two-dimensional, the amount of data required for processing by the model can be reduced, so as to reduce the resource occupation in the data processing process and improve the data processing efficiency.

[0062] Among them, the dimension-adjusted text data is used for data operation with the allocated natural language model expert network, that is, according to the routing decision data, the dimension-adjusted each text data in the text data set is dynamically divided, each text data is allocated to at least one natural language model expert network in the plurality of natural language model expert networks, so as to perform data operation according to the text data and the natural language model expert network to which it is allocated, to obtain the data operation result.

[0063] Exemplarily, the shape of the hidden state tensor before adjustment is three-dimensional, including

batch_size, sequence_length, hidden_dim

batch_size, sequence_length, hidden_dim

batch_size*sequence_length, hidden_dim

[0064] In step S204, for each natural language model expert network, the text data allocated to the natural language model expert network is determined, and the natural language model expert network and the text data allocated to the natural language model expert network are taken as a training group.

[0065] Specifically, for each natural language model expert network, the text data allocated to the natural language model expert network is determined, i.e., one natural language model expert network can be allocated one or more text data, and the same text data can also be allocated to one or more natural language model expert networks. After determining the text data allocated to the natural language model expert network, the natural language model expert network and the text data allocated to the natural language model expert network are taken as a training group.

[0066] It can be understood that each natural language model expert network corresponds to a training group, i.e., a training group specifically includes a natural language model expert network and at least one text data allocated to the natural language model expert network.

[0067] In an exemplary embodiment, as shown in Figure 4 , a process of rearranging and grouping natural language model expert networks and text data is provided, and as shown in Figure 4 , it can be known that the natural language model gating network can determine how each text data in the input text data set is allocated to different natural language model expert networks according to the corresponding routing decision data. As shown in Figure 4 , it can be known that the text data token0 and the text data token1 are allocated to the natural language model expert network 1, the text data token2 to the text data token4 are allocated to the natural language model expert network 0, and the text data token5 is allocated to the natural language model expert network 2.

[0068] Specifically, referring to Figure 4 It can be seen that after determining the text data allocated to each natural language model expert network, the multiple natural language model expert networks and the multiple text data are rearranged according to the network identifiers of the natural language model expert networks (such as expert id, including expert0, expert1, expert2, etc.), and the text data allocated to expert0 (i.e. natural language model expert network 0) is selected, the text data allocated to expert1 (i.e. natural language model expert network 1) is selected, and the text data allocated to expert2 (i.e. natural language model expert network 2) is selected in the order of the network identifiers of the natural language model expert networks (such as expert0→expert1→expert2).

[0069] Further, referring to Figure 4 It can be seen that after rearranging and grouping the multiple natural language model expert networks and the multiple text data, multiple training groups based on the network identifiers of the natural language model expert networks can be obtained, such as a first training group corresponding to expert0, a second training group corresponding to expert1, and a third training group corresponding to expert2. Among them, the first training group, the second training group, and the third training group can be added to the same matrix operation core, and different training groups in the same matrix operation core use parallel operation to simultaneously obtain data operation results corresponding to different training groups.

[0070] Among them, referring to Figure 4 It can be seen that after analyzing and subsequent operation of the data operation results corresponding to different training groups, output data corresponding to different text data respectively (i.e. text data processing results) can be obtained, such as output data corresponding to token2, token3, token4, token0, token1 and token5 in the order of expert0→expert1→expert2.

[0071] In an exemplary embodiment, as Figure 5 It can be seen that another process of rearranging and grouping natural language model expert networks and text data is provided, referring to Figure 5 It can be seen that the text data input into the natural language model includes token0, token1, token2, token3 and token4, and the natural language model expert network includes expert0, expert1, expert2 and expert3.

[0072] Specifically, after determining that each text data is assigned to two natural language model expert networks, such as determining that text data token0 is assigned to natural language model expert network expert1 and expert3, text data token1 is assigned to natural language model expert network expert0 and expert3, text data token2 is assigned to natural language model expert network expert0 and expert1, text data token3 is assigned to natural language model expert network expert1 and expert3, and text data token4 is assigned to natural language model expert network expert0 and expert1, the multiple natural language model expert networks and the multiple text data are rearranged according to the network identifier of the natural language model expert network (such as expert id, including expert0, expert1, expert2, expert3, etc.), to obtain multiple training groups according to the network identifier of the natural language model expert network.

[0073] Specifically, according to the order of the network identifier of the natural language model expert network (such as the order of expert0→expert1→expert2→expert3), the text data assigned to expert0 (i.e. natural language model expert network 0) (including token1, token2, and token4), the text data assigned to expert1 (i.e. natural language model expert network 1) (including token0, token2, token3, and token4), the text data assigned to expert2 (i.e. natural language model expert network 2) (empty), and the text data assigned to expert3 (i.e. natural language model expert network 3) (including token1, token2, and token3) are selected respectively.

[0074] Further, referring to Figure 5 It can be seen that after rearranging and grouping the multiple natural language model expert networks and the multiple text data, multiple training groups according to the network identifier of the natural language model expert network can be obtained, such as a first training group corresponding to expert0, a second training group corresponding to expert1, and a fourth training group corresponding to expert3. Among them, the first training group, the second training group, and the fourth training group can be added to the same matrix operation core, and different training groups in the same matrix operation core use parallel operation to obtain data operation results corresponding to different training groups at the same time.

[0075] In step S206, parallel operation is performed on the training groups corresponding to the plurality of natural language model expert networks respectively to obtain data operation results corresponding to each training group.

[0076] Specifically, when performing parallel operation on the training groups corresponding to the plurality of natural language model expert networks respectively, a matrix operation kernel can be constructed according to the training groups corresponding to the plurality of natural language model expert networks respectively, and parallel operation is performed on the plurality of training groups based on the matrix operation kernel to obtain data operation results corresponding to each training group.

[0077] The matrix operation kernel can be constructed according to the training groups corresponding to the plurality of natural language model expert networks respectively, that is, the training groups corresponding to the plurality of natural language model expert networks respectively are added to the same matrix operation kernel, and parallel operation is performed on the plurality of training groups based on the matrix operation kernel, that is, different training groups in the same matrix operation kernel are operated in parallel to obtain data operation results corresponding to the different training groups respectively.

[0078] When parallel operation is performed on the training group corresponding to the natural language model expert network, MLP calculation is performed according to the text data allocated to the natural language model expert network in the training group, that is, calculation is performed to map a plurality of input text data to a single output data set. The MLP calculation for each natural language model expert network includes operations and calculations on three linear layers. In a traditional operation mode, if a natural language model has 60 expert networks, 60 operations need to be performed on the 60 expert networks in sequence, and each expert network performs actual MLP calculation according to the token (input text data) acting on the expert, which is equivalent to serial matrix operation of 60*3. The matrix operation needs to be performed in sequence according to the expert network to which the text data is allocated, and there are still problems of complicated operation steps and long operation waiting time for each text data.

[0079] Further, in the text data processing method, a natural language model expert network and text data allocated to the natural language model expert network are first taken as a training group, a plurality of training groups are obtained, and then parallel operation is performed on the training groups corresponding to the plurality of natural language model expert networks respectively to obtain data operation results corresponding to each training group. Therefore, serial operation is not needed for the text data according to each natural language model expert network selected by the text data, the complicated operation steps in the data processing process and the operation waiting time for each text data are reduced, and the text data processing efficiency is further improved.

[0080] In an exemplary embodiment, as shown in FIG. 2, the text data processing method includes the following steps. Figure 6As shown, a schematic of parallel operation of respective training groups of the plurality of natural language model expert networks is provided, with reference to Figure 6 It can be seen that, after rearranging and grouping the plurality of natural language model expert networks and the plurality of text data, a plurality of training groups based on the network identifiers of the natural language model expert networks are obtained, such as a first training group corresponding to expert0 (including the natural language model expert network expert0, the text data token1, token2, and token4), a second training group corresponding to expert1 (including the natural language model expert network expert1, the text data token0, token2, token3, and token4), and a fourth training group corresponding to expert3 (including the natural language model expert network expert3, the text data token1, token2, and token3). By adding the first training group, the second training group, and the fourth training group to the same matrix operation core, parallel operation processing is performed based on the matrix operation core, and different training groups in the matrix operation core are operated in parallel, and data operation results corresponding to different training groups are obtained.

[0081] In step S208, the text data processing result of each text data is determined according to the data operation result corresponding to each training group.

[0082] Among them, since one natural language model expert network can be allocated one or more text data, and the same text data can be allocated to one or more natural language model expert networks, when the data operation result corresponding to each training group is analyzed or further operated to obtain the text data processing result of each text data, the number of natural language model expert networks to which each text data is allocated needs to be determined.

[0083] Specifically, for each text data, if the number of natural language model expert networks to which a certain text data is allocated is single, then the data operation result of the training group corresponding to the natural language model expert network to which the text data is allocated is obtained, and the sub-operation result corresponding to the text data is extracted from the data operation result of the training group corresponding to the natural language model expert network to which the text data is allocated, and the sub-operation result is taken as the text data processing result of the corresponding text data.

[0084] Further, for each text data, if the number of natural language model expert networks to which the text data is assigned is multiple, the data operation results of the training groups corresponding to the natural language model expert networks to which the text data is assigned are specifically required to be obtained, and the sub-operation results corresponding to the text data are extracted from the data operation results of the training groups corresponding to each of the natural language model expert networks to which the text data is assigned in turn, and the network weights of the respective natural language model expert networks to which the text data is assigned are determined, so as to perform weighted calculation according to the multiple sub-operation results corresponding to the text data and the network weights of the respective natural language model expert networks, and obtain the text data processing result corresponding to the text data.

[0085] In the above text data processing method, by obtaining the text data set, each text data in the text data set is respectively assigned to at least one natural language model expert network in the multiple natural language model expert networks, and for each natural language model expert network, the text data assigned to the corresponding natural language model expert network is further determined, so that the natural language model expert network and the text data assigned to the natural language model expert network are taken as a training group, thereby subsequent parallel operation can be performed on the multiple training groups, and the waiting time of serial operation in the data processing process is reduced. Further, by performing parallel operation on the training groups corresponding to the respective natural language model expert networks, the data operation results corresponding to each training group can be obtained at the same time, so that the text data processing result of each text data is determined according to the data operation results corresponding to each training group. Since it is not necessary to perform serial operation on the text data according to each natural language model expert network selected by the text data, the cumbersome operation steps in the data processing process and the operation waiting time for each text data are reduced, and the text data processing efficiency is further improved.

[0086] In an exemplary embodiment, as shown in Figure 7 The step of determining the text data processing result of each text data, i.e., the step of determining the text data processing result of each text data according to the data operation results corresponding to each training group, specifically includes steps S702 to S706. Wherein:

[0087] Step S702, for each text data, determining the natural language model expert network to which the text data is assigned.

[0088] Specifically, one natural language model expert network can be assigned one or more text data, and the same text data can be assigned to one or more natural language model expert networks. When it is required to determine the text data processing result of each text data, the natural language model expert network to which each text data is assigned is required to be determined first, and the number of natural language model expert networks to which the text data is assigned is required to be counted.

[0089] For example, the text data input into the natural language model specifically includes token0, token1, token2, token3 and token4, and the natural language model expert network specifically includes expert0, expert1, expert2 and expert3. For example, it is determined that the text data token0 is assigned to the natural language model expert network expert1 and expert3, the text data token1 is assigned to the natural language model expert network expert0 and expert3, the text data token2 is assigned to the natural language model expert network expert0 and expert1, the text data token3 is assigned to the natural language model expert network expert1 and expert3, and the text data token4 is assigned to the natural language model expert network expert0 and expert1.

[0090] In step S704, a sub-computation result corresponding to the text data is extracted from the data computation result of the training group corresponding to the natural language model expert network to which the text data is assigned.

[0091] Specifically, for each text data, the data computation result of the training group corresponding to the natural language model expert network to which the text data is assigned is obtained. For example, for the text data token0, it is assigned to the natural language model expert network expert1 and expert3, and the data computation result of the training group corresponding to the natural language model expert network expert1 and expert3 is obtained, and a sub-computation result corresponding to the text data token0 is extracted from the data computation result of the training group corresponding to the natural language model expert network to which the text data is assigned, i.e. from the data computation result of the training group corresponding to the natural language model expert network expert1 and expert3, including the sub-computation result extracted from the data computation result of the second training group corresponding to the natural language model expert network expert1 and the sub-computation result extracted from the data computation result of the fourth training group corresponding to the natural language model expert network expert3.

[0092] For example, if the text data is only assigned to one natural language model expert network, for example, the text data input into the natural language model also includes text data token5, and the text data token5 is assigned to the natural language model expert network expert2, and the data computation result of the training group corresponding to the natural language model expert network expert2 is obtained, and a sub-computation result corresponding to the text data token5 is extracted from the data computation result of the third training group corresponding to the natural language model expert network expert2.

[0093] In step S706, the text data processing result corresponding to the text data is obtained according to the sub-operation result corresponding to the text data.

[0094] In the determination of the text data processing result of each text data, the natural language model expert network to which each text data is allocated is determined, and the number of the natural language model expert networks to which the text data is allocated is counted. That is, one text data can be allocated to one or more natural language model expert networks.

[0095] Specifically, for a certain text data, if it is determined that the number of the natural language model expert networks to which the text data is allocated is one, the sub-operation result corresponding to the text data is extracted from the data operation result of the training set corresponding to the natural language model expert network to which the text data is allocated, and the extracted sub-operation result is taken as the text data processing result of the text data.

[0096] For example, if the text data token5 is allocated to the natural language model expert network expert2, the data operation result of the training set corresponding to the natural language model expert network expert2 is obtained, and the sub-operation result corresponding to the text data token5 is extracted from the data operation result of the third training set corresponding to the natural language model expert network expert2, and the extracted sub-operation result is taken as the text data processing result of the text data token5.

[0097] In one example embodiment, the text data processing result corresponding to the text data is obtained according to the sub-operation result corresponding to the text data, including:

[0098] If the number of the natural language model expert networks to which the text data is allocated is multiple, the network weights of the multiple natural language model expert networks are determined, and the weighted calculation is performed according to the multiple sub-operation results corresponding to the text data and the network weights of the multiple natural language model expert networks, to obtain the text data processing result corresponding to the text data.

[0099] Specifically, for a certain text data, if it is determined that the number of the natural language model expert networks to which the text data is allocated is multiple, the network weights of the multiple natural language model expert networks to which the text data is allocated are determined respectively.

[0100] The network weights of each natural language model expert network in the natural language model, such as the hybrid expert model, are specifically determined according to the gating network in the hybrid expert model. Specifically, the gating network can fuse information from multiple sources through a learned gating mechanism of the information integration mode. The gating network includes various paths (which can be understood as paths for allocating data to different expert networks) for processing different types of data, and the gating mechanism determines how much weight each path contributes to the final output. That is, by learning the gating mechanism, the gating network can dynamically adjust the expert networks to which different data is allocated and the network weights of the allocated expert networks.

[0101] Further, for a certain text data, after determining the network weights of each of the multiple natural language model expert networks to which the text data is allocated, weighted calculation is performed according to the multiple sub-computation results respectively extracted from the data computation results of the training set corresponding to the allocated natural language model expert networks and the network weights of the multiple natural language model expert networks, to obtain a text data processing result corresponding to the text data.

[0102] For example, for text data token0, it is determined that the text data token0 is allocated to natural language model expert networks expert1 and expert3, the network weight of the natural language model expert network expert1 is 0.3, and the network weight of the natural language model expert network expert3 is 0.7. When determining the text data processing result corresponding to the text data token0, the sub-computation results corresponding to the text data token0 are extracted from the data computation results of the training set corresponding to the natural language model expert networks expert1 and expert3, including the sub-computation result token0-2 extracted from the data computation result of the second training set corresponding to the natural language model expert network expert1 and the sub-computation result token0-4 extracted from the data computation result of the fourth training set corresponding to the natural language model expert network expert3. Then, weighted calculation is performed according to the network weight of the natural language model expert network expert1, the sub-computation result token0-2, the network weight of the natural language model expert network expert3, and the sub-computation result token0-4, to obtain the text data processing result corresponding to the text data token0.

[0103] Optionally, the text data processing result of the text data token0 = the network weight of the natural language model expert network expert1 * the sub-operation result token0-2 + the network weight of the natural language model expert network expert3 * the sub-operation result token0-4 = 0.3 * the sub-operation result token0-2 + 0.7 * the sub-operation result token0-4.

[0104] In one exemplary embodiment, as shown in FIG. 1, a process diagram for determining the text data processing result of each text data with respect to the data operation results of a plurality of training groups is provided, referring to FIG. 2. Figure 8 As shown, the process diagram specifically includes a first training group corresponding to expert0 (including the natural language model expert network expert0, text data token1, token2, and token4), a second training group corresponding to expert1 (including the natural language model expert network expert1, text data token0, token2, token3, and token4), and a fourth training group corresponding to expert3 (including the natural language model expert network expert3, text data token1, token2, and token3). Figure 8 Specifically, referring to FIG. 2, in the first training group, for the text data token1, token2, and token4, the network weight of the natural language model expert network expert0 is 0.2, 0.9, and 0.7, respectively. In the second training group, for the text data token0, token2, token3, and token4, the network weight of the natural language model expert network expert1 is 0.3, 0.1, 0.4, and 0.3, respectively. In the fourth training group, for the text data token1, token2, and token3, the network weight of the natural language model expert network expert3 is 0.7, 0.8, and 0.6, respectively.

[0105] Figure 8 Further, referring to FIG. 3, the process diagram for determining the text data processing result of each text data with respect to the data operation results of a plurality of training groups is provided, referring to FIG. 4.

[0106] As shown, the process diagram specifically includes a first training group corresponding to expert0 (including the natural language model expert network expert0, text data token1, token2, and token4), a second training group corresponding to expert1 (including the natural language model expert network expert1, text data token0, token2, token3, and token4), and a fourth training group corresponding to expert3 (including the natural language model expert network expert3, text data token1, token2, and token3). Figure 8 ​It can be seen that, for the text data token0, the natural language model expert network to which it is allocated is the natural language model expert network expert1 and expert3, and the network weights corresponding to the natural language model expert network expert1 and expert3 are 0.3 and 0.7 respectively. Similarly, for the text data token1, the natural language model expert network to which it is allocated is the natural language model expert network expert0 and expert3, and the network weights corresponding to the natural language model expert network expert0 and expert3 are 0.2 and 0.8 respectively. For the text data token2, the natural language model expert network to which it is allocated is the natural language model expert network expert0 and expert1, and the network weights corresponding to the natural language model expert network expert0 and expert1 are 0.9 and 0.1 respectively. For the text data token3, the natural language model expert network to which it is allocated is the natural language model expert network expert1 and expert3, and the network weights corresponding to the natural language model expert network expert1 and expert3 are 0.4 and 0.6 respectively. For the text data token4, the natural language model expert network to which it is allocated is the natural language model expert network expert0 and expert1, and the network weights corresponding to the natural language model expert network expert0 and expert1 are 0.7 and 0.3 respectively.

[0107] For example, for the text data token0, the weighted calculation is performed according to the network weights 0.3 and 0.7 corresponding to the natural language model expert network expert1 and expert3, the sub-operation result token0-2 extracted from the data operation result of the second training group corresponding to the natural language model expert network expert1, and the sub-operation result token0-4 extracted from the data operation result of the fourth training group corresponding to the natural language model expert network expert3, to obtain the text data processing result corresponding to the text data token0.

[0108] Similarly, for the text data token1, token2, token3, and token4, the weighted calculation is also performed according to the network weights corresponding to the natural language model expert network to which the corresponding text data is allocated, and the sub-operation result extracted from the data operation result of the training group corresponding to the natural language model expert network to which the corresponding text data is allocated, to obtain the text data processing result corresponding to the corresponding text data token1, token2, token3, and token4 respectively.

[0109] In this embodiment, for each text data, the natural language model expert network to which the text data is assigned is determined, and the sub-operation result corresponding to the text data is extracted from the data operation result of the training group corresponding to the assigned natural language model expert network, so as to obtain the text data processing result corresponding to the text data according to the sub-operation result corresponding to the text data. After parallel operation based on the respective training groups of at least one natural language model expert network, the obtained data operation result of the training group is parsed and further processed, the text data processing result corresponding to each text data can be accurately determined. Since it is not necessary to perform serial operation on each text data according to each natural language model expert network selected by the text data, the tedious operation steps in the data processing process and the operation waiting time for each text data can be reduced, thereby reducing the resource consumption in the data processing process and improving the data processing efficiency for the text data.

[0110] In one exemplary embodiment, as shown in Figure 9 the operation manner for each training group in the plurality of training groups includes the following steps S902 to S904, wherein:

[0111] Step S902, for each training group in the plurality of training groups, a training text matrix is constructed according to the text data in the training group.

[0112] Specifically, when performing parallel operation on the respective training groups of the plurality of natural language model expert networks, for each training group in the plurality of training groups, the text data in the training group is obtained, that is, at least one text data assigned to the natural language model expert network in the training group, and a training text matrix is constructed according to the obtained text data.

[0113] Wherein, before performing operation on each training group in the plurality of training groups, a matrix operation kernel needs to be constructed according to the respective training groups of the plurality of natural language model expert networks, and parallel operation is performed on the plurality of training groups based on the matrix operation kernel, that is, data operation is performed on each training group in the plurality of training groups at the same time, so as to obtain the respective data operation results of the plurality of training groups at the same time.

[0114] Step S904, an expert network matrix corresponding to the natural language model expert network in the training group is obtained, data operation is performed based on the expert network matrix and the training text matrix, and the data operation result corresponding to the training group is obtained.

[0115] Specifically, for each of the plurality of training groups, an expert network matrix corresponding to the natural language model expert network in the training group is obtained, wherein the natural language model expert network belongs to a neural network, the natural language model expert network can be converted from a network form to a matrix representation, and the matrix includes levels of the model and model parameters corresponding to different levels, so that the natural language model expert network can be converted into a matrix representation to obtain the expert network matrix.

[0116] Further, based on the expert network matrix and the training text matrix, data operation is performed, specifically, matrix multiplication between the expert network matrix and the training text matrix is calculated to obtain a data operation result corresponding to the training group. Wherein, the number of columns of the expert network matrix is equal to the number of rows of the training text matrix before multiplication, and the data operation result corresponding to the training group obtained is also a matrix, and the shape of the obtained operation result matrix is: the number of rows of the operation result matrix is equal to the number of rows of the expert network matrix, and the number of columns of the operation result matrix is equal to the number of columns of the training text matrix.

[0117] Exemplarily, for each of the plurality of training groups, when calculating the matrix multiplication between the expert network matrix and the training text matrix corresponding to the training group, a gemm algorithm (i.e. General Matrix to Matrix Multiplication, representing general matrix to matrix multiplication) can be used to multiply the two matrices to obtain an output matrix, i.e. to obtain the data operation result corresponding to the training group. It can be understood that when performing parallel operation on the plurality of training groups based on the matrix operation core, the gemm algorithm can be used to calculate the matrix multiplication between the expert network matrix and the training text matrix corresponding to each of the plurality of training groups, so as to simultaneously obtain the data operation result corresponding to each of the plurality of training groups.

[0118] In an exemplary embodiment, as shown in Figure 10 , a schematic diagram of the training text matrix and the expert network matrix constructed for each training group corresponding to the natural language model expert network is provided, referring to Figure 10 It can be known that parallel operation can be performed on N training groups, m0-k0 represents a training text matrix composed of all text data tokens allocated to the natural language model expert network 0, and k0-n0 represents an expert network matrix corresponding to the natural language model expert network 0, m1-k1 represents a training text matrix composed of all text data tokens allocated to the natural language model expert network 1, and k1-n1 represents an expert network matrix corresponding to the natural language model expert network 1. Similarly, m N-1 -k N-1 N-1 represents a training text matrix composed of all text data tokens allocated to the natural language model expert network N-1, and kN-1 - n N-1 denotes the expert network matrix corresponding to the natural language model expert network N-1.

[0119] Specifically, for each natural language model expert network, the text data allocated to the natural language model expert network is determined, and the natural language model expert network and the text data allocated to the natural language model expert network are taken as a training group, for example, N training groups are obtained, and then the N training groups are operated simultaneously to obtain the data operation result corresponding to each training group. Wherein, referring to Figure 10 It can be seen that the matrix operation of the training text matrix m0-k0and the expert network matrix k0-n0, the matrix operation of the training text matrix m1-k1and the expert network matrix k1-n1, …, and the matrix operation of the training text matrix m N-1 - k N-1 and the expert network matrix k N-1 - n N-1 are performed in parallel to obtain the data operation results corresponding to multiple different training groups.

[0120] In this embodiment, when performing parallel operation on multiple training groups corresponding to multiple natural language model expert networks, for each training group in the multiple training groups, a training text matrix is constructed according to the text data in the training group, and an expert network matrix corresponding to the natural language model expert network in the training group is obtained, so as to perform data operation based on the expert network matrix and the training text matrix to obtain the data operation result corresponding to the training group. Thus, the multiple training groups can be operated in parallel to quickly obtain the data operation result corresponding to each training group without performing serial operation on the text data according to each natural language model expert network selected by the text data, thereby reducing the cumbersome operation steps in the data processing process and the operation waiting time for each text data, and further improving the text data processing efficiency.

[0121] In an exemplary embodiment, as shown in Figure 11 , a text data processing method is provided, which is applied to the server 104 in Figure 1 for example, and includes the following steps S1101 to S1109. Wherein:

[0122] In step S1101, a set of text data is obtained, the state tensor data corresponding to each text data in the set of text data is extracted, the dimension of the state tensor data corresponding to each text data is adjusted, and the text data after dimension adjustment is obtained.

[0123] Specifically, by obtaining a text data set for inputting a natural language model, and extracting state tensor data corresponding to each text data in the text data set, such as hidden state tensor of the text data (i.e. hidden_states), and obtaining the shape of the hidden state tensor, such as the shape of the obtained hidden state tensor before adjustment is three-dimensional.

[0124] Further, the state tensor data corresponding to each text data is adjusted in dimension, such as adjusting the three-dimensional state tensor data (i.e. hidden state tensor) to two-dimensional, to obtain the text data after dimension adjustment. Among them, the text data after dimension adjustment is used for data operation with the natural language model expert network allocated to, that is, according to the routing decision data, each text data in the text data set after dimension adjustment is dynamically divided, each text data is allocated to at least one natural language model expert network in the plurality of natural language model expert networks, so as to perform data operation according to the text data and the natural language model expert network allocated to, to obtain the data operation result.

[0125] Step S1102, obtaining routing decision data corresponding to the natural language model gating network, and according to the routing decision data, dynamically dividing each text data in the text data set after dimension adjustment, and allocating each text data to at least one natural language model expert network in the plurality of natural language model expert networks.

[0126] Specifically, the natural language model for data processing of text data can be a hybrid expert model, which specifically includes a gating network and a plurality of expert networks. The gating network is used to determine which token (also known as token, which can be a word, phrase or other text data) is allocated to which expert network. Specifically, by obtaining routing decision data (also known as gating mechanism learned by the gating network) corresponding to the natural language model gating network, it is determined which expert network or combination of expert networks is suitable for processing text data according to the routing decision data, so that each text data in the text data set can be dynamically divided according to the routing decision data (i.e. gating mechanism learned by the gating network), and each text data in the text data set can be allocated to at least one natural language model expert network in the plurality of natural language model expert networks.

[0127] Step S1103, for each natural language model expert network, determining the text data allocated to the natural language model expert network, and taking the natural language model expert network and the text data allocated to the natural language model expert network as a training group.

[0128] Specifically, one natural language model expert network can be assigned one or more text data, and the same text data can also be assigned to one or more natural language model expert networks. For each natural language model expert network, after determining the text data assigned to the natural language model expert network, the natural language model expert network and the text data assigned to the natural language model expert network are taken as a training group. Each natural language model expert network corresponds to a training group, that is, a training group specifically includes one natural language model expert network and at least one text data assigned to the natural language model expert network.

[0129] Step S1104, constructing a matrix operation kernel according to the training group corresponding to each of the plurality of natural language model expert networks, and performing parallel operation on the plurality of training groups based on the matrix operation kernel to obtain a data operation result corresponding to each training group.

[0130] Specifically, when performing parallel operation on the training group corresponding to each of the plurality of natural language model expert networks, a matrix operation kernel is constructed according to the training group corresponding to each of the plurality of natural language model expert networks, and parallel operation is performed on the plurality of training groups based on the matrix operation kernel, that is, data operation is performed on each of the plurality of training groups at the same time to simultaneously obtain a data operation result corresponding to each of the plurality of training groups.

[0131] For each of the plurality of training groups, the text data in the training group is obtained, that is, at least one text data assigned to the natural language model expert network in the training group, and a training text matrix is constructed according to the obtained text data, and an expert network matrix corresponding to the natural language model expert network in the training group is obtained, so as to perform data operation based on the expert network matrix and the training text matrix to obtain a data operation result corresponding to the training group.

[0132] Further, for each of the plurality of training groups, the natural language model expert network in the training group is converted from a network form to a matrix representation to obtain an expert network matrix, so as to perform data operation based on the expert network matrix and the training text matrix, such as calculating the matrix product between the expert network matrix and the training text matrix to obtain a data operation result corresponding to the training group.

[0133] Step S1105, for each text data, determining the natural language model expert network to which the text data is assigned.

[0134] Specifically, one natural language model specialist network can be assigned one or more text data, and the same text data can also be assigned to one or more natural language model specialist networks. When the text data processing result of each text data needs to be determined, the natural language model specialist network to which each text data is assigned needs to be determined first, and the number of natural language model specialist networks to which the text data is assigned needs to be counted.

[0135] Step S1106, from the data operation results of the training set corresponding to the natural language model specialist network to which the text data is assigned, the sub-operation result corresponding to the text data is extracted.

[0136] Specifically, for each text data, the data operation result of the training set corresponding to the natural language model specialist network to which the text data is assigned is obtained, and the sub-operation result corresponding to the text data is extracted from the data operation result of the training set corresponding to the natural language model specialist network to which the text data is assigned.

[0137] Step S1107, if the number of natural language model specialist networks to which the text data is assigned is single, the sub-operation result corresponding to the text data is taken as the text data processing result of the text data.

[0138] Specifically, for a certain text data, if it is determined that the number of natural language model specialist networks to which the text data is assigned is single, the sub-operation result corresponding to the text data is extracted from the data operation result of the training set corresponding to the natural language model specialist network to which the text data is assigned, and the extracted sub-operation result is taken as the text data processing result of the text data.

[0139] After step S1106 is performed, step S1108 is performed, and if the number of natural language model specialist networks to which the text data is assigned is multiple, the network weights of the multiple natural language model specialist networks are determined.

[0140] Specifically, for a certain text data, if it is determined that the number of natural language model specialist networks to which the text data is assigned is multiple, the network weights corresponding to the multiple natural language model specialist networks to which the text data is assigned are determined respectively. Among them, the network weights of each natural language model specialist network in the natural language model such as the hybrid specialist model are specifically determined according to the gating network in the hybrid specialist model.

[0141] Step S1109, according to the multiple sub-operation results corresponding to the text data and the network weights of the multiple natural language model specialist networks, weighted calculation is performed to obtain the text data processing result corresponding to the text data.

[0142] Specifically, for a certain text data, after determining the multiple natural language model expert networks to which the text data is allocated, extracting multiple sub-computation results from the computation results of the data of the training set corresponding to the natural language model expert networks to which the text data is allocated respectively, and determining the network weights of the multiple natural language model expert networks respectively, the multiple sub-computation results corresponding to the text data and the network weights of the multiple natural language model expert networks are weighted and calculated to obtain a text data processing result corresponding to the text data.

[0143] In the above text data processing method, by obtaining a text data set, each text data in the text data set is allocated to at least one natural language model expert network in the multiple natural language model expert networks, and for each natural language model expert network, the text data allocated to the corresponding natural language model expert network is further determined, so that the natural language model expert network and the text data allocated to the natural language model expert network are used as a training set, so that subsequent parallel computation can be performed on multiple training sets, reducing the waiting time of serial computation in the data processing process. Further, by performing parallel computation on the training sets corresponding to the multiple natural language model expert networks respectively, the data computation results corresponding to each training set can be obtained at the same time, so that the text data processing result of each text data is determined according to the data computation result corresponding to each training set. Since it is not necessary to perform serial computation on the text data according to each natural language model expert network selected by the text data, the cumbersome operation steps in the data processing process and the operation waiting time for each text data are reduced, and the text data processing efficiency is further improved.

[0144] It should be understood that, although each step in the flowchart involved in the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with other steps or steps or stages in other steps.

[0145] Based on the same inventive concept, the embodiments of the present application further provide a text data processing apparatus for implementing the above-mentioned text data processing method. The implementation scheme of the apparatus for solving the problem is similar to the implementation scheme described in the above method, and therefore the specific limitations in one or more text data processing apparatus embodiments provided below can refer to the limitations of the text data processing method described above, which will not be repeated here.

[0146] In one exemplary embodiment, as shown in Figure 12 A text data processing apparatus is provided, comprising a text data distribution module 1202, a training group obtaining module 1204, a parallel operation module 1206, and a text data processing result obtaining module 1208, wherein:

[0147] The text data distribution module 1202 is configured to obtain a text data set, and distribute each text data in the text data set to at least one natural language model expert network in a plurality of natural language model expert networks respectively;

[0148] The training group obtaining module 1204 is configured to, for each natural language model expert network, determine the text data distributed to the natural language model expert network, and take the natural language model expert network and the text data distributed to the natural language model expert network as a training group;

[0149] The parallel operation module 1206 is configured to perform parallel operation on the training group corresponding to each natural language model expert network in the plurality of natural language model expert networks, to obtain a data operation result corresponding to each training group;

[0150] The text data processing result obtaining module 1208 is configured to determine a text data processing result of each text data according to the data operation result corresponding to each training group.

[0151] In the text data processing apparatus, each text data in the text data set is assigned to at least one natural language model expert network in the plurality of natural language model expert networks, and for each natural language model expert network, text data assigned to the corresponding natural language model expert network is further determined, so that the natural language model expert network and the text data assigned to the natural language model expert network are taken as a training group, and subsequent parallel operation can be performed on the plurality of training groups, thereby reducing the waiting time of serial operation in the data processing process. Further, by performing parallel operation on the training groups corresponding to the plurality of natural language model expert networks, the data operation results corresponding to each training group can be obtained at the same time, so that the text data processing result of each text data is determined according to the data operation result corresponding to each training group. Since serial operation is not required for each natural language model expert network selected by the text data, the cumbersome operation steps in the data processing process and the operation waiting time for each text data are reduced, and the text data processing efficiency is further improved.

[0152] In an exemplary embodiment, the text data processing result obtaining module is further configured to: for each text data, determine the natural language model expert network to which the text data is assigned; extract the sub-operation result corresponding to the text data from the data operation result of the training group corresponding to the natural language model expert network to which the text data is assigned; and obtain the text data processing result corresponding to the text data according to the sub-operation result corresponding to the text data.

[0153] In an exemplary embodiment, the text data processing result obtaining module is further configured to: if the number of natural language model expert networks to which the text data is assigned is one, take the sub-operation result corresponding to the text data as the text data processing result of the text data.

[0154] In an exemplary embodiment, the text data processing result obtaining module is further configured to: if the number of natural language model expert networks to which the text data is assigned is one, take the sub-operation result corresponding to the text data as the text data processing result of the text data.

[0155] In an exemplary embodiment, the parallel operation module is further configured to: construct a matrix operation core according to the training groups corresponding to the plurality of natural language model expert networks, and perform parallel operation on the plurality of training groups based on the matrix operation core to obtain the data operation result corresponding to each training group.

[0156] In an example embodiment, the parallel operation module is further configured to: for each of the plurality of training groups, construct a training text matrix according to the text data in the training group; obtain an expert network matrix corresponding to the natural language model expert network in the training group, and perform data operation based on the expert network matrix and the training text matrix to obtain a data operation result corresponding to the training group.

[0157] In an example embodiment, the text data distribution module is further configured to: obtain routing decision data corresponding to the natural language model gating network; and dynamically divide each text data in the text data set according to the routing decision data, and distribute each text data to at least one natural language model expert network in the plurality of natural language model expert networks.

[0158] In an example embodiment, a text data processing apparatus is provided, and further comprises a dimension adjustment module configured to:

[0159] extract state tensor data corresponding to each text data in the text data set, adjust the dimension of the state tensor data corresponding to each text data, and obtain dimension-adjusted text data; and the dimension-adjusted text data is used for data operation with the natural language model expert network to which the text data is distributed.

[0160] The above modules in the text data processing apparatus can be realized by software, hardware, or a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above modules.

[0161] In an example embodiment, a computer device is provided, which can be a server or a terminal. Taking the computer device as a server, the internal structure diagram thereof can be as shown in Figure 13As shown in the figure. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store text data set, text data, natural language model expert network, training group, data operation result and text data processing result and other data. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to realize a text data processing method.

[0162] Those skilled in the art can understand that, Figure 13 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0163] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to realize the steps in each of the above method embodiments.

[0164] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by the processor to realize the steps in each of the above method embodiments.

[0165] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by the processor to realize the steps in each of the above method embodiments.

[0166] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0167] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0168] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0169] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A text data processing method, characterized in that, The method includes: Obtain a text dataset, and assign each text data in the text dataset to at least one of multiple natural language model expert networks; For each of the natural language model expert networks, determine the text data assigned to the natural language model expert network, and use the natural language model expert network and the text data assigned to the natural language model expert network as a training group; Parallel computation is performed on the training groups corresponding to each of the multiple natural language model expert networks to obtain the data computation results corresponding to each training group; Based on the data calculation results corresponding to each training group, the text data processing result for each text data is determined.

2. The method according to claim 1, characterized in that, The step of determining the text data processing result for each piece of text data based on the data calculation results corresponding to each training group includes: For each piece of text data, determine the natural language model expert network to which the text data is assigned; Extract the sub-operation results corresponding to the text data from the data operation results of the training group corresponding to the assigned natural language model expert network; Based on the sub-operation results corresponding to the text data, the text data processing result corresponding to the text data is obtained.

3. The method according to claim 2, characterized in that, The step of obtaining the text data processing result corresponding to the text data based on the sub-operation result corresponding to the text data includes: If the number of natural language model expert networks to which the text data is assigned is a single one, the sub-operation result corresponding to the text data will be used as the text data processing result of the text data.

4. The method according to claim 2, characterized in that, The step of obtaining the text data processing result corresponding to the text data based on the sub-operation result corresponding to the text data includes: If the text data is assigned to multiple natural language model expert networks, determine the network weights of each of the multiple natural language model expert networks; Based on the sub-operation results corresponding to the text data and the network weights of the various natural language model expert networks, a weighted calculation is performed to obtain the text data processing result corresponding to the text data.

5. The method according to any one of claims 1 to 4, characterized in that, The parallel computation of the training groups corresponding to the multiple natural language model expert networks to obtain the data computation results corresponding to each training group includes: A matrix operation kernel is constructed based on the training groups corresponding to the multiple natural language model expert networks. Parallel operations are performed on the multiple training groups based on the matrix operation kernel to obtain the data operation results corresponding to each training group. The method of performing calculations for each of the multiple training groups includes: For each of the multiple training groups, a training text matrix is ​​constructed based on the text data in the training group; Obtain the expert network matrix corresponding to the natural language model expert network in the training group, and perform data operations based on the expert network matrix and the training text matrix to obtain the data operation results corresponding to the training group.

6. The method according to any one of claims 1 to 4, characterized in that, The step of assigning each text data in the text dataset to at least one of multiple natural language model expert networks includes: Obtain routing decision data corresponding to the natural language model gating network; Based on the routing decision data, each text data in the text dataset is dynamically partitioned, and each text data is assigned to at least one of the multiple natural language model expert networks.

7. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Extract the state tensor data corresponding to each text data in the text dataset, and adjust the dimensions of the state tensor data corresponding to each text data to obtain dimension-adjusted text data; the dimension-adjusted text data is used for data computation with the natural language model expert network to which it is assigned.

8. A text data processing apparatus, characterized in that, The device includes: The text data allocation module is used to acquire a text dataset and allocate each text data in the text dataset to at least one of the multiple natural language model expert networks. The training group acquisition module is used to determine the text data assigned to each natural language model expert network, and to use the natural language model expert network and the text data assigned to the natural language model expert network as a training group. The parallel computing module is used to perform parallel computing on the training groups corresponding to the multiple natural language model expert networks to obtain the data computing results corresponding to each training group. The text data processing result acquisition module is used to determine the text data processing result of each text data based on the data operation result corresponding to each training group.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images