Model training method and device, model reasoning method and device, and equipment

Abstract

The invention provides a model training method and device, a model reasoning method and device and equipment, and relates to the field of data processing, in particular to the technical fields of artificial intelligence, big data, deep learning, large models and the like. According to the specific implementation scheme, on the basis of a folding feature value used for folding a lexical element sequence, folding processing is conducted on an initial lexical element sequence used for model training, so that at least a first lexical element sequence obtained after folding processing is obtained; wherein the initial lexical element sequence represents a lexical element sequence composed of T1 lexical elements; the sequence length of the first lexical element sequence is smaller than that of the initial lexical element sequence; and at least inputting the first lexical element sequence into a preset model to perform model training on the preset model to obtain a target model.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to technical fields such as artificial intelligence, big data, deep learning, and large models. Background Art

[0002] Currently, the development of the natural language field is moving towards the era of ultra-large-scale models. By using ultra-powerful computing power to train ultra-large parameter models on massive text data, the language models produced can have the general semantic understanding and generation capabilities of multi-task and few-sample learning. Although large models have demonstrated powerful general capabilities, their computing consumption and video memory usage increase at the quadratic level with the input length, which brings huge cost overhead to model training and deployment, and also additionally limits their ability to solve long text tasks. Summary of the invention

[0003] The present invention provides a model training method, a model reasoning method, and a device and equipment thereof.

[0004] According to one aspect of the present disclosure, a model training method is provided, comprising:

[0005] Based on the folding feature value used to fold the word-gram sequence, the initial word-gram sequence used for model training is folded to obtain at least the first word-gram sequence after folding; wherein the initial word-gram sequence represents T 1 The first word sequence is composed of word units; the sequence length of the first word sequence is less than the sequence length of the initial word sequence;

[0006] At least the first word-gram sequence is input into a preset model to perform model training on the preset model to obtain a target model.

[0007] According to another aspect of the present disclosure, a model reasoning method is provided, comprising:

[0008] Obtain an initial sequence of tokens to be inferred;

[0009] Based on the folding feature value used to fold the word-gram sequence, the initial word-gram sequence to be inferred is folded to obtain at least a first target word-gram sequence to be inferred; wherein the initial word-gram sequence to be inferred represents T 2 The first target word-gram sequence to be inferred is composed of word-grams; the sequence length of the first target word-gram sequence to be inferred is less than the sequence length of the initial word-gram sequence to be inferred;

[0010] At least a first target word-gram sequence to be inferred is input into a target model to obtain a target inference result, wherein the target inference result is a predicted next word-gram sequence of the target word-gram sequence to be inferred.

[0011] According to another aspect of the present disclosure, a model training device is provided, comprising:

[0012] The first data processing unit is used to fold the initial word-gram sequence used for model training based on the folding feature value used for folding the word-gram sequence, so as to obtain at least a first word-gram sequence after folding; wherein the initial word-gram sequence represents T 1 The first word sequence is composed of word units; the sequence length of the first word sequence is less than the sequence length of the initial word sequence;

[0013] The model training unit is used to input at least the first word-unit sequence into a preset model to perform model training on the preset model to obtain a target model.

[0014] According to another aspect of the present disclosure, there is provided a model reasoning device, comprising:

[0015] The second data processing unit is used to obtain an initial word-gram sequence to be inferred; based on the folding feature value used to fold the word-gram sequence, the initial word-gram sequence to be inferred is folded to obtain at least a first target word-gram sequence to be inferred; wherein the initial word-gram sequence to be inferred represents T 2 The first target word-gram sequence to be inferred is composed of word-grams; the sequence length of the first target word-gram sequence to be inferred is less than the sequence length of the initial word-gram sequence to be inferred;

[0016] The model inference unit is used to input at least a first target word-gram sequence to be inferred into a target model to obtain a target inference result, wherein the target inference result is a predicted next word-gram sequence of the target word-gram sequence to be inferred.

[0017] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0021] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0022] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0023] In this way, the disclosed scheme can fold the initial word sequence according to the folding feature value, and obtain a sequence whose length of the folded word sequence is less than the length of the initial word sequence (that is, the first word sequence), and then use the folded word sequence to train the preset model. In this way, the training efficiency of the model is improved by compressing the input of the model, laying the foundation for subsequently improving the efficiency of model reasoning.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0026] Figure 1 This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 1 ;

[0027] Figure 2 Schematic diagram of an initial word sequence before and after folding in an example according to an embodiment of the present application Figure 1 ;

[0028] Figure 3 This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 2 ;

[0029] Figure 4 Schematic diagram of an initial word sequence before and after folding in an example according to an embodiment of the present application Figure 2 ;

[0030] FIG5( a) is a schematic diagram illustrating preset model input information according to an embodiment of the present application;

[0031] FIG5( b ) is a schematic diagram of the structure of N network layers in a preset model according to an embodiment of the present application;

[0032] FIG6( a ) is a schematic diagram illustrating input information of a non-first layer in a preset model according to an embodiment of the present application;

[0033] FIG6( b ) is a schematic diagram illustrating input information of each layer in a preset model according to an embodiment of the present application;

[0034] Figure 7This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 3 ;

[0035] Figure 8 is a schematic diagram illustrating output information of part of the network layer in a preset model according to an embodiment of the present application;

[0036] Fig. 9 is a schematic flow chart of a model reasoning method according to an embodiment of the present application;

[0037] Fig.10 is a schematic diagram of a prediction scenario in a specific example according to a model reasoning method in an embodiment of the present application;

[0038] Fig.11 is a structural schematic diagram of a model training device according to an embodiment of the present application;

[0039] Fig.12 is a schematic diagram of the structure of a model reasoning device according to an embodiment of the present application;

[0040] Fig.13 It is a block diagram of an electronic device used to implement the model training method or model inference method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0042] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0043] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0044] The present disclosure proposes a model training method to improve the training efficiency of large models.

[0045] Specifically, Figure 1 This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 1 The method may be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0046] Further, the method includes at least part of the following contents. Figure 1 As shown, including:

[0047] Step S101: Based on the folding feature value used to fold the word-gram sequence, folding processing is performed on the initial word-gram sequence used for model training to obtain at least a first word-gram sequence after folding processing.

[0048] Here, the initial word sequence represents T 1 The first word sequence after folding contains T 1 Further, the sequence length of the first word-gram sequence is less than the sequence length of the initial word-gram sequence.

[0049] For example, in one example, the initial word sequence is folded in depth to obtain the first word sequence after folding, which effectively compresses the length of the input word. In other words, it effectively reduces the number of words first input to the preset model, thereby laying a foundation for improving the subsequent model training efficiency and then improving the model reasoning efficiency. Figure 2 As shown, the initial word sequence containing 4 words is {x 0 ,x 1 ,x 2 ,x 3}, assuming that the folding feature value is 2, then, using this folding feature value, the initial word sequence {x 0 ,x 1 ,x 2 ,x 3} is folded twice, and the first word sequence after folding can be recorded as {x 0 ,x 2}.

[0050] Step S102: at least input the first word-gram sequence into a preset model to perform model training on the preset model to obtain a target model.

[0051] In this way, the disclosed scheme can fold the initial word sequence according to the folding feature value, and obtain a sequence whose length of the folded word sequence is less than the length of the initial word sequence (that is, the first word sequence), and then use the folded word sequence to train the preset model. In this way, the training efficiency of the model is improved by compressing the input of the model, laying the foundation for subsequently improving the efficiency of model reasoning.

[0052] Furthermore, since the disclosed solution compresses the input length of the model, the disclosed solution can effectively reduce the computational complexity and the occupation of video memory resources, thereby reducing the cost required for model training and deployment. In addition, the disclosed solution also effectively improves the model's ability to solve long text tasks (such as long text summarization and long text question answering).

[0053] In a specific example, the preset model may be a large model, and further, a large language model. Alternatively, it may be other models, which are not limited in the present disclosure.

[0054] Figure 3 This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 2 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figure 1 and Figure 2 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0055] Further, the method includes at least part of the following contents. Figure 3 As shown, including:

[0056] Step S301: Based on the folding feature value used to fold the word-gram sequence, fold the initial word-gram sequence used for model training to obtain a first word-gram sequence and a second word-gram sequence.

[0057] Here, the initial word sequence represents T 1 The first word sequence is composed of word units; the sequence length of the first word sequence is less than the sequence length of the initial word sequence.

[0058] Furthermore, the first word sequence includes t displayed after folding processing. 11 word, the t 11 The word is the T 1Further, the second word-unit sequence includes the t covered by the displayed word-unit after folding. 12 word, the t 12 The word is also the T 1 Here, t 11 +t 12 =T 1 .

[0059] For example, continuing with Figure 2 For example, for the initial word sequence {x 0 ,x 1 ,x 2 ,x 3 After folding twice, we can get Figure 4 The two words shown after the folding process are shown, namely the first word sequence {x 0 ,x 2}, and the two words hidden by the displayed words after folding, that is, the second word sequence {x 1 ,x 3}.

[0060] Step S302: Use each word in the first word-gram sequence as the input of the first layer of the N network layers included in the preset model, and use at least each word in the second word-gram sequence as part of the input of other layers except the first layer of the N network layers, so as to adjust at least part of the network parameters in the N network layers to obtain the target model, for example, fine-tune at least part of the network parameters in the N network layers to obtain the target model.

[0061] For instance, in one example, each word in the first word sequence is input into the corresponding position of the first layer in the preset model according to the order of each word in the first word sequence. Here, "position" may specifically refer to the input position of the word.

[0062] For example, continuing with Figure 4 The initial word sequence shown in the figure is obtained by 0 ,x 2} and the second word sequence {x 1 ,x 3} as an example, as shown in FIG5(a), according to the first word sequence {x 0 ,x 2}, the first word sequence {x 0 ,x 2} 0 Input to the first layer position 1 of the preset model, and convert the first word sequence {x 0 ,x 2} 2 Enter into position 2 of the first floor.

[0063] In this way, the disclosed solution uses each word in the first word sequence after folding as the input of the first layer in the preset model, and uses each word in the second word sequence as the input of other layers in the preset model except the first layer. In this way, the integrity of the input information after the model input is compressed can be effectively guaranteed, thereby improving the speed of model training while ensuring that the training effect of the model is not affected, laying the foundation for subsequently improving the reasoning efficiency of the model and reducing the cost required for model training and deployment.

[0064] Furthermore, in one example, the N network layers are connected in series; the output of the jth layer in the N network layers is used as the input of the j+1th layer in the N network layers. For example, as shown in FIG5(b), when j is an integer greater than or equal to 0 and less than N, the output of the jth layer in the N network layers included in the preset model is used as the input of the j+1th layer in the N network layers, wherein when j is 0, the input of the 0th layer (also referred to as the first layer) is the first word sequence; when j is N-1, the output of the N-1th layer is the output of the last layer in the preset model. In this way, the input information can be feature extracted layer by layer, thereby capturing richer and deeper information, laying a foundation for improving the accuracy and robustness of model prediction.

[0065] In a specific example of the disclosed solution, the input of each layer in the N network layers may be obtained in the following manner; the specific manner includes:

[0066] Based on the folded feature value and the position of the jth layer in the N network layers, the input of the jth layer is determined; here, the value of j is related to N. For example, the value of j is a natural number greater than or equal to 0 and less than or equal to N-1.

[0067] Further, in an example, the above-mentioned determining the input of the jth layer based on the folded feature value and the position of the jth layer in the N network layers may specifically include:

[0068] Based on the numerical relationship between the number of layers of the j-th layer in the N network layers and the folded feature value, the input of the j-th layer is determined.

[0069] That is to say, in this example, the input of the first layer (for example, when j is 0) of the N network layers included in the preset model is the first word sequence, and the input of other layers except the first layer in the N network layers (that is, when j is an integer greater than 0 and less than N) needs to be determined based on the numerical relationship between the number of layers of the jth layer in the N network layers and the folded feature value. In this way, a refined scheme is provided for determining the input of other word sequences except the first word sequence in the initial word sequence into the preset model, which is simple and efficient.

[0070] In this way, the disclosed scheme provides a detailed scheme for determining how to input word sequences other than the first word sequence (that is, words covered due to folding) into a preset model according to the degree of folding of the input information. The scheme is simple and efficient, and thus effectively avoids information loss caused by folding of model input information, balances the computational complexity of each network layer, and reduces the pressure on the server when the model processes information, thereby laying the foundation for accelerating the training speed of the model and reducing the cost required for model training and deployment.

[0071] Further, in a specific example, the input of the jth layer may be obtained in the following manner; specifically, the above-described determination of the input of the jth layer based on the numerical relationship between the number of layers of the jth layer in the N network layers and the folded eigenvalue may specifically include at least one of the following two manners:

[0072] Method 1: Determine the input of the non-first layer (for example, when j is 1 to N-1); specifically, determine that the number of layers of the j-th layer in the N network layers (for example, which can be understood as the value of j) is less than the folded feature value (for example, in one example, when (layer depth - 1) is less than the folded feature value s), then obtain the input of the j-th layer based on the implicit output result of the j-1th layer and at least one word in the second word sequence.

[0073] It should be noted that, in this example, the number of the jth layer in the N network layers may specifically refer to the value of j; further, the layer depth may specifically refer to the position in the total number of layers. For example, for the first layer among the 4 layers, its layer depth may specifically be 1, and the layer below the first layer may have a layer depth of 2.

[0074] Here, the implicit output result can be specifically understood as the implicit result obtained after the network layer processes the word unit.

[0075] It should be noted that, in one example, j takes a value of 0 to N-1; further, the present disclosure is exemplified by taking j taking a value of 0 to N-1 as an example. It is understandable that j can also take a value of 1 to N, in which case the superscript can be adjusted accordingly based on the actual numerical value, and the present disclosure does not limit this.

[0076] It can be understood that for the first layer (for example, j takes the value of 0), its input is each word in the first word sequence; for non-first layers, it can be obtained based on method 1. For example, for j taking the value of 1 and the folding feature value taking the value of 2, at this time, j is less than s, or (layer depth-1) (for example, 2-1) is less than 2, then the input of the first layer can be specifically: the implicit output result of the 0th layer, and at least one word in the second word sequence (that is, a word covered by the folding process). In other words, in a scenario, for some layers that meet the conditions, in addition to using the implicit output result output by the previous layer (for example, the j-1th layer) as the input of the next layer (for example, the jth layer), it is also necessary to introduce the word covered by the displayed word after the folding process as another input of the next layer (for example, the jth layer). This lays the foundation for effectively avoiding the loss of original input information.

[0077] Further, in an example, obtaining the input of the jth layer based on the implicit output result of the j-1th layer and at least one word in the second word sequence described in the first embodiment may specifically include:

[0078] Implicit output result based on the i-th position of the j-1th layer and word x j+i×s , and get the input of the i-th position of the j-th layer.

[0079] Here, j represents the number of layers; i represents the input position of the word unit, and its value is the same as that of T 1 The value of is related to the value of the folding feature value. For example, in one example, i is 0 to [(T 1 / s)―1]. s represents the folded eigenvalue.

[0080] For example, with 4 network layers (remembered as {L 0 ,L 1 ,L 2 ,L 3}) as an example, and continue with the initial word sequence {x 0 ,x 1 ,x 2 ,x 3} is folded twice to obtain the first word sequence {x 0 ,x 2} and the second word sequence {x 1 ,x 3} as an example; in this case, j takes a value from 0 to 3, and i takes a value from 0 to 1. Further, as shown in FIG6(a), for the first layer, for example, it can be recorded as the 0th layer L 0 (i.e. j is 0), at this time, the 0th layer L 0The input at the 0th position (where i takes the value 0) is x 0 , the 0th layer L 0 The input at the 1st position (where i takes the value 1) is x 2 . Further, for the 1st layer L 1 , that is, when j takes the value 1, at this time, satisfying j < s, then according to the 0th layer L in the preset model 0 The implicit output result at the 0th position and the token x in the second token sequence 1 , the input at the 0th position of the 1st layer L is obtained 0 . Correspondingly, according to the 0th layer L in the preset model 0 The implicit output result at the 1st position and the token x in the second token sequence 3 , the input at the 1st position of the 1st layer L is obtained 1 .

[0081] In this way, in the input of the network layer that meets the layer depth requirement in the present disclosure solution, at least one token in the second token sequence is additionally introduced. Thus, the loss of the original input information is effectively avoided. In other words, while compressing the input of the preset model, the present disclosure solution utilizes the depth dimension of the network layer to obtain the specific position of the input required for the compressed token, effectively avoiding the loss of the original input information. Furthermore, while ensuring the training effect of the model, it lays a foundation for improving the inference efficiency of the model and reducing the costs required for the training and deployment of the model in the future.

[0082] Further, in a specific example, the above-mentioned obtaining the input at the i-th position of the j-th layer based on the implicit output result at the i-th position of the (j - 1)-th layer and the token x j+i×s may specifically include:[[]]

[0083] Using the depth folding function, fuse the implicit output result at the i-th position of the (j - 1)-th layer and the token x j+i×s , and use the fused result as the input at the i-th position of the j-th layer.

[0084] For example, in an example, the input at the i-th position of the j-th layer can be denoted as Denote the depth folding function as representing the sequence depth folding function, which is used to fuse the implicit output result of the previous layer (for example, represented by a vector) and the additionally compressed input x j+i×s , for example,[[]] The function can simply implement the bitwise addition operation.

[0085] At this time, if the number of layers in the jth layer of the N network layers is less than the folded eigenvalue s (for example, when (layer depth - 1) is less than the folded eigenvalue s), then the input of the i-th position of the jth layer is It can be expressed as:

[0086]

[0087] In this way, the disclosed scheme provides a specific scheme for determining the input of the i-th position of the j-th layer. In this way, the scheme can be used to quickly determine the word-grams in the second word-gram sequence that need to be additionally introduced for the input of the i-th position of the j-th layer, thereby effectively ensuring that the compressed word-gram information is not lost. In this way, while ensuring the training effect of the model, it lays the foundation for subsequently improving the reasoning efficiency of the model and reducing the cost required for model training and deployment.

[0088] Method 2: Determine the input of the non-first layer (for example, when j is 1 to N-1); determine that the number of layers of the j-th layer in the N network layers is greater than or equal to the folded eigenvalue (for example, in one example, when (layer depth - 1) is greater than or equal to the folded eigenvalue s), and obtain the input of the i-th position of the j-th layer based on the implicit output result of the j-1-th layer.

[0089] That is to say, in this example, when the number of layers of the jth (j is an integer greater than 0 and less than N) layer in the N network layers is greater than or equal to the folded feature value (for example, (layer depth - 1) is greater than or equal to the folded feature value s), the input of the i-th position of the j-th layer can be directly obtained based on the implicit output result of the j-1th layer, without the need to additionally introduce the word-units of the second word-unit sequence. For example, in one example, when the number of layers of the jth (j is an integer greater than 0 and less than N) layer in the N network layers is greater than or equal to the folded feature value (for example, (layer depth - 1) is greater than or equal to the folded feature value s), the implicit output result of the i-th position of the j-1th layer can be directly obtained. As the input of the i-th position of the j-th layer.

[0090] For example, continuing with the 4 network layers {L 0 ,L 1 ,L 2 ,L 3}, and continue with the initial word sequence {x 0 ,x 1 ,x 2 ,x 3} is folded twice to obtain the first word sequence {x 0 ,x 2} and the second word sequence {x 1 ,x 3} as an example, in this case, j takes a value from 0 to 3, and i takes a value from 0 to 1. Further, as shown in FIG6(b), for the second layer L 2 For example, j is 2. At this time, j≥s is satisfied, and the first layer L 1 Implicit output result at position 0 Directly as the second layer L 2 The input of the 0th position, and the first layer L 1 Implicit output result at position 1 Directly as the second layer L 2 The first position of the input; further, for the third layer L 3 For example, j is 3. At this time, j≥s is satisfied, and the second layer L 2 Implicit output result at position 0 Directly as the 3rd layer L 3 The input of the 0th position, and the second layer L 2 Implicit output result at position 1 Directly as the 3rd layer L 3 Input for the 1st position.

[0091] For example, in one example, the input at the i-th position of the j-th layer is At this time, if the number of layers in the jth (non-first layer, for example, j is an integer greater than 0 and less than N) layer in the N network layers (for example, the value of j) is greater than or equal to the folded eigenvalue s, then the input of the i-th position of the jth layer is It can be expressed as:

[0092]

[0093] That is to say, when the number of layers is greater than or equal to the folding feature value (also called the sequence depth folding multiple) s, the input of the i-th position of the j-th layer is the output of the previous layer, and no additional folded and compressed words need to be fused.

[0094] It should be noted that the disclosed solution linearly compresses the input length that the model needs to receive, and in order to ensure that the information after the input compression is not lost, the compressed words are sequentially used as additional inputs for other layers except the first layer in the original order of input. This input method fully utilizes the layer depth dimension, effectively reduces the input length, and avoids information loss.

[0095] In this way, the disclosed scheme determines the specific scheme for inputting the i-th position of the j-th layer. The scheme makes full use of the layer depth dimension of the network layer to determine the specific input information required for different network layers. In this way, while effectively compressing the model input, the loss of original input information caused by compression is avoided. Furthermore, while ensuring the training effect of the model, the training speed of the model is improved, laying the foundation for subsequently improving the reasoning ability of the model and reducing the cost required for training and deployment of the model.

[0096] Figure 7 This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 3 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figure 1 The relevant contents of the method shown in FIG. 6 can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0097] Further, the method includes at least part of the following contents. Figure 7 As shown, including:

[0098] Step S701: Based on the folding feature value used to fold the word-gram sequence, fold the initial word-gram sequence used for model training to obtain a first word-gram sequence and a second word-gram sequence.

[0099] Here, the initial word sequence represents T 1 The first word sequence is composed of word units; the sequence length of the first word sequence is less than the sequence length of the initial word sequence.

[0100] Furthermore, the first word sequence includes t displayed after folding processing. 11 word units, the second word unit sequence includes t that is covered by the displayed word units after folding. 12 A word.

[0101] Here, for the relevant description of the initial word sequence, the first word sequence and the second word sequence, please refer to the above examples and will not be repeated here.

[0102] Step S702: Use each word in the first word sequence as the input of the first layer of N network layers included in the preset model, and use at least each word in the second word sequence as part of the input of other layers of the N network layers except the first layer.

[0103] Here, for the relevant description of the specific input of different network layers, please refer to the above examples, which will not be repeated here.

[0104] Step S703: Obtain the output predicted word-gram sequence for predicting the next word-gram of each word-gram in the initial word-gram sequence.

[0105] Step S704: Based at least on the degree of difference between the predicted word-gram sequence and the target sequence (eg, a pre-labeled theoretical word-gram sequence), at least some of the network parameters in the N network layers are adjusted (eg, fine-tuned) to obtain a target model.

[0106] In this way, the disclosed scheme provides a refined training scheme for a preset model to obtain a trained model (i.e., a target model). This effectively reduces the computing resources required for model training, thereby improving the training efficiency of the model and improving the ability of the trained model to solve long text tasks, laying the foundation for subsequently improving the reasoning efficiency of the model and reducing the cost required for model training and deployment.

[0107] Further, in a specific example, the predicted word-unit sequence includes the predicted word-unit output by the last layer of the N network layers included in the preset model, and the predicted word-unit output by the other layers of the N network layers except the last layer (for example, the predicted word-unit output by the last s (folded feature value) layer); for example, continuing with 4 network layers {L 0 ,L 1 ,L 2 ,L 3}, and the initial word sequence {x 0 ,x 1 ,x 2 ,x 3} is folded twice to obtain the first word sequence {x 0 ,x 2} and the second word sequence {x 1 ,x 3} as an example, Figure 8 As shown, the predicted word sequence includes the network layer L 3 The predicted word output and and contains the network layer L 2 The predicted word output and In this way, single-step multi-word prediction is achieved, which effectively improves the speed of model reasoning and prediction.

[0108] Furthermore, in one example, except for the last layer, the number of network layers used to output the predicted word unit is related to the folded feature value, for example, the number of network layers used to output the predicted word unit (for example, the value of j) is greater than the folded feature value. In other words, not all network layers need to output the final prediction result, only some network layers except the last layer (for example, the last s network layers) need to output the prediction result, thus laying a foundation for improving the reasoning efficiency of the model and reducing the cost required for model training and deployment.

[0109] For example, in one example, when j is greater than or equal to the difference between the total number of layers and the folded feature value s (for example, for j values ​​ranging from 0 to N-1, in this case, it can be specifically j>N-1-s), the output of the i-th position of the j-th layer is used for prediction:

[0110] x ((s―1)―(N―1―j))+(i+1)×s ;

[0111] Here, the value of i is related to the value of T and the value of the folding feature value.

[0112] Here, for the relevant description of the value of i, please refer to the above example, which will not be repeated here.

[0113] For example, continuing with Figure 8 For example, the total number of network layers contained in the preset model is 4, and the folded feature value is 2; at this time, i takes the value of 0 or 1. Further, the output of the i-th position of the j-th layer can be recorded as That is, when j>N―1―s, the output of the i-th position of the j-th layer is The expression for the predicted word is:

[0114]

[0115] Or, when j is between 0 and 3, the output of the i-th position of the j-th layer is The expression for predicting the word can be specifically as follows:

[0116]

[0117] That is to say, in the last s layers of the preset model (for example, from the N-1-s+1th layer to the N-1th layer, a total of s layers), each layer needs to predict the next word element of the input word element. For example, for the input at the i-th position, predict the word element required at the (i+1)th position.

[0118] It should be noted that The word x used for prediction ((s―1)―(N―1―j))+(i+1)×s , which can be specifically understood as: word element is the predicted word x ((s―1)―(N―1―j))+(i+1)×s―1 The next word of .

[0119] For example, continuing with Figure 8 For example, the predicted word sequence includes: the output of the 0th position of the 2nd layer (for predicting x 2 , that is, the predicted word x 1 The next word), the output of the first position of the second layer (for predicting x 4 , that is, the predicted word x 3 The next word), the output of the 0th position of the 3rd layer (for predicting x 3 , that is, the predicted word x 2 The next word), the output of the first position of the third layer (for predicting x 5 , that is, the predicted word x 4 the next word of ).

[0120] That is, in the output prediction stage of the disclosed solution, because an input position contains multiple compressed word units (for example, Figure 8 As shown, the input position 1 contains two words, namely word x 0 and word x 1 ), so in the output prediction stage, it is necessary to predict and output multiple words at the same input position. For example, in this stage, analogous to the way of sequential input by layer, when outputting, multiple words are predicted in sequence according to the order of layer depth and the original input order, thus completing the prediction reasoning.

[0121] In this way, the disclosed scheme provides a specific scheme for determining the word elements required to be predicted for the output position of the i-th position of the j-th layer. The scheme makes full use of the layer depth dimension of the network layer to determine the word elements required to be predicted for the outputs of different network layers, so as to complete the predictive reasoning. In this way, the reasoning and prediction efficiency of the model can be effectively improved, thereby laying the foundation for subsequently improving the reasoning ability of the model and reducing the cost required for model training and deployment.

[0122] Fig. 9 This is a schematic flow chart of a model reasoning method according to an embodiment of the present application. Figure 1 The method may be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0123] Further, the method includes at least part of the following contents. Fig. 9 As shown, including:

[0124] Step S901: Obtain an initial word sequence to be inferred.

[0125] Step S902: Based on the folding feature value used to fold the word-gram sequence, folding processing is performed on the initial word-gram sequence to be inferred, so as to obtain at least a first target word-gram sequence to be inferred.

[0126] Here, the initial word sequence to be inferred is represented by T 2 The first target word sequence to be inferred after folding contains T 2 Further, the sequence length of the first target word-gram sequence to be inferred is less than the sequence length of the initial word-gram sequence to be inferred.

[0127] Here, for an example of folding the initial word sequence to be inferred, see the above Figure 2 The corresponding examples will not be described in detail here.

[0128] Step S903: at least input the first target word-gram sequence to be inferred into the target model to obtain a target inference result.

[0129] Here, the target inference result is the predicted next word-gram sequence of the target word-gram sequence to be inferred.

[0130] In this way, the disclosed scheme can fold the initial word sequence to be inferred according to the folding feature value, and obtain a sequence in which the length of the word sequence after folding is less than the length of the initial word sequence to be inferred (that is, the first target word sequence to be inferred), so that the target model can perform model inference according to the word sequence after folding. In this way, the reasoning efficiency of the model is improved by compressing the input of the model, thereby effectively improving the user experience.

[0131] Furthermore, since the disclosed solution compresses the input length of the model, the disclosed solution can effectively reduce the computational complexity and the occupation of video memory resources, thereby reducing the cost required for model reasoning. In addition, the disclosed solution also effectively improves the model's ability to solve long text tasks (such as long text summarization and long text question answering).

[0132] In a specific example, the target model may be a large model, and further, a large language model. Alternatively, it may be other models, which are not limited in the present disclosure.

[0133] Furthermore, the target model is obtained by performing model training using any of the above-mentioned model training methods.

[0134] Further, in a specific example, the above-described folding processing of the initial word-gram sequence to be inferred to obtain at least a first target word-gram sequence to be inferred (for example, step S902) may specifically include: folding processing of the initial word-gram sequence to be inferred to obtain the first target word-gram sequence to be inferred and a second target word-gram sequence to be inferred.

[0135] Accordingly, the above step of inputting at least the first target word-gram sequence to be inferred into the target model to obtain the target inference result (for example, step S903) specifically includes:

[0136] Each word in the first target word-gram sequence to be inferred is used as the input of the first layer of the N target network layers contained in the target model, and at least each word in the second target word-gram sequence to be inferred is used as a partial input of other layers of the N target network layers except the first layer, so as to obtain a target reasoning result.

[0137] For example, in one example, each word in the first target word sequence to be inferred can be input to the corresponding position of the first layer of the N target network layers included in the target model according to the order of each word in the first target word sequence to be inferred. Here, "position" can specifically refer to the input position of the word. For a specific example, please refer to the example shown in Figure 5(a) above, which will not be repeated here.

[0138] Furthermore, the first target word sequence to be inferred includes t 21 word, the t 21 The word is the T 2 Further, the second target word sequence to be inferred includes the t covered by the displayed word after folding. 22 word, the t 22 The word is also the T 2 Here, t 21 +t 22 =T 1 .

[0139] Here, for the relevant contents of the first target word sequence to be inferred and the second target word sequence to be inferred, please refer to the above Figure 4 The examples shown are not repeated here.

[0140] In this way, the disclosed solution uses each word in the first target word sequence to be inferred after folding as the input of the first layer in the target model, and uses each word in the second target word sequence to be inferred as the input of other layers in the target model except the first layer. In this way, the integrity of the input information after the model input is compressed can be effectively guaranteed, thereby improving the speed of model reasoning while ensuring that the reasoning effect of the model is not affected.

[0141] Furthermore, in a specific example, the above-mentioned obtaining of the target reasoning result specifically includes:

[0142] Obtain the target inference result output by the last position of the last n target network layers among the N target network layers.

[0143] Here, in one example, the value of n is related to the folded eigenvalue s. Further, n=folded eigenvalue. At this time, the target inference result output by the last position of the last s target network layers in the N target network layers can be obtained.

[0144] Further, the target inference result is at least the next word of the last word in the predicted initial word sequence to be inferred. Further, the number of words included in the target inference result is also related to the folded feature value s, for example, the number of words included in the target inference result is equal to the folded feature value s.

[0145] For example, with 4 target network layers The target model composed of the initial word sequence to be inferred {x 0 ,x 1 ,x 2 ,x 3} is folded twice to obtain the first target word sequence to be inferred {x 0 ,x 2} and the second target word sequence to be inferred {x 1 ,x 3} as an example, at this time, if Fig.10 As shown, the target network layer included in the target model The last position of the input is x 2 , target network layer The last position of the input is x 3 At this time, the target model will be based on the target network layer The last position of the input x 2 and the target network layer The last position of the input x 3 , predict the word and word Here, the word Represents the predicted word x 3 The next word (for example, it can be word x 4 ), word Represents the predicted word x 4 The next word (for example, it can be word x 5 ).

[0146] In this way, the disclosed solution can quickly obtain the inference results of the target model, effectively saving the computing resources required for model inference, and can respond to users instantly and quickly give accurate inference results, thereby improving the user experience.

[0147] Furthermore, in one example, the N target network layers described above are connected in series; the output of the jth layer in the N target network layers is used as the input of the j+1th layer in the N target network layers. It should be noted that for the relevant content of this example, please refer to the example of Figure 5(b) above, which will not be repeated here. In this way, the input information can be feature extracted layer by layer, thereby capturing richer and deeper information, laying the foundation for improving the accuracy and robustness of model prediction.

[0148] In a specific example of the disclosed solution, the input of each layer in the N target network layers may be obtained in the following manner; the specific manner includes:

[0149] Based on the folded feature value and the position of the jth layer in the N target network layers, the input of the jth layer in the N target network layers is determined. Here, the value of j is related to N. For example, the value of j is a natural number greater than or equal to 0 and less than or equal to N-1.

[0150] Further, in an example, the above-described determining the input of the jth layer in the N target network layers based on the folded feature value and the position of the jth layer in the N target network layers may specifically include:

[0151] Based on the numerical relationship between the number of layers of the jth layer in the N target network layers and the folded feature value, the input of the jth layer in the N target network layers is determined.

[0152] That is to say, in this example, the input of the first layer (for example, when j is 0) of the N target network layers contained in the target model is the first target word sequence to be inferred, and the input of other layers except the first layer of the N target network layers (that is, when j is an integer greater than 0 and less than N) needs to be determined based on the numerical relationship between the number of layers of the jth layer in the N target network layers and the folded feature value. In this way, a refined scheme is provided for determining the input of other word sequences except the first target word sequence to be inferred in the initial word sequence to be inferred into the target model, which is simple and efficient.

[0153] In this way, the disclosed scheme provides a detailed scheme for determining how to input other word-meta sequences (that is, words covered by folding) except for the first target word-meta sequence to be inferred into the target model according to the degree of folding of the input information. The scheme is simple and efficient, and thus effectively avoids the information loss caused by the folding of the model input information, and balances the computational complexity of each target network layer, reducing the pressure on the server when the model processes information, thereby laying the foundation for accelerating the reasoning speed of the model and reducing the cost required for model reasoning.

[0154] Further, in a specific example, the input of the jth layer in the N target network layers can be obtained in the following manner; specifically, the above-described determination of the input of the jth layer in the N target network layers based on the numerical relationship between the number of layers of the jth layer in the N target network layers and the folded eigenvalue can specifically include at least one of the following two methods:

[0155] Method 1: Determine the input of the non-first layer (for example, when j takes the value of 1 to N-1) in the N target network layers; specifically, determine that when the number of layers of the j-th layer in the N target network layers (for example, which can be understood as the value of j) is less than the folded feature value (for example, in one example, when (layer depth - 1) is less than the folded feature value s), based on the implicit output result of the j-1-th layer in the N target network layers and at least one word in the second target word sequence to be inferred, obtain the input of the j-th layer in the N target network layers.

[0156] Here, for the relevant contents about the number of layers, layer depth and implicit output result of the jth layer in the N target network layers, refer to the above example, which will not be repeated here.

[0157] It should be noted that, in one example, j takes a value of 0 to N-1; further, the present disclosure is exemplified by taking j taking a value of 0 to N-1 as an example. It is understandable that j can also take a value of 1 to N, in which case the superscript can be adjusted accordingly based on the actual numerical value, and the present disclosure does not limit this.

[0158] It can be understood that, for the first layer in the N target network layers (for example, j takes the value of 0), its input is each word in the first target word sequence to be inferred; for the non-first layer in the N target network layers, it can be obtained based on method 1. For example, for j taking the value of 1 and the folding feature value taking the value of 2 as an example, at this time, j is less than s, or (layer depth-1) (for example, 2-1) is less than 2, then the input of the first layer in the N target network layers can be specifically: the implicit output result of the 0th layer in the N target network layers, and at least one word in the second target word sequence to be inferred (that is, a word covered by the folding process). In other words, in a scenario, for some layers that meet the conditions, in addition to using the implicit output result of the previous layer (such as the j-1th layer in the N target network layers) as the input of the next layer (such as the jth layer in the N target network layers), it is also necessary to additionally introduce the word units masked by the displayed word units after the folding process as another input of the next layer (such as the jth layer in the N target network layers). This lays the foundation for effectively avoiding the loss of original input information.

[0159] Further, in an example, the method described in the first embodiment above based on the implicit output result of the j-1th layer in the N target network layers and at least one word in the second target word-word sequence to be inferred, obtaining the input of the jth layer in the N target network layers may specifically include:

[0160] Based on the implicit output result of the i-th position of the j-1th layer in the N target network layers and word x j+i×s , get the input of the i-th position of the j-th layer in the N target network layers.

[0161] Here, j represents the number of layers; i represents the input position of the word unit, and its value is the same as that of T 2 The value of is related to the value of the folding feature value. For example, in one example, i is 0 to [(T 2 / s)―1]. s represents the folded eigenvalue.

[0162] Here, for the relevant content of this part, please refer to the example shown in Figure 6(a) above, which will not be repeated here.

[0163] In this way, the disclosed scheme additionally introduces at least one word in the second target word sequence to be inferred into the input of the target network layer that meets the layer depth requirements, thereby effectively avoiding the loss of original input information. In other words, while compressing the target model input, the disclosed scheme utilizes the depth dimension of the target network layer to obtain the specific position of the compressed word required for input, thereby effectively avoiding the loss of original input information, thereby further improving the reasoning speed of the model while ensuring the reasoning effect of the model.

[0164] Further, in a specific example, the implicit output result based on the i-th position of the j-1th layer in the N target network layers described above is and word x j+i×s , obtaining the input of the j-th layer and the i-th position of the N target network layers may specifically include:

[0165] Using the deep folding function, the implicit output result of the i-th position of the j-1th layer in the N target network layers is and word x j+i×s The fusion is performed to use the fused result as the input of the i-th position of the j-th layer in the N target network layers.

[0166] For example, in one example, the input of the i-th position of the j-th layer in the N target network layers can be recorded as Let the depth folding function be represents a sequence deep folding function, which is used to fuse the implicit output of the previous layer (for example, represented by a vector) with the additional compressed input x j+i×s ,for example, The function can simply implement the bitwise addition operation.

[0167] At this time, when the number of layers in the jth layer of the N target network layers is less than the folded eigenvalue s (for example, when (layer depth - 1) is less than the folded eigenvalue s), the input of the i-th position of the jth layer in the N target network layers is It can be expressed as:

[0168]

[0169] In this way, the disclosed scheme provides a specific scheme for determining the input of the i-th position of the j-th layer in N target network layers. In this way, the scheme can be used to quickly determine the word-grams in the second target word-gram sequence to be additionally introduced for the input of the i-th position of the j-th layer in the N target network layers, thereby effectively ensuring that the compressed word-gram information is not lost. In this way, while ensuring the reasoning effect of the model, the reasoning efficiency of the model is further improved.

[0170] Method 2: Determine the input of the non-first layer in the N target network layers (for example, when j takes the value of 1 to N-1); specifically, determine that the number of layers of the j-th layer in the N target network layers is greater than or equal to the folded eigenvalue (for example, in one example, when (layer depth-1) is greater than or equal to the folded eigenvalue s), then obtain the input of the i-th position of the j-th layer in the N target network layers based on the implicit output result of the j-1-th layer in the N target network layers.

[0171] That is to say, in this example, when the number of layers of the jth (j is an integer greater than 0 and less than N) layer in the N target network layers is greater than or equal to the folded feature value (for example, (layer depth - 1) is greater than or equal to the folded feature value s), the input of the i-th position of the j-th layer in the N target network layers can be directly obtained based on the implicit output result of the j-1th layer in the N target network layers, without the need to additionally introduce the word elements of the second target word element sequence to be inferred. For example, in one example, when the number of layers of the jth (j is an integer greater than 0 and less than N) layer in the N target network layers is greater than or equal to the folded feature value (for example, (layer depth - 1) is greater than or equal to the folded feature value s), the implicit output result of the i-th position of the j-1th layer in the N target network layers can be directly obtained. As the input of the i-th position of the j-th layer in the N target network layers. That is, when the number of layers is greater than or equal to the folding feature value (also known as the sequence depth folding multiple) s, the input of the i-th position of the j-th layer is the output of the previous layer, and no additional folded and compressed words need to be fused.

[0172] Here, the relevant content of this example can be referred to the example shown in Figure 6(b), which will not be repeated here.

[0173] It should be noted that the disclosed solution linearly compresses the input length that the model needs to receive, and in order to ensure that the information after the input compression is not lost, the compressed words are sequentially used as additional inputs for other layers except the first layer in the original order of input. This input method fully utilizes the layer depth dimension, effectively reduces the input length, and avoids information loss.

[0174] In this way, the disclosed scheme determines the specific scheme for inputting the i-th position of the j-th layer in N target network layers. The scheme makes full use of the layer depth dimension of the target network layer to determine the specific information required for inputting different target network layers. In this way, while effectively compressing the model input, the loss of original input information caused by compression is avoided, thereby further improving the reasoning speed of the model while ensuring the reasoning effect of the model.

[0175] In summary, the disclosed solution has the following advantages:

[0176] First, it is more efficient. Compared with the improved solution of the efficient Transformer structure, the disclosed solution is simple to implement, and in actual application scenarios, it can improve the efficiency of model training and reasoning, and the disclosed solution will not be affected by the compression of input information. Therefore, the disclosed solution supports unlimited long text input;

[0177] Second, both training and reasoning are highly efficient. Compared with the efficient reasoning solution in low-resource scenarios, the disclosed solution can not only accelerate the training phase, but also accelerate the efficiency of the reasoning phase, unify training and reasoning, and thus ensure better results.

[0178] The disclosed solution provides a model training device, such as Fig.11 As shown, including:

[0179] The first data processing unit 1101 is used to fold the initial word-gram sequence used for model training based on the folding feature value used for folding the word-gram sequence, so as to obtain at least a first word-gram sequence after folding; wherein the initial word-gram sequence represents T 1 The first word sequence is composed of word units; the sequence length of the first word sequence is less than the sequence length of the initial word sequence;

[0180] The model training unit 1102 is used to input at least the first word-gram sequence into a preset model to perform model training on the preset model to obtain a target model.

[0181] In a specific example of the disclosed solution, the first data processing unit is specifically used to: perform folding processing on the initial word-gram sequence to obtain the first word-gram sequence and the second word-gram sequence, wherein the first word-gram sequence includes the t displayed after the folding processing. 11 word units, the second word unit sequence includes t that is covered by the displayed word units after folding. 12 word unit;

[0182] The model training unit is specifically used to use each word in the first word sequence as the input of the first layer of N network layers contained in the preset model, and at least use each word in the second word sequence as a partial input of other layers in the N network layers except the first layer, so as to adjust at least part of the network parameters in the N network layers to obtain the target model.

[0183] In a specific example of the disclosed solution, the N network layers are connected in series; the output of the jth layer among the N network layers serves as the input of the j+1th layer among the N network layers.

[0184] In a specific example of the present disclosure, the first data processing unit is further used to determine the input of the jth layer based on the folded feature value and the position of the jth layer in the N network layers; wherein the value of j is related to N.

[0185] In a specific example of the disclosed solution, the first data processing unit is specifically configured to include:

[0186] Based on the numerical relationship between the number of layers of the j-th layer in the N network layers and the folded feature value, the input of the j-th layer is determined.

[0187] In a specific example of the disclosed solution, the first data processing unit is specifically used to:

[0188] When determining that the number of layers of the j-th layer in the N network layers is less than the folded feature value, obtaining an input of the j-th layer based on an implicit output result of the j-1-th layer and at least one word in the second word-word sequence;

[0189] or,

[0190] When it is determined that the number of layers of the j-th layer in the N network layers is greater than or equal to the folded feature value, an input of the i-th position of the j-th layer is obtained based on an implicit output result of the j-1-th layer.

[0191] In a specific example of the disclosed solution, the first data processing unit is specifically used to:

[0192] Implicit output result based on the i-th position of the j-1th layer and word x j+i×s , get the input of the i-th position of the j-th layer, where the value of i is related to the T 1 The value of and the value of the folding characteristic value are related, and s represents the folding characteristic value.

[0193] In a specific example of the disclosed solution, the first data processing unit is specifically used to:

[0194] Using the deep folding function, the implicit output result of the i-th position of the j-1th layer is and word x j+i×s The fusion is performed to use the fused result as the input of the i-th position of the j-th layer.

[0195] In a specific example of the disclosed solution, the first data processing unit is specifically used to:

[0196] The implicit output result of the i-th position in the j-1th layer As the input of the i-th position of the j-th layer.

[0197] In a specific example of the disclosed solution, the model training unit is specifically used to:

[0198] Using each word in the first word-unit sequence as an input to a first layer of N network layers included in a preset model, and using at least each word in the second word-unit sequence as a partial input to other layers of the N network layers except the first layer;

[0199] Obtaining the output predicted word-unit sequence for predicting the next word-unit of each word-unit in the initial word-unit sequence;

[0200] At least based on the degree of difference between the predicted word-unit sequence and the target sequence, at least part of the network parameters in the N network layers are adjusted to obtain a target model.

[0201] In a specific example of the disclosed scheme, the predicted word-gram sequence includes the predicted word-grams output by the last layer of the N network layers included in the preset model, and the predicted word-grams output by some of the layers other than the last layer of the N network layers.

[0202] In a specific example of the disclosed solution, except for the last layer, the number of network layers used for outputting predicted word units is related to the folded feature values.

[0203] In a specific example of the disclosed solution, the number of network layers used for outputting predicted word units is greater than the folded feature value.

[0204] In a specific example of the disclosed solution, when j is greater than or equal to the difference between the total number of layers and the folded feature value s, the predicted word element output at the i-th position of the j-th layer is used to predict:

[0205] x (( s―1)―(N―1―j))+(i+1)×s;

[0206] The value of i is related to the value of T and the value of the folding characteristic value.

[0207] The disclosed solution also provides a model reasoning device, such as Fig.12 As shown, including:

[0208] The second data processing unit 1201 is used to obtain an initial word-gram sequence to be inferred; based on the folding feature value used to fold the word-gram sequence, the initial word-gram sequence to be inferred is folded to obtain at least a first target word-gram sequence to be inferred; wherein the initial word-gram sequence to be inferred represents T 2 The first target word-gram sequence to be inferred is composed of word-grams; the sequence length of the first target word-gram sequence to be inferred is less than the sequence length of the initial word-gram sequence to be inferred;

[0209] The model inference unit 1202 is used to input at least a first target word-gram sequence to be inferred into a target model to obtain a target inference result, wherein the target inference result is a predicted next word-gram sequence of the target word-gram sequence to be inferred.

[0210] In a specific example of the disclosed solution, the second data processing unit is specifically used to: perform folding processing on the initial word-gram sequence to be inferred to obtain the first target word-gram sequence to be inferred and the second target word-gram sequence to be inferred, wherein the first target word-gram sequence to be inferred includes the t displayed after the folding processing. 21 word units, the second target word unit sequence to be inferred includes t units covered by the displayed word units after folding. 22 word unit;

[0211] The model inference unit is specifically used to include: using each word in the first target word-gram sequence to be inferred as the input of the first layer of N target network layers contained in the target model, and at least using each word in the second target word-gram sequence to be inferred as a partial input of other layers of the N target network layers except the first layer, so as to obtain a target inference result.

[0212] In a specific example of the disclosed solution, the N target network layers are connected in series; the output of the jth layer among the N target network layers serves as the input of the j+1th layer among the N target network layers.

[0213] In a specific example of the present disclosure, the second data processing unit is further used to determine the input of the jth layer in the N target network layers based on the folded feature value and the position of the jth layer in the N target network layers; wherein the value of j is related to N.

[0214] In a specific example of the disclosed solution, the second data processing unit is specifically used to:

[0215] Based on the numerical relationship between the number of layers of the jth layer in the N target network layers and the folded feature value, the input of the jth layer in the N target network layers is determined.

[0216] In a specific example of the disclosed solution, the second data processing unit is specifically used to:

[0217] When determining that the number of layers of the jth layer in the N target network layers is less than the folded feature value, obtaining the input of the jth layer in the N target network layers based on the implicit output result of the j-1th layer in the N target network layers and at least one word in the second target word-word sequence to be inferred;

[0218] or,

[0219] When it is determined that the number of layers of the j-th layer in the N target network layers is greater than or equal to the folded characteristic value, the input of the i-th position of the j-th layer in the N target network layers is obtained based on the implicit output result of the j-1-th layer in the N target network layers.

[0220] In a specific example of the disclosed solution, the second data processing unit is specifically used to:

[0221] Based on the implicit output result of the i-th position of the j-1th layer in the N target network layers and word x j+i×s , get the input of the jth position of the N target network layers, where the value of i is the same as the T 2 The value of and the value of the folding characteristic value are related, and s represents the folding characteristic value.

[0222] In a specific example of the disclosed solution, the second data processing unit is specifically used to:

[0223] Using the deep folding function, the implicit output result of the i-th position of the j-1th layer in the N target network layers is and word x j+i×s The fusion is performed to use the fused result as the input of the i-th position of the j-th layer in the N target network layers.

[0224] In a specific example of the disclosed solution, the second data processing unit is specifically used to:

[0225] The implicit output result of the i-th position of the j-1th layer in the N target network layers As the input of the i-th position of the j-th layer in the N target network layers.

[0226] In a specific example of the disclosed solution, the model reasoning unit is specifically used to:

[0227] Obtain a target inference result output by the last position of the last n target network layers among the N target network layers; wherein the target inference result is at least the next word of the last word in the predicted initial word sequence to be inferred.

[0228] For the description of the specific functions and examples of each unit of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0229] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0230] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0231] Fig.13 A schematic block diagram of an example electronic device 1300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0232] like Fig.13 As shown, the device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0233] A number of components in the device 1300 are connected to the I / O interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0234] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as a model training method or a model reasoning method. For example, in some embodiments, the model training method or the model reasoning method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the model training method or the model reasoning method described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to execute a model training method or a model reasoning method in any other appropriate manner (for example, by means of firmware).

[0235] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0236] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0237] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0238] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0239] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0240] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0241] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0242] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model training method, comprising: Based on the folding feature value used to fold the word-gram sequence, folding the initial word-gram sequence used for model training to obtain at least a first word-gram sequence after folding; wherein the initial word-gram sequence represents a word-gram sequence composed of T1 words; and the sequence length of the first word-gram sequence is less than the sequence length of the initial word-gram sequence; At least the first word-gram sequence is input into a preset model to perform model training on the preset model to obtain a target model.

2. The method according to claim 1, wherein: The folding process is performed on the initial word-gram sequence used for model training to obtain at least a first word-gram sequence after the folding process, including: The initial word sequence is folded to obtain the first word sequence and the second word sequence, wherein the first word sequence includes the t displayed after the folding process. 11 word units, the second word unit sequence includes t that is covered by the displayed word units after folding. 12 word unit; The step of inputting at least the first word sequence into a preset model to perform model training on the preset model to obtain a target model includes: Each word in the first word-gram sequence is used as the input of the first layer of N network layers included in the preset model, and at least each word in the second word-gram sequence is used as a partial input of other layers in the N network layers except the first layer, so as to adjust at least part of the network parameters in the N network layers to obtain the target model.

3. The method according to claim 2, wherein: The N network layers are connected in series; the output of the jth layer among the N network layers serves as the input of the j+1th layer among the N network layers.

4. The method according to claim 2 or 3, further comprising: Based on the folded feature value and the position of the jth layer in the N network layers, the input of the jth layer is determined; wherein the value of j is related to the N.

5. The method according to claim 4, wherein: The step of determining the input of the jth layer based on the folded feature value and the position of the jth layer in the N network layers comprises: Based on the numerical relationship between the number of layers of the j-th layer in the N network layers and the folded feature value, the input of the j-th layer is determined.

6. The method according to claim 5, wherein: The step of determining the input of the j-th layer based on the numerical relationship between the number of layers of the j-th layer in the N network layers and the folded feature value comprises: When determining that the number of layers of the j-th layer in the N network layers is less than the folded feature value, obtaining an input of the j-th layer based on an implicit output result of the j-1-th layer and at least one word in the second word-word sequence; or, When it is determined that the number of layers of the j-th layer in the N network layers is greater than or equal to the folded feature value, an input of the i-th position of the j-th layer is obtained based on an implicit output result of the j-1-th layer.

7. The method according to claim 6, wherein: The step of obtaining the input of the jth layer based on the implicit output result of the j-1th layer and at least one word in the second word sequence includes: Implicit output result based on the i-th position of the j-1th layer and word x j+i×s , obtain the input of the i-th position of the j-th layer, where the value of i is related to the value of T1 and the value of the folding feature value, and s represents the folding feature value.

8. The method according to claim 7, wherein: The implicit output result based on the i-th position of the j-1th layer and word x j+i×s , get the input of the i-th position of the j-th layer, including: Using the deep folding function, the implicit output result of the i-th position of the j-1th layer is and word x j+i×s The fusion is performed to use the fused result as the input of the i-th position of the j-th layer.

9. The method according to claim 6, wherein: The step of obtaining the input of the i-th position of the j-th layer based on the implicit output result of the j-1-th layer includes: The implicit output result of the i-th position in the j-1th layer As the input of the i-th position of the j-th layer.

10. The method according to any one of claims 2 to 9, wherein: The step of using each word in the first word-gram sequence as an input of a first layer of N network layers included in a preset model, and at least using each word in the second word-gram sequence as a partial input of other layers of the N network layers except the first layer, so as to adjust at least part of the network parameters of the N network layers to obtain a target model, includes: Using each word in the first word-unit sequence as an input to a first layer of N network layers included in a preset model, and using at least each word in the second word-unit sequence as a partial input to other layers of the N network layers except the first layer; Obtaining the output predicted word-unit sequence for predicting the next word-unit of each word-unit in the initial word-unit sequence; At least based on the degree of difference between the predicted word-unit sequence and the target sequence, at least part of the network parameters in the N network layers are adjusted to obtain a target model.

11. The method according to claim 10, wherein: The predicted word-gram sequence includes the predicted word-gram output by the last layer of the N network layers included in the preset model, and the predicted word-gram output by some layers of the N network layers except the last layer.

12. The method according to claim 10, wherein: Except for the last layer, the number of network layers used to output predicted tokens is related to the folded feature values.

13. The method according to claim 12, wherein: The number of network layers used to output predicted tokens is greater than the folded feature value.

14. The method according to claim 13, wherein: When j is greater than or equal to the difference between the total number of layers and the folded feature value s, the predicted word output at the i-th position of the j-th layer is used for prediction: x (( s―1)―(N―1―j))+(i+1)×s; The value of i is related to the value of T1 and the value of the folding characteristic value.

15. A model reasoning method, comprising: Obtain an initial sequence of tokens to be inferred; Based on the folding feature value used to fold the word-gram sequence, the initial word-gram sequence to be inferred is folded to obtain at least a first target word-gram sequence to be inferred; wherein the initial word-gram sequence to be inferred represents a word-gram sequence composed of T2 words; and the sequence length of the first target word-gram sequence to be inferred is less than the sequence length of the initial word-gram sequence to be inferred; At least a first target word-gram sequence to be inferred is input into a target model to obtain a target inference result, wherein the target inference result is a predicted next word-gram sequence of the target word-gram sequence to be inferred.

16. The method according to claim 15, wherein: The folding process is performed on the initial word-gram sequence to be inferred to obtain at least a first target word-gram sequence to be inferred, including: The initial word-gram sequence to be inferred is folded to obtain the first target word-gram sequence to be inferred and the second target word-gram sequence to be inferred, wherein the first target word-gram sequence to be inferred includes the t displayed after the folding process. 21 word units, the second target word unit sequence to be inferred includes t units covered by the displayed word units after folding. 22 word unit; The step of inputting at least the first target word-meta sequence to be inferred into the target model to obtain a target inference result includes: Each word in the first target word-gram sequence to be inferred is used as the input of the first layer of the N target network layers contained in the target model, and at least each word in the second target word-gram sequence to be inferred is used as a partial input of other layers of the N target network layers except the first layer, so as to obtain a target reasoning result.

17. The method according to claim 16, wherein: The N target network layers are connected in series; the output of the jth layer among the N target network layers is used as the input of the j+1th layer among the N target network layers.

18. The method according to claim 16 or 17, further comprising: Based on the folded feature value and the position of the jth layer in the N target network layers, the input of the jth layer in the N target network layers is determined; wherein the value of j is related to the N.

19. The method according to claim 18, wherein: The step of determining the input of the jth layer among the N target network layers based on the folded feature value and the position of the jth layer among the N target network layers includes: Based on the numerical relationship between the number of layers of the jth layer in the N target network layers and the folded feature value, the input of the jth layer in the N target network layers is determined.

20. The method according to claim 19, wherein: The step of determining the input of the jth layer in the N target network layers based on the numerical relationship between the number of layers of the jth layer in the N target network layers and the folded feature value comprises: When determining that the number of layers of the jth layer in the N target network layers is less than the folded feature value, obtaining the input of the jth layer in the N target network layers based on the implicit output result of the j-1th layer in the N target network layers and at least one word in the second target word-word sequence to be inferred; or, When it is determined that the number of layers of the j-th layer in the N target network layers is greater than or equal to the folded characteristic value, the input of the i-th position of the j-th layer in the N target network layers is obtained based on the implicit output result of the j-1-th layer in the N target network layers.

21. The method according to claim 20, wherein: The step of obtaining the input of the jth layer among the N target network layers based on the implicit output result of the j-1th layer among the N target network layers and at least one word in the second target word-word sequence to be inferred includes: Based on the implicit output result of the i-th position of the j-1th layer in the N target network layers and word x j+i×s , obtain the input of the i-th position of the j-th layer in the N target network layers, where the value of i is related to the value of T2 and the value of the folding feature value, and s represents the folding feature value.

22. The method according to claim 21, wherein: The implicit output result based on the i-th position of the j-1th layer in the N target network layers and word x j+i×s , obtaining the input of the j-th layer i-th position of the N target network layers, including: Using the deep folding function, the implicit output result of the i-th position of the j-1th layer in the N target network layers is and word x j+i×s The fusion is performed to use the fused result as the input of the i-th position of the j-th layer in the N target network layers.

23. The method according to claim 20, wherein: The step of obtaining the input of the i-th position of the j-th layer in the N target network layers based on the implicit output result of the j-1-th layer in the N target network layers includes: The implicit output result of the i-th position of the j-1th layer in the N target network layers As the input of the i-th position of the j-th layer in the N target network layers.

24. The method according to claim 16, wherein: The obtaining of the target reasoning result includes: Obtain a target inference result output by the last position of the last n target network layers among the N target network layers; wherein the target inference result is at least the next word of the last word in the predicted initial word sequence to be inferred.

25. A model training device, comprising: A first data processing unit is used to fold an initial word-gram sequence for model training based on a folding feature value for folding a word-gram sequence, so as to obtain at least a first word-gram sequence after folding; wherein the initial word-gram sequence represents a word-gram sequence composed of T1 words; and the sequence length of the first word-gram sequence is less than the sequence length of the initial word-gram sequence; The model training unit is used to input at least the first word-unit sequence into a preset model to perform model training on the preset model to obtain a target model.

26. The device according to claim 25, wherein The first data processing unit is specifically configured to: perform folding processing on the initial word-gram sequence to obtain the first word-gram sequence and the second word-gram sequence, wherein the first word-gram sequence includes the t displayed after the folding processing. 11 word units, the second word unit sequence includes t units that are covered by the displayed word units after folding. 12 word unit; The model training unit is specifically used to use each word in the first word sequence as the input of the first layer of N network layers contained in the preset model, and at least use each word in the second word sequence as a partial input of other layers in the N network layers except the first layer, so as to adjust at least part of the network parameters in the N network layers to obtain the target model.

27. The device according to claim 26, wherein: The N network layers are connected in series; the output of the jth layer among the N network layers serves as the input of the j+1th layer among the N network layers.

28. The device according to claim 26 or 27, wherein The first data processing unit is further used to determine the input of the jth layer based on the folded feature value and the position of the jth layer in the N network layers; wherein the value of j is related to N.

29. The device according to claim 28, wherein The first data processing unit is specifically used to include: Based on the numerical relationship between the number of layers of the j-th layer in the N network layers and the folded feature value, the input of the j-th layer is determined.

30. The device according to claim 29, wherein: The first data processing unit is specifically used for: When determining that the number of layers of the j-th layer in the N network layers is less than the folded feature value, obtaining an input of the j-th layer based on an implicit output result of the j-1-th layer and at least one word in the second word-word sequence; or, When it is determined that the number of layers of the j-th layer in the N network layers is greater than or equal to the folded feature value, an input of the i-th position of the j-th layer is obtained based on an implicit output result of the j-1-th layer.

31. The device according to claim 30, wherein The first data processing unit is specifically used for: Implicit output result based on the i-th position of the j-1th layer and word x j+i×s , obtain the input of the i-th position of the j-th layer, where the value of i is related to the value of T1 and the value of the folding feature value, and s represents the folding feature value.

32. The device according to claim 31, wherein The first data processing unit is specifically used for: Using the deep folding function, the implicit output result of the i-th position of the j-1th layer is and word x j+i×s The fusion is performed to use the fused result as the input of the i-th position of the j-th layer.

33. The device according to claim 30, wherein: The first data processing unit is specifically used for: The implicit output result of the i-th position in the j-1th layer As the input of the i-th position of the j-th layer.

34. The device according to any one of claims 26 to 33, wherein: The model training unit is specifically used for: Using each word in the first word-unit sequence as an input to a first layer of N network layers included in a preset model, and using at least each word in the second word-unit sequence as a partial input to other layers of the N network layers except the first layer; Obtaining the output predicted word-unit sequence for predicting the next word-unit of each word-unit in the initial word-unit sequence; At least based on the degree of difference between the predicted word-unit sequence and the target sequence, at least part of the network parameters in the N network layers are adjusted to obtain a target model.

35. The device according to claim 34, wherein The predicted word-gram sequence includes the predicted word-gram output by the last layer of the N network layers included in the preset model, and the predicted word-gram output by some layers of the N network layers except the last layer.

36. The device according to claim 34, wherein Except for the last layer, the number of network layers used to output predicted tokens is related to the folded feature values.

37. The device according to claim 36, wherein The number of network layers used to output predicted tokens is greater than the folded feature value.

38. The device according to claim 37, wherein When j is greater than or equal to the difference between the total number of layers and the folded feature value s, the predicted word output at the i-th position of the j-th layer is used for prediction: x ((s―1)―(N―1―j))+(i+1)×s ； The value of i is related to the value of T1 and the value of the folding characteristic value.

39. A model reasoning device, comprising: A second data processing unit is used to obtain an initial word sequence to be inferred; Based on the folding feature value used to fold the word-gram sequence, the initial word-gram sequence to be inferred is folded to obtain at least a first target word-gram sequence to be inferred; wherein the initial word-gram sequence to be inferred represents a word-gram sequence composed of T2 words; and the sequence length of the first target word-gram sequence to be inferred is less than the sequence length of the initial word-gram sequence to be inferred; The model inference unit is used to input at least a first target word-gram sequence to be inferred into a target model to obtain a target inference result, wherein the target inference result is a predicted next word-gram sequence of the target word-gram sequence to be inferred.

40. The device according to claim 39, wherein The second data processing unit is specifically used to: perform folding processing on the initial word-gram sequence to be inferred to obtain the first target word-gram sequence to be inferred and the second target word-gram sequence to be inferred, wherein the first target word-gram sequence to be inferred includes the t displayed after the folding processing. 21 word units, the second target word unit sequence to be inferred includes t units covered by the displayed word units after folding. 22 word unit; The model inference unit is specifically used to include: using each word in the first target word-gram sequence to be inferred as the input of the first layer of N target network layers contained in the target model, and at least using each word in the second target word-gram sequence to be inferred as a partial input of other layers of the N target network layers except the first layer, so as to obtain a target inference result.

41. The device according to claim 40, wherein The N target network layers are connected in series; the output of the jth layer among the N target network layers is used as the input of the j+1th layer among the N target network layers.

42. The device according to claim 40 or 41, wherein The second data processing unit is further used to determine the input of the jth layer in the N target network layers based on the folded feature value and the position of the jth layer in the N target network layers; wherein the value of j is related to N.

43. The device according to claim 42, wherein: The second data processing unit is specifically used for: Based on the numerical relationship between the number of layers of the jth layer in the N target network layers and the folded feature value, the input of the jth layer in the N target network layers is determined.

44. The device according to claim 43, wherein The second data processing unit is specifically used for: When determining that the number of layers of the jth layer in the N target network layers is less than the folded feature value, obtaining the input of the jth layer in the N target network layers based on the implicit output result of the j-1th layer in the N target network layers and at least one word in the second target word-word sequence to be inferred; or, When it is determined that the number of layers of the j-th layer in the N target network layers is greater than or equal to the folded characteristic value, the input of the i-th position of the j-th layer in the N target network layers is obtained based on the implicit output result of the j-1-th layer in the N target network layers.

45. The device according to claim 44, wherein The second data processing unit is specifically used for: Based on the implicit output result of the i-th position of the j-1th layer in the N target network layers and word x j+i×s , obtain the input of the i-th position of the j-th layer in the N target network layers, where the value of i is related to the value of T2 and the value of the folding feature value, and s represents the folding feature value.

46. ​​The device according to claim 45, wherein The second data processing unit is specifically used for: Using the deep folding function, the implicit output result of the i-th position of the j-1th layer in the N target network layers is and word x j+i×s The fusion is performed to use the fused result as the input of the i-th position of the j-th layer in the N target network layers.

47. The apparatus of claim 44, wherein: The second data processing unit is specifically used for: The implicit output result of the i-th position of the j-1th layer in the N target network layers As the input of the i-th position of the j-th layer in the N target network layers.

48. The apparatus of claim 40, wherein: The model reasoning unit is specifically used for: Obtain a target inference result output by the last position of the last n target network layers among the N target network layers; wherein the target inference result is at least the next word of the last word in the predicted initial word sequence to be inferred.

49. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 24.

50. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-24.

51. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 24.

Images