Model training method, model inference method, and device thereof

Abstract

The present disclosure provides a model training method, a model inference method and apparatuses and devices thereof, relating to the technical field of data processing, and particularly to the technical field of artificial intelligence, big data, deep learning, and large models. The specific implementation scheme is as follows: based on a folding eigenvalue used for folding a token sequence, an initial token sequence used for model training is subjected to folding processing to obtain at least a first token sequence after folding processing; the initial token sequence represents a token sequence composed of T1 tokens; the sequence length of the first token sequence is less than the sequence length of the initial token sequence; and at least the first token sequence is input into a preset model to perform model training on the preset model to obtain a target model.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to technical fields such as artificial intelligence, big data, deep learning, and large models. Background Art

[0002] Currently, the field of natural language processing is entering an era of ultra-large-scale models. By leveraging ultra-high computing power to train models with extremely large parameters on massive amounts of text data, the resulting language models possess general semantic understanding and generation capabilities for multi-task, few-shot learning. While these large models demonstrate powerful general capabilities, their computational overhead and graphics memory usage increase quadratically with the input length, resulting in significant costs for model training and deployment. Furthermore, they limit their ability to solve long-text tasks. Summary of the Invention

[0003] The present disclosure provides a model training method, a model reasoning method, and a device and equipment thereof.

[0004] According to one aspect of the present disclosure, a model training method is provided, comprising:

[0005] Based on the folding feature value used to fold the word-gram sequence, folding the initial word-gram sequence used for model training to obtain at least a first word-gram sequence after folding; wherein the initial word-gram sequence represents a word-gram sequence consisting of T1 words; and the sequence length of the first word-gram sequence is less than the sequence length of the initial word-gram sequence;

[0006] At least the first word sequence is input into a preset model to perform model training on the preset model to obtain a target model.

[0007] According to another aspect of the present disclosure, a model reasoning method is provided, comprising:

[0008] Obtain the initial word sequence to be inferred;

[0009] Based on the folding feature value used to fold the word-gram sequence, folding the initial word-gram sequence to be inferred to obtain at least a first target word-gram sequence to be inferred; wherein the initial word-gram sequence to be inferred represents a word-gram sequence consisting of T2 words; and the sequence length of the first target word-gram sequence to be inferred is less than the sequence length of the initial word-gram sequence to be inferred;

[0010] At least a first target word-gram sequence to be inferred is input into a target model to obtain a target inference result, wherein the target inference result is a predicted next word-gram sequence of the target word-gram sequence to be inferred.

[0011] According to another aspect of the present disclosure, a model training device is provided, comprising:

[0012] a first data processing unit configured to fold an initial word-gram sequence for model training based on a folding feature value for folding the word-gram sequence to obtain at least a first word-gram sequence after folding; wherein the initial word-gram sequence represents a word-gram sequence consisting of T1 words; and the sequence length of the first word-gram sequence is less than the sequence length of the initial word-gram sequence;

[0013] The model training unit is used to input at least the first word-gram sequence into a preset model to perform model training on the preset model to obtain a target model.

[0014] According to another aspect of the present disclosure, there is provided a model reasoning device, comprising:

[0015] A second data processing unit is configured to obtain an initial word-gram sequence to be inferred; and fold the initial word-gram sequence to be inferred based on a folding feature value for folding the word-gram sequence to obtain at least a first target word-gram sequence to be inferred; wherein the initial word-gram sequence to be inferred represents a word-gram sequence consisting of T2 word-grams; and the sequence length of the first target word-gram sequence to be inferred is less than the sequence length of the initial word-gram sequence to be inferred;

[0016] The model inference unit is used to input at least a first target word-gram sequence to be inferred into a target model to obtain a target inference result, wherein the target inference result is the predicted next word-gram sequence of the target word-gram sequence to be inferred.

[0017] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method in the embodiments of the present disclosure.

[0021] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0022] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements any one of the methods according to the embodiments of the present disclosure.

[0023] In this way, the disclosed solution can fold the initial word sequence according to the folding feature value, and obtain a sequence in which the length of the folded word sequence is less than the length of the initial word sequence (that is, the first word sequence), and then use the folded word sequence to train the preset model. In this way, the training efficiency of the model is improved by compressing the input of the model, laying the foundation for subsequently improving the efficiency of model reasoning.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0026] Figure 1 This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 1 ;

[0027] Figure 2 This is a schematic diagram of an initial word sequence before and after folding in an example according to an embodiment of the present application. Figure 1 ;

[0028] Figure 3 This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 2 ;

[0029] Figure 4 This is a schematic diagram of an initial word sequence before and after folding in an example according to an embodiment of the present application. Figure 2 ;

[0030] FIG5( a ) is a schematic diagram illustrating preset model input information according to an embodiment of the present application;

[0031] FIG5( b ) is a schematic diagram of the structure of N network layers in a preset model according to an embodiment of the present application;

[0032] FIG6( a ) is a schematic diagram illustrating input information of a non-first layer in a preset model according to an embodiment of the present application;

[0033] FIG6( b ) is a schematic diagram illustrating input information of each layer in a preset model according to an embodiment of the present application;

[0034] Figure 7 This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 3 ;

[0035] Figure 8is a schematic diagram illustrating output information of part of the network layer in a preset model according to an embodiment of the present application;

[0036] Figure 9 is a schematic flow chart of a model reasoning method according to an embodiment of the present application;

[0037] Figure 10 This is a schematic diagram of a prediction scenario in a specific example of a model reasoning method according to an embodiment of the present application;

[0038] Figure 11 is a structural diagram of a model training device according to an embodiment of the present application;

[0039] Figure 12 is a structural diagram of a model reasoning device according to an embodiment of the present application;

[0040] Figure 13 It is a block diagram of an electronic device used to implement the model training method or model inference method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0042] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them, and do not mean to limit the order or to limit to only two. For example, the first feature and the second feature refer to two categories / two features. The first feature can be one or more, and the second feature can also be one or more.

[0043] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0044] This disclosure proposes a model training method to improve the training efficiency of large models.

[0045] Specifically, Figure 1 This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 1 The method may be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0046] Furthermore, the method includes at least part of the following contents. Figure 1 Shown, including:

[0047] Step S101: performing folding processing on an initial word-gram sequence used for model training based on a folding feature value used for folding the word-gram sequence, so as to obtain at least a first word-gram sequence after folding processing.

[0048] Here, the initial word-gram sequence is a word-gram sequence consisting of T1 words; the first word-gram sequence obtained after folding includes some of the words in T1 words. Furthermore, the sequence length of the first word-gram sequence is less than the sequence length of the initial word-gram sequence.

[0049] For example, in one example, the initial word sequence is folded in depth to obtain the first word sequence after folding. This effectively compresses the length of the input word. In other words, it effectively reduces the number of words first input to the preset model, thereby laying the foundation for improving the efficiency of subsequent model training and then improving the efficiency of model reasoning. Figure 2 As shown, the initial word sequence containing 4 words is recorded as {x0, x1, x2, x3}. Assume that the folding feature value is 2. At this time, the initial word sequence {x0, x1, x2, x3} is folded twice using the folding feature value, and the first word sequence after folding can be recorded as {x0, x2}.

[0050] Step S102: inputting at least the first word-gram sequence into a preset model to perform model training on the preset model to obtain a target model.

[0051] In this way, the disclosed solution can fold the initial word sequence according to the folding feature value, and obtain a sequence in which the length of the folded word sequence is less than the length of the initial word sequence (that is, the first word sequence), and then use the folded word sequence to train the preset model. In this way, the training efficiency of the model is improved by compressing the input of the model, laying the foundation for subsequently improving the efficiency of model reasoning.

[0052] Furthermore, because the disclosed solution compresses the model's input length, it can effectively reduce computational complexity and memory usage, thereby reducing the cost of model training and deployment. Furthermore, the disclosed solution effectively improves the model's ability to solve long-text tasks (such as long-text summarization and long-text question-answering).

[0053] In a specific example, the preset model can be a large model, and further, a large language model. Alternatively, it can be other models, which are not limited in the present disclosure.

[0054] Figure 3 This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 2 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters and other electronic devices. It is understood that the above Figure 1 and Figure 2 The relevant contents of the method shown can also be applied to this example, and this example will not elaborate on the relevant contents.

[0055] Furthermore, the method includes at least part of the following contents. Figure 3 Shown, including:

[0056] Step S301: Based on the folding feature value used for folding the word-gram sequence, folding processing is performed on the initial word-gram sequence used for model training to obtain a first word-gram sequence and a second word-gram sequence.

[0057] Here, the initial word-gram sequence represents a word-gram sequence composed of T1 words; the sequence length of the first word-gram sequence is shorter than the sequence length of the initial word-gram sequence.

[0058] Furthermore, the first word sequence includes t displayed after folding processing. 11 word, the t 11 The word-grams are part of the T1 word-grams; further, the second word-gram sequence includes the t word-grams covered by the displayed word-grams after folding. 12 word, the t 12 The word element is also part of the T1 word element. Here, t 11 +t 12 =T1.

[0059] For example, continue with Figure 2 For example, after folding the initial word sequence {x0,x1,x2,x3} by 2 times, we can get Figure 4The two words displayed after the folding process are shown, namely, the first word sequence {x0, x2}, and the two words hidden by the displayed words after the folding process are shown, namely, the second word sequence {x1, x3}.

[0060] Step S302: Use each word in the first word sequence as the input of the first layer of the N network layers included in the preset model, and use at least each word in the second word sequence as part of the input of other layers except the first layer of the N network layers, so as to adjust at least part of the network parameters in the N network layers to obtain a target model, for example, fine-tune at least part of the network parameters in the N network layers to obtain a target model.

[0061] For example, in one example, each word in the first word sequence is input into the corresponding position of the first layer in the preset model according to the order of each word in the first word sequence. Here, "position" can specifically refer to the input position of the word.

[0062] For example, continue with Figure 4 Taking the initial word sequence shown in the figure, and obtaining the first word sequence {x0, x2} and the second word sequence {x1, x3} as an example, as shown in Figure 5(a), according to the order of the words in the first word sequence {x0, x2}, the word x0 in the first word sequence {x0, x2} can be input to position 1 of the first layer contained in the preset model, and the word x2 in the first word sequence {x0, x2} can be input to position 2 of the first layer.

[0063] In this way, the disclosed solution uses each word in the first word sequence after folding as the input of the first layer in the preset model, and uses each word in the second word sequence as the input of other layers in the preset model except the first layer. In this way, the integrity of the input information after the model input is compressed can be effectively guaranteed, thereby improving the speed of model training while ensuring that the training effect of the model is not affected, laying the foundation for subsequently improving the reasoning efficiency of the model and reducing the cost required for model training and deployment.

[0064] Furthermore, in one example, the N network layers are connected in series; the output of the jth layer in the N network layers serves as the input of the j+1th layer in the N network layers. For example, as shown in FIG5(b), when j is an integer greater than or equal to 0 and less than N, the output of the jth layer in the N network layers contained in the preset model serves as the input of the j+1th layer in the N network layers, wherein when j is 0, the input of the 0th layer (also called the first layer) is the first word sequence; when j is N-1, the output of the N-1th layer is the output of the last layer in the preset model. In this way, the input information can be subjected to layer-by-layer feature extraction, thereby capturing richer and deeper information, laying the foundation for improving the accuracy and robustness of the model prediction.

[0065] In a specific example of the disclosed solution, the input of each of the N network layers may be obtained in the following manner; the specific manner includes:

[0066] Based on the folded feature value and the position of the jth layer in the N network layers, the input of the jth layer is determined; wherein the value of j is related to N. For example, the value of j is a natural number greater than or equal to 0 and less than or equal to N-1.

[0067] Furthermore, in one example, the above-mentioned determining the input of the j-th layer based on the folded feature value and the position of the j-th layer in the N network layers may specifically include:

[0068] An input of the j-th layer is determined based on a numerical relationship between the number of layers of the j-th layer in the N network layers and the folded feature value.

[0069] That is to say, in this example, the input of the first layer (for example, when j is 0) of the N network layers included in the preset model is the first word sequence, and the input of other layers except the first layer in the N network layers (that is, when j is an integer greater than 0 and less than N) needs to be determined based on the numerical relationship between the number of layers of the jth layer in the N network layers and the folded feature value. In this way, a refined scheme is provided for determining the input of other word sequences except the first word sequence in the initial word sequence into the preset model, which is simple and efficient.

[0070] In this way, the disclosed solution provides a detailed solution for determining how to input word sequences other than the first word sequence (i.e., words covered due to folding) into a preset model based on the degree of folding of the input information. The solution is simple and efficient, and thus effectively avoids information loss caused by folding of model input information. It also balances the computational complexity of each network layer, and reduces the pressure on the server when the model processes information, thereby laying the foundation for accelerating the training speed of the model and reducing the cost required for model training and deployment.

[0071] Furthermore, in a specific example, the input of the j-th layer can be obtained in the following manner. Specifically, the above-mentioned determination of the input of the j-th layer based on the numerical relationship between the number of layers of the j-th layer in the N network layers and the folded eigenvalue can specifically include at least one of the following two manners:

[0072] Method 1: Determine the input of the non-first layer (for example, when j takes the value of 1 to N-1); specifically, determine that the number of layers of the j-th layer in the N network layers (for example, which can be understood as the value of j) is less than the folded feature value (for example, in one example, when (layer depth - 1) is less than the folded feature value s), based on the implicit output result of the j-1-th layer and at least one word in the second word sequence, obtain the input of the j-th layer.

[0073] It should be noted that, in this example, the number of the jth layer in the N network layers may specifically refer to the value of j; further, the layer depth may specifically refer to the position in the total number of layers. For example, for the first layer in the 4 layers, its layer depth may specifically be 1, and the layer below the first layer may have a layer depth of 2.

[0074] Here, the implicit output result can be specifically understood as the implicit result obtained after the network layer processes the word unit.

[0075] It should be noted that in one example, j ranges from 0 to N-1. Furthermore, the present disclosure uses j ranging from 0 to N-1 as an example for illustrative purposes. It is understood that j can also range from 1 to N. In this case, the subscript can be adjusted accordingly based on the actual numerical value, and the present disclosure does not impose any restrictions on this.

[0076] It can be understood that for the first layer (for example, j takes the value of 0), its input is each word in the first word sequence; for non-first layers, it can be obtained based on method 1. For example, for j taking the value of 1 and the folding feature value taking the value of 2, at this time, j is less than s, or (layer depth-1) (for example, 2-1) is less than 2, then the input of the first layer can be specifically: the implicit output result of the 0th layer, and at least one word in the second word sequence (that is, a word covered by the folding process). In other words, in a scenario, for some layers that meet the conditions, in addition to using the implicit output result output by the previous layer (for example, the j-1th layer) as the input of the next layer (for example, the jth layer), it is also necessary to introduce the word covered by the displayed word after the folding process as another input of the next layer (for example, the jth layer). This lays the foundation for effectively avoiding the loss of original input information.

[0077] Furthermore, in one example, obtaining the input of the jth layer based on the implicit output result of the j-1th layer and at least one word in the second word sequence described in the first embodiment may specifically include:

[0078] Based on the implicit output result of the i-th position in the j-1th layer and word x j+i×s , get the input of the i-th position of the j-th layer.

[0079] Here, j represents the number of layers; i represents the input position of the word, and its value is related to the value of T1 and the value of the folding feature value. For example, in one example, i ranges from 0 to [(T1 / s) - 1]. s represents the folding feature value.

[0080] For example, take the preset model composed of 4 network layers (which can be recorded as {L0, L1, L2, L3}) as an example, and continue to take the initial word sequence {x0, x1, x2, x3} folded 2 times to obtain the first word sequence {x0, x2} and the second word sequence {x1, x3} as an example; at this time, j takes a value from 0 to 3, and i takes a value of 0 or 1. Further, as shown in Figure 6 (a), for the first layer, for example, it can be recorded as the 0th layer L0 (that is, j takes a value of 0), at this time, the input of the 0th position of the 0th layer L0 (i takes a value of 0) is x0, and the input of the 1st position of the 0th layer L0 (i takes a value of 1) is x2. Further, for the 1st layer L1, that is, j takes a value of 1, at this time, j is satisfied. <s,则可根据预设模型中的第0层L0第0位置的隐式输出结果 And the word x1 in the second word sequence, get the input of the 0th position of the first layer L0, accordingly, according to the implicit output result of the 1st position of the 0th layer L0 in the preset model And the word x3 in the second word sequence obtains the input of the first position of the first layer L1.

[0081] In this way, the disclosed solution additionally introduces at least one word in the second word sequence into the input of the network layer that meets the layer depth requirements, thereby effectively avoiding the loss of original input information. In other words, while compressing the preset model input, the disclosed solution utilizes the depth dimension of the network layer to obtain the specific position of the compressed word required for input, effectively avoiding the loss of original input information, and thus laying the foundation for subsequently improving the model's reasoning efficiency and reducing the cost required for model training and deployment while ensuring the model's training effect.

[0082] Furthermore, in a specific example, the implicit output result based on the i-th position of the j-1th layer described above is and word x j+i×s , to obtain the input of the i-th position of the j-th layer, which can specifically include:

[0083] Using the depth folding function, the implicit output result of the i-th position of the j-1th layer is and word x j+i×s The fusion is performed to use the fused result as the input of the i-th position of the j-th layer.

[0084] For example, in one example, the input at position i in layer j can be written as Let the depth folding function be Represents a sequence depth folding function that is used to fuse the implicit output of the previous layer (for example, represented by a vector) with the additional compressed input x j+i×s ,for example, The function can simply implement bitwise addition operation.

[0085] At this time, if the number of layers in the j-th layer in the N network layers is less than the folded eigenvalue s (for example, when (layer depth - 1) is less than the folded eigenvalue s), then the input of the i-th position in the j-th layer is It can be expressed as:

[0086]

[0087] In this way, the disclosed solution provides a specific solution for determining the input at the i-th position of the j-th layer. In this way, the solution can quickly determine the word in the second word sequence that needs to be additionally introduced for the input at the i-th position of the j-th layer, thereby effectively ensuring that the compressed word information is not lost. In this way, while ensuring the training effect of the model, it lays the foundation for subsequently improving the reasoning efficiency of the model and reducing the cost required for model training and deployment.

[0088] Method 2: Determine the input of the non-first layer (for example, when j takes the value of 1 to N-1); determine that the number of layers of the j-th layer in the N network layers is greater than or equal to the folded eigenvalue (for example, in one example, when (layer depth - 1) is greater than or equal to the folded eigenvalue s), and obtain the input of the i-th position of the j-th layer based on the implicit output result of the j-1-th layer.

[0089] That is to say, in this example, when the number of layers of the jth (j is an integer greater than 0 and less than N) layer in the N network layers is greater than or equal to the folded eigenvalue (for example, (layer depth - 1) is greater than or equal to the folded eigenvalue s), the input of the i-th position of the jth layer can be directly obtained based on the implicit output result of the j-1th layer, without the need to additionally introduce the word of the second word sequence. For example, in one example, when the number of layers of the jth (j is an integer greater than 0 and less than N) layer in the N network layers is greater than or equal to the folded eigenvalue (for example, (layer depth - 1) is greater than or equal to the folded eigenvalue s), the implicit output result of the i-th position of the j-1th layer can be directly obtained. As the input of the i-th position in the j-th layer.

[0090] For example, let's continue with the preset model consisting of four network layers {L0, L1, L2, L3}, and continue with the example of folding the initial word sequence {x0, x1, x2, x3} by 2 times to obtain the first word sequence {x0, x2} and the second word sequence {x1, x3}. In this case, j takes a value from 0 to 3, and i takes a value from 0 to 1. Furthermore, as shown in Figure 6(b), for the second layer L2, that is, j takes a value of 2. At this time, if j≥s is satisfied, the implicit output result of the 0th position of the first layer L1 can be Directly as the input of the 0th position of the second layer L2, and the implicit output result of the 1st position of the first layer L1 Directly as the input of the first position of the second layer L2; further, for the third layer L3, that is, j takes the value of 3, at this time, satisfying j ≥ s, then the implicit output result of the 0th position of the second layer L2 can be Directly as the input of the 0th position of the 3rd layer L3, and the implicit output result of the 1st position of the 2nd layer L2 Directly serve as the input of the first position of the third layer L3.

[0091] For example, in one example, the input of the i-th position in the j-th layer is At this time, if the number of layers (for example, the value of j) in the jth layer (not the first layer, for example, the value of j is an integer greater than 0 and less than N) in the N network layers is greater than or equal to the folded eigenvalue s, then the input of the i-th position of the jth layer is It can be expressed as:

[0092]

[0093] That is to say, when the number of layers is greater than or equal to the folding feature value (also called the sequence depth folding multiple) s, the input of the i-th position of the j-th layer is the output of the previous layer, and no additional folded and compressed words need to be fused.

[0094] It should be noted that this disclosed solution linearly compresses the input length required by the model. To ensure that information is not lost after the input compression, the compressed tokens are sequentially used as additional input for all layers beyond the first layer in their original input order. This input method fully utilizes the layer depth dimension, effectively reducing the input length and avoiding information loss.

[0095] In this way, the disclosed solution determines the specific scheme for inputting the i-th position of the j-th layer. The solution makes full use of the layer depth dimension of the network layer to determine the specific input information required for different network layers. In this way, while effectively compressing the model input, the loss of original input information caused by compression is avoided. Furthermore, while ensuring the training effect of the model, the training speed of the model is improved, laying the foundation for subsequently improving the reasoning ability of the model and reducing the cost required for model training and deployment.

[0096] Figure 7 This is a schematic flow chart of a model training method according to an embodiment of the present application. Figure 3 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters and other electronic devices. It is understood that the above Figure 1 The relevant contents of the method shown in FIG6 can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0097] Furthermore, the method includes at least part of the following contents. Figure 7 Shown, including:

[0098] Step S701: Based on the folding feature value used for folding the word-gram sequence, folding processing is performed on the initial word-gram sequence used for model training to obtain a first word-gram sequence and a second word-gram sequence.

[0099] Here, the initial word-gram sequence represents a word-gram sequence composed of T1 words; the sequence length of the first word-gram sequence is shorter than the sequence length of the initial word-gram sequence.

[0100] Furthermore, the first word sequence includes t displayed after folding processing. 11 word units, the second word unit sequence includes the t units covered by the displayed word units after folding. 12 A word.

[0101] Here, for the relevant description of the initial word sequence, the first word sequence and the second word sequence, please refer to the above examples and will not be repeated here.

[0102] Step S702: Use each word in the first word sequence as the input of the first layer of the N network layers included in the preset model, and use at least each word in the second word sequence as part of the input of other layers in the N network layers except the first layer.

[0103] Here, for the relevant description of the specific input of different network layers, please refer to the above examples and will not be repeated here.

[0104] Step S703: Obtain the output predicted word-gram sequence for predicting the next word-gram of each word-gram in the initial word-gram sequence.

[0105] Step S704: Based at least on the degree of difference between the predicted word sequence and the target sequence (e.g., a pre-labeled theoretical word sequence), adjust (e.g., fine-tune) at least some of the network parameters in the N network layers to obtain a target model.

[0106] In this way, the disclosed solution provides a refined training scheme for a preset model to obtain a trained model (i.e., a target model). This effectively reduces the computing resources required for model training, thereby improving the training efficiency of the model, and improving the ability of the trained model to solve long text tasks, laying the foundation for subsequently improving the model's reasoning efficiency and reducing the cost required for model training and deployment.

[0107] Furthermore, in a specific example, the predicted word-gram sequence includes the predicted word-gram output by the last layer of the N network layers included in the preset model, and the predicted word-gram output by the other layers of the N network layers except the last layer (for example, the predicted word-gram output by the last s (folded feature value) layer); for example, continuing with the preset model composed of 4 network layers {L0, L1, L2, L3}, and taking the initial word-gram sequence {x0, x1, x2, x3} being folded 2 times to obtain the first word-gram sequence {x0, x2} and the second word-gram sequence {x1, x3} as an example, as shown Figure 8 As shown, the predicted word sequence includes the predicted word output by the network layer L3 and And the predicted word unit output by network layer L2 and In this way, single-step multi-word prediction is achieved, which effectively improves the speed of model inference and prediction.

[0108] Furthermore, in one example, the number of network layers used to output predicted tokens, excluding the last layer, is related to the collapsed eigenvalue. For example, the number of network layers used to output predicted tokens (e.g., the value of j) is greater than the collapsed eigenvalue. This means that not all network layers need to output the final prediction result; only some network layers excluding the last layer (e.g., the last s network layers) are required to output the prediction result. This lays the foundation for improving the model's inference efficiency and reducing the cost of model training and deployment.

[0109] For example, in one example, when j is greater than or equal to the difference between the total number of layers and the folded feature value s (for example, for j values ​​ranging from 0 to N-1, in this case, it can be specifically j>N-1-s), the output of the i-th position of the j-th layer is used for prediction:

[0110] x ((s―1)―(N―1―j))+(i+1)×s ;

[0111] Here, the value of i is related to the value of T and the value of the folding feature value.

[0112] Here, for the relevant explanation of the value of i, please refer to the above example and will not be repeated here.

[0113] For example, continue with Figure 8 For example, the total number of network layers included in the preset model is 4, and the folded feature value is 2; at this time, i takes the value of 0 or 1. Further, the output of the i-th position of the j-th layer can be recorded as That is, when j>N―1―s, the output of the i-th position of the j-th layer is The expression for the predicted word is:

[0114]

[0115] Or, when j is between 0 and 3, the output of position i in layer j is The expression for the predicted word can be specifically:

[0116]

[0117] That is to say, in the last s layers of the preset model (for example, from the N-1-s+1 layer to the N-1 layer, a total of s layers), each layer needs to predict the next word element of the input word element. For example, for the input at the i-th position, predict the word element required at the (i+1)-th position.

[0118] It should be noted that The word x used for prediction ((s―1)―(N―1―j))+(i+1)×s , which can be specifically understood as: word element is the predicted word x ((s―1)―(N―1―j))+(i+1)×s―1 The next word of .

[0119] For example, continue with Figure 8 For example, the predicted word sequence includes: the output of the 0th position of the 2nd layer (used to predict x2, which is the next word of the predicted word x1), the output of the first position of the second layer (used to predict x4, which is the next word of the predicted word x3), the output of the 0th position of the 3rd layer (used to predict x3, which is the next word of the predicted word x2), the output of the first position of the third layer (used to predict x5, which is the next word of the predicted word x4).

[0120] That is, in the output prediction stage, the disclosed solution contains multiple compressed word units (e.g., Figure 8 As shown in the figure, input position 1 contains two tokens, token x0 and token x1. Therefore, the output prediction stage requires predicting multiple tokens at the same input position. For example, in this stage, analogous to the sequential input method of layers, the output is predicted by using the order of layer depth and following the original input order, thus completing the prediction inference.

[0121] In this way, the disclosed solution provides a specific solution for determining the word element required to be predicted for the output position of the i-th position of the j-th layer. The solution makes full use of the layer depth dimension of the network layer to determine the word element required to be predicted for the output of different network layers, thereby completing the predictive reasoning. In this way, the reasoning and prediction efficiency of the model can be effectively improved, thereby laying the foundation for subsequently improving the reasoning ability of the model and reducing the cost required for model training and deployment.

[0122] Figure 9 This is a schematic flow chart of the model reasoning method according to an embodiment of the present application. Figure 1 The method may be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0123] Furthermore, the method includes at least part of the following contents. Figure 9 Shown, including:

[0124] Step S901: Obtain an initial word sequence to be inferred.

[0125] Step S902: Based on the folding feature value used to fold the word-gram sequence, folding processing is performed on the initial word-gram sequence to be inferred, so as to obtain at least a first target word-gram sequence to be inferred.

[0126] Here, the initial to-be-inferred word-gram sequence represents a word-gram sequence consisting of T2 word-grams; the first target to-be-inferred word-gram sequence obtained after folding includes some of the T2 word-grams. Furthermore, the sequence length of the first target to-be-inferred word-gram sequence is less than the sequence length of the initial to-be-inferred word-gram sequence.

[0127] Here, for an example of folding the initial word sequence to be inferred, please refer to the above Figure 2 The corresponding examples will not be repeated here.

[0128] Step S903: Input at least the first target word-gram sequence to be inferred into the target model to obtain a target inference result.

[0129] Here, the target inference result is the predicted next word-gram sequence of the target word-gram sequence to be inferred.

[0130] In this way, the disclosed solution can fold the initial word sequence to be inferred according to the folding feature value, and obtain a sequence in which the length of the word sequence after folding is less than the length of the initial word sequence to be inferred (that is, the first target word sequence to be inferred), so that the target model can perform model inference based on the word sequence after folding. In this way, the reasoning efficiency of the model is improved by compressing the input of the model, thereby effectively improving the user experience.

[0131] Furthermore, because the disclosed solution compresses the model's input length, it can effectively reduce computational complexity and memory usage, thereby reducing the cost of model inference. Furthermore, the disclosed solution effectively enhances the model's ability to solve long-text tasks (such as long-text summarization and long-text question-answering).

[0132] In a specific example, the target model can be a large model, and further, a large language model. Alternatively, it can be other models, which are not limited in the present disclosure.

[0133] Furthermore, the target model is obtained by performing model training using any of the above-mentioned model training methods.

[0134] Further, in a specific example, the above-mentioned folding processing of the initial word sequence to be inferred to obtain at least the first target word sequence to be inferred (for example, step S902) can specifically include: folding processing of the initial word sequence to be inferred to obtain the first target word sequence to be inferred and the second target word sequence to be inferred.

[0135] Accordingly, the above-mentioned step of inputting at least the first target word-gram sequence to be inferred into the target model to obtain the target inference result (e.g., step S903) specifically includes:

[0136] Each word in the first target word-gram sequence to be inferred is used as the input of the first layer of the N target network layers contained in the target model, and at least each word in the second target word-gram sequence to be inferred is used as a partial input of other layers of the N target network layers except the first layer to obtain a target reasoning result.

[0137] For example, in one example, each word in the first target word sequence to be inferred can be input into the corresponding position of the first layer of the N target network layers included in the target model according to the order of the words in the first target word sequence to be inferred. Here, "position" can specifically refer to the input position of the word. A specific example can be referred to the example shown in Figure 5(a) above, and will not be repeated here.

[0138] Furthermore, the first target word sequence to be inferred includes t 21 word, the t 21 The word-grams are part of the T2 word-grams; further, the second target word-gram sequence to be inferred includes t words covered by the displayed word-grams after folding. 22 word, the t 22 The word element is also part of the T2 word elements. Here, t 21 +t 22 =T1.

[0139] Here, for the relevant content of the first target word sequence to be inferred and the second target word sequence to be inferred, please refer to the above Figure 4 The examples shown are not repeated here.

[0140] In this way, the disclosed solution uses each word in the first target word sequence to be inferred after folding as the input of the first layer in the target model, and uses each word in the second target word sequence to be inferred as the input of other layers in the target model except the first layer. In this way, the integrity of the input information after the model input is compressed can be effectively guaranteed, thereby improving the speed of model reasoning while ensuring that the reasoning effect of the model is not affected.

[0141] Furthermore, in a specific example, obtaining the target reasoning result described above specifically includes:

[0142] Obtain the target inference result output by the last position of the last n target network layers in the N target network layers.

[0143] Here, in one example, the value of n is related to the folded eigenvalue s. Further, n = folded eigenvalue. At this time, the target inference result output by the last position of the last s target network layers in the N target network layers can be obtained.

[0144] Furthermore, the target inference result is at least the next word-gram of the last word-gram in the predicted initial sequence of word-grams to be inferred. Furthermore, the number of word-grams included in the target inference result is also related to the collapsed feature value s. For example, the number of word-grams included in the target inference result is equal to the collapsed feature value s.

[0145] For example, with 4 target network layers The target model composed of , and taking the initial word sequence to be inferred {x0, x1, x2, x3} folded twice to obtain the first target word sequence to be inferred {x0, x2} and the second target word sequence to be inferred {x1, x3} as an example, at this time, if Figure 10 As shown, the target network layer included in the target model The input of the last position is x2, the target network layer The input of the last position is x3. At this time, the target model will be based on the target network layer. The last position of the input x2 and the target network layer The last position of the input x3 is used to predict the word and word element Here, the word Indicates the next word of the predicted word x3 (for example, it can be word x4), word Indicates the next word of the predicted word x4 (for example, it may be word x5).

[0146] In this way, the disclosed solution can quickly obtain the inference results of the target model, effectively saving the computing resources required for model inference. Moreover, it can respond to users instantly and quickly give accurate inference results, thereby improving the user experience.

[0147] Furthermore, in one example, the N target network layers described above are connected in series; the output of layer j in the N target network layers serves as the input to layer j+1 in the N target network layers. It should be noted that for details regarding this example, please refer to the example in Figure 5(b) above and will not be repeated here. This allows for layer-by-layer feature extraction of the input information, capturing richer and deeper information and laying the foundation for improving the accuracy and robustness of model predictions.

[0148] In a specific example of the disclosed solution, the input of each layer in the N target network layers can be obtained in the following manner; the specific manner includes:

[0149] Based on the folded feature value and the position of the jth layer in the N target network layers, the input of the jth layer in the N target network layers is determined. Here, the value of j is related to N. For example, the value of j is a natural number greater than or equal to 0 and less than or equal to N-1.

[0150] Further, in one example, the above-mentioned determining the input of the j-th layer in the N target network layers based on the folded feature value and the position of the j-th layer in the N target network layers may specifically include:

[0151] Based on a numerical relationship between the number of layers of the j-th layer in the N target network layers and the folded eigenvalue, an input of the j-th layer in the N target network layers is determined.

[0152] That is to say, in this example, the input of the first layer (for example, when j is 0) of the N target network layers contained in the target model is the first target word sequence to be inferred, and the input of other layers except the first layer in the N target network layers (that is, when j is an integer greater than 0 and less than N) needs to be determined based on the numerical relationship between the number of layers of the jth layer in the N target network layers and the folded feature value. In this way, a refined scheme is provided for determining the input of other word sequences except the first target word sequence to be inferred in the initial word sequence into the target model, which is simple and efficient.

[0153] In this way, the disclosed solution provides a detailed solution for determining how to input other word sequences (i.e., words covered due to folding) other than the first target word sequence to be inferred into the target model according to the degree of folding of the input information. The solution is simple and efficient, and thus effectively avoids the information loss caused by the folding of the model input information, and balances the computational complexity of each target network layer, reducing the pressure on the server when the model processes information, thereby laying the foundation for accelerating the model's reasoning speed and reducing the cost required for model reasoning.

[0154] Furthermore, in a specific example, the input of the j-th layer in the N target network layers can be obtained in the following manner; specifically, the above-mentioned determination of the input of the j-th layer in the N target network layers based on the numerical relationship between the number of layers of the j-th layer in the N target network layers and the folded eigenvalue can specifically include at least one of the following two methods:

[0155] Method 1: Determine the input of the non-first layer (for example, when j takes the value of 1 to N-1) in the N target network layers; specifically, determine that the number of layers of the j-th layer in the N target network layers (for example, which can be understood as the value of j) is less than the folded feature value (for example, in one example, when (layer depth-1) is less than the folded feature value s), based on the implicit output result of the j-1-th layer in the N target network layers, and at least one word in the second target word sequence to be inferred, obtain the input of the j-th layer in the N target network layers.

[0156] Here, for the relevant contents about the number of layers, layer depth and implicit output result of the jth layer in the N target network layers, reference may be made to the above examples, which will not be repeated here.

[0157] It should be noted that in one example, j ranges from 0 to N-1. Furthermore, the present disclosure uses j ranging from 0 to N-1 as an example for illustrative purposes. It is understood that j can also range from 1 to N. In this case, the subscript can be adjusted accordingly based on the actual numerical value, and the present disclosure does not impose any restrictions on this.

[0158] It can be understood that, for the first layer in the N target network layers (for example, j takes the value of 0), its input is each word in the first target word sequence to be inferred; for the non-first layer in the N target network layers, it can be obtained based on method 1. For example, taking j as 1 and the folding feature value as 2 as an example, at this time, j is less than s, or (layer depth - 1) (for example, 2-1) is less than 2, then the input of the first layer in the N target network layers can be specifically: the implicit output result of the 0th layer in the N target network layers, and at least one word in the second target word sequence to be inferred (that is, a word covered by the folding process). In other words, in one scenario, for some layers that meet the conditions, in addition to using the implicit output results of the previous layer (such as the j-1th layer in the N target network layers) as the input of the next layer (such as the jth layer in the N target network layers), it is also necessary to additionally introduce the word units obscured by the displayed word units after the folding process as another input of the next layer (such as the jth layer in the N target network layers). This lays the foundation for effectively avoiding the loss of original input information.

[0159] Furthermore, in one example, the method described in the first embodiment above, based on the implicit output result of the j-1th layer among the N target network layers and at least one word in the second target word sequence to be inferred, obtains the input of the jth layer among the N target network layers, which may specifically include:

[0160] Based on the implicit output result of the i-th position of the j-1th layer in the N target network layers and word xj+i×s , get the input of the j-th layer i-th position in the N target network layers.

[0161] Here, j represents the number of layers; i represents the input position of the word, and its value is related to the value of T2 and the value of the folding feature value. For example, in one example, i ranges from 0 to [(T2 / s) - 1]. s represents the folding feature value.

[0162] Here, for the relevant content of this part, please refer to the example shown in Figure 6(a) above, and no further details will be given here.

[0163] In this way, the disclosed solution additionally introduces at least one word in the second target word sequence to be inferred into the input of the target network layer that meets the layer depth requirements, thereby effectively avoiding the loss of original input information. In other words, while compressing the target model input, the disclosed solution utilizes the depth dimension of the target network layer to obtain the specific position of the compressed word required for input, effectively avoiding the loss of original input information, and thereby further improving the reasoning speed of the model while ensuring the reasoning effect of the model.

[0164] Furthermore, in a specific example, the implicit output result based on the j-1th layer i position in the N target network layers described above is and word x j+i×s , obtaining the input of the j-th layer i-th position of the N target network layers may specifically include:

[0165] Using the depth folding function, the implicit output result of the i-th position of the j-1th layer in the N target network layers is and word x j+i×s The fusion is performed to use the fused result as the input of the j-th layer i-th position in the N target network layers.

[0166] For example, in one example, the input of the jth layer i position in the N target network layers can be recorded as Let the depth folding function be Represents a sequence depth folding function that is used to fuse the implicit output of the previous layer (for example, represented by a vector) with the additional compressed input x j+i×s ,for example, The function can simply implement bitwise addition operation.

[0167] At this time, when the number of layers in the j-th layer of the N target network layers is less than the folded eigenvalue s (for example, when (layer depth - 1) is less than the folded eigenvalue s), the input of the i-th position of the j-th layer in the N target network layers is It can be expressed as:

[0168]

[0169] In this way, the disclosed solution provides a specific solution for determining the input of the i-th position of the j-th layer in N target network layers. In this way, the solution can quickly determine the word elements in the second target word element sequence to be inferred that need to be additionally introduced for the input of the i-th position of the j-th layer in N target network layers, thereby effectively ensuring that the compressed word element information is not lost. In this way, while ensuring the reasoning effect of the model, the reasoning efficiency of the model is further improved.

[0170] Method 2: Determine the input of the non-first layer (for example, when j takes a value of 1 to N-1) in the N target network layers; specifically, determine that the number of layers of the j-th layer in the N target network layers is greater than or equal to the folded eigenvalue (for example, in one example, when (layer depth-1) is greater than or equal to the folded eigenvalue s), and obtain the input of the i-th position of the j-th layer in the N target network layers based on the implicit output result of the j-1-th layer in the N target network layers.

[0171] That is to say, in this example, when the number of layers of the jth (j is an integer greater than 0 and less than N) layer in the N target network layers is greater than or equal to the folded feature value (for example, (layer depth - 1) is greater than or equal to the folded feature value s), the input of the i-th position of the jth layer in the N target network layers can be directly obtained based on the implicit output result of the j-1th layer in the N target network layers, without the need to additionally introduce the word elements of the second target word element sequence to be inferred. For example, in one example, when the number of layers of the jth (j is an integer greater than 0 and less than N) layer in the N target network layers is greater than or equal to the folded feature value (for example, (layer depth - 1) is greater than or equal to the folded feature value s), the implicit output result of the i-th position of the j-1th layer in the N target network layers can be directly obtained. As the input of the i-th position of the j-th layer in the N target network layers. That is, when the number of layers is greater than or equal to the folding feature value (also known as the sequence depth folding multiple) s, the input of the i-th position of the j-th layer is the output of the previous layer, and no additional folded and compressed words need to be fused.

[0172] Here, for the relevant content of this example, please refer to the example shown in Figure 6(b), which will not be repeated here.

[0173] It should be noted that this disclosed solution linearly compresses the input length required by the model. To ensure that information is not lost after the input compression, the compressed tokens are sequentially used as additional input for all layers beyond the first layer in their original input order. This input method fully utilizes the layer depth dimension, effectively reducing the input length and avoiding information loss.

[0174] In this way, the disclosed solution determines the specific scheme for inputting the i-th position of the j-th layer in N target network layers. The solution makes full use of the layer depth dimension of the target network layer to determine the specific information required for inputting different target network layers. In this way, while effectively compressing the model input, the loss of original input information caused by compression is avoided, thereby further improving the reasoning speed of the model while ensuring the reasoning effect of the model.

[0175] In summary, the disclosed solution has the following advantages:

[0176] First, it is more efficient. Compared with the improved solution of the efficient Transformer structure, the disclosed solution is simple to implement and can improve the efficiency of model training and reasoning in actual application scenarios. Moreover, the disclosed solution will not be affected by the compression of input information, so the disclosed solution supports unlimited long text input;

[0177] Second, both training and inference are highly efficient. Compared to efficient inference solutions for low-resource scenarios, this disclosed solution not only accelerates the training phase, but also the inference phase, unifying training and inference, thereby ensuring better results.

[0178] The present disclosure provides a model training device, such as Figure 11 Shown, including:

[0179] A first data processing unit 1101 is configured to fold an initial word-gram sequence used for model training based on a folding feature value used for folding the word-gram sequence, to obtain at least a first word-gram sequence after folding; wherein the initial word-gram sequence represents a word-gram sequence consisting of T1 words; and the sequence length of the first word-gram sequence is less than the sequence length of the initial word-gram sequence;

[0180] The model training unit 1102 is used to input at least the first word-gram sequence into a preset model to perform model training on the preset model to obtain a target model.

[0181] In a specific example of the disclosed solution, the first data processing unit is specifically configured to: perform folding processing on the initial word sequence to obtain the first word sequence and the second word sequence, wherein the first word sequence includes the t displayed after the folding processing. 11 word units, the second word unit sequence includes the t units covered by the displayed word units after folding. 12 word unit;

[0182] The model training unit is specifically configured to take each word unit in the first word unit sequence as input of a first layer in N network layers included in a preset model, take at least each word unit in the second word unit sequence as partial input of other layers in the N network layers except the first layer, adjust at least partial network parameters in the N network layers, and obtain a target model.

[0183] In a specific example of the present disclosure, the N network layers are connected in series; and an output of a jth layer in the N network layers is taken as input of a (j+1)th layer in the N network layers.

[0184] In a specific example of the present disclosure, the first data processing unit is further configured to determine input of the jth layer based on the folding feature value and a position of the jth layer in the N network layers; and the value of j is related to the N.

[0185] In a specific example of the present disclosure, the first data processing unit is specifically configured to include:

[0186] The input of the jth layer is determined based on a numerical relationship between the folding feature value and a layer number of the jth layer in the N network layers.

[0187] In a specific example of the present disclosure, the first data processing unit is specifically configured to:

[0188] When the layer number of the jth layer in the N network layers is less than the folding feature value, the input of the jth layer is obtained based on the implicit output result of the (j-1)th layer and at least one word unit in the second word unit sequence.

[0189] Or,

[0190] When the layer number of the jth layer in the N network layers is greater than or equal to the folding feature value, the input of the ith position of the jth layer is obtained based on the implicit output result of the (j-1)th layer.

[0191] In a specific example of the present disclosure, the first data processing unit is specifically configured to:

[0192] The input of the ith position of the jth layer is obtained based on the implicit output result of the ith position of the (j-1)th layer and the word unit x j+i×s , wherein the value of i is related to the value of T1 and the value of the folding feature value, and s represents the folding feature value.

[0193] In a specific example of the present disclosure, the first data processing unit is specifically configured to:

[0194] Using the depth folding function, the implicit output result of the i-th position of the j-1th layer is and word x j+i×s The fusion is performed to use the fused result as the input of the i-th position of the j-th layer.

[0195] In a specific example of the disclosed solution, the first data processing unit is specifically configured to:

[0196] The implicit output result of the i-th position in the j-1th layer As the input of the i-th position in the j-th layer.

[0197] In a specific example of the present disclosure, the model training unit is specifically used to:

[0198] Using each word in the first word sequence as an input to a first layer of N network layers included in a preset model, and using at least each word in the second word sequence as a partial input to other layers of the N network layers except the first layer;

[0199] Obtaining the output predicted word-gram sequence for predicting the next word-gram of each word-gram in the initial word-gram sequence;

[0200] At least part of the network parameters in the N network layers are adjusted based on at least the degree of difference between the predicted word sequence and the target sequence to obtain a target model.

[0201] In a specific example of the presently disclosed scheme, the predicted word sequence includes the predicted word output by the last layer of the N network layers included in the preset model, and the predicted word output by other layers of the N network layers except the last layer.

[0202] In a specific example of the disclosed solution, except for the last layer, the number of layers of the network layer used for outputting the predicted word unit is related to the folded feature value.

[0203] In a specific example of the disclosed solution, the number of network layers used for outputting predicted word units is greater than the folded feature value.

[0204] In a specific example of the disclosed solution, when j is greater than or equal to the difference between the total number of layers and the folded feature value s, the predicted word element output at the i-th position of the j-th layer is used to predict:

[0205] x (( s―1)―(N―1―j))+(i+1)×s;

[0206] The value of i is related to the value of T and the value of the folding characteristic value.

[0207] The disclosed solution also provides a model reasoning device, such as Figure 12 Shown, including:

[0208] The second data processing unit 1201 is configured to obtain an initial word-gram sequence to be inferred; perform folding processing on the initial word-gram sequence to be inferred based on a folding feature value for folding the word-gram sequence to obtain at least a first target word-gram sequence to be inferred; wherein the initial word-gram sequence to be inferred represents a word-gram sequence consisting of T2 words; and the sequence length of the first target word-gram sequence to be inferred is less than the sequence length of the initial word-gram sequence to be inferred;

[0209] The model inference unit 1202 is used to input at least the first target word-gram sequence to be inferred into the target model to obtain a target inference result, wherein the target inference result is the predicted next word-gram sequence of the target word-gram sequence to be inferred.

[0210] In a specific example of the disclosed solution, the second data processing unit is specifically configured to: perform folding processing on the initial word-gram sequence to be inferred to obtain the first target word-gram sequence to be inferred and the second target word-gram sequence to be inferred, wherein the first target word-gram sequence to be inferred includes the t displayed after the folding processing. 21 word units, the second target word unit sequence to be inferred includes t 22 word unit;

[0211] The model inference unit is specifically used to include: using each word in the first target word-gram sequence to be inferred as the input of the first layer of the N target network layers contained in the target model, and at least using each word in the second target word-gram sequence to be inferred as a partial input of other layers of the N target network layers except the first layer, so as to obtain a target inference result.

[0212] In a specific example of the disclosed solution, the N target network layers are connected in series; the output of the jth layer among the N target network layers serves as the input of the j+1th layer among the N target network layers.

[0213] In a specific example of the present disclosure, the second data processing unit is further used to determine the input of the jth layer in the N target network layers based on the folded feature value and the position of the jth layer in the N target network layers; wherein the value of j is related to N.

[0214] In a specific example of the present disclosure, the second data processing unit is specifically configured to:

[0215] determining the input of the jth layer in the N target network layers based on a numerical relationship between the number of layers of the jth layer in the N target network layers and the folding eigenvalue.

[0216] In a specific example of the present disclosure, the second data processing unit is specifically configured to:

[0217] determining the input of the jth layer in the N target network layers based on the implicit output result of the j-1th layer in the N target network layers and at least one token in the second target token sequence in a case where the number of layers of the jth layer in the N target network layers is less than the folding eigenvalue;

[0218] or,

[0219] determining the input of the ith position of the jth layer in the N target network layers based on the implicit output result of the j-1th layer in the N target network layers in a case where the number of layers of the jth layer in the N target network layers is greater than or equal to the folding eigenvalue.

[0220] In a specific example of the present disclosure, the second data processing unit is specifically configured to:

[0221] based on the implicit output result of the ith position of the j-1th layer in the N target network layers and the token x j+i×s , obtaining the input of the ith position of the jth layer in the N target network layers, wherein the value of i is related to the value of the T2 and the value of the folding eigenvalue, and s represents the folding eigenvalue.

[0222] In a specific example of the present disclosure, the second data processing unit is specifically configured to:

[0223] fusing the implicit output result of the ith position of the j-1th layer in the N target network layers and the token x j+i×s by using the deep folding function, so as to take the fused result as the input of the ith position of the jth layer in the N target network layers.

[0224] In a specific example of the present disclosure, the second data processing unit is specifically configured to:

[0225] taking the implicit output result of the ith position of the j-1th layer in the N target network layers as the input of the ith position of the jth layer in the N target network layers.

[0226] In a specific example of the present disclosure, the model inference unit is specifically configured to:

[0227] Obtain a target inference result output by the last position of the last n target network layers among the N target network layers; wherein the target inference result is at least the next word of the last word in the predicted initial word sequence to be inferred.

[0228] For the description of specific functions and examples of each unit of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0229] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0230] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0231] Figure 13 A schematic block diagram of an example electronic device 1300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0232] like Figure 13 As shown, device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. Various programs and data required for the operation of device 1300 can also be stored in RAM 1303. Computing unit 1301, ROM 1302, and RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to bus 1304.

[0233] Various components in device 1300 are connected to I / O interface 1305, including an input unit 1306, such as a keyboard and mouse; an output unit 1307, such as various types of displays and speakers; a storage unit 1308, such as a magnetic disk and optical disk; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1309 allows device 1300 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0234] The computing unit 1301 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as a model training method or a model reasoning method. For example, in some embodiments, the model training method or the model reasoning method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the model training method or the model reasoning method described above can be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to execute the model training method or the model inference method in any other appropriate manner (e.g., by means of firmware).

[0235] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0236] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0237] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0238] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0239] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0240] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0241] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0242] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A model training method, comprising: Based on the folding feature value used to fold the word-gram sequence, folding the initial word-gram sequence used for model training to obtain at least a first word-gram sequence after folding, including: The initial word sequence is folded to obtain the first word sequence and the second word sequence, wherein the initial word sequence represents The first word sequence includes the word sequence displayed after folding. The second word sequence includes the words that are covered by the displayed words after folding. word unit; Inputting the first word sequence into a preset model to perform model training on the preset model to obtain a target model, including: Each word in the first word sequence is used as the input of the first layer of the N network layers contained in the preset model, and each word in the second word sequence is used as the input of other layers in the N network layers except the first layer, so as to adjust at least part of the network parameters in the N network layers to obtain the target model.

2. The method according to claim 1, wherein The N network layers are connected in series; the output of the jth layer among the N network layers serves as the input of the j+1th layer among the N network layers.

3. The method according to claim 1, further comprising: Based on the folded feature value and the position of the jth layer in the N network layers, the input of the jth layer is determined; wherein the value of j is related to the N.

4. The method according to claim 3, wherein: The determining, based on the folded feature value and a position of the jth layer in the N network layers, an input of the jth layer includes: An input of the j-th layer is determined based on a numerical relationship between the number of layers of the j-th layer in the N network layers and the folded feature value.

5. The method according to claim 4, wherein The determining the input of the j-th layer based on the numerical relationship between the number of layers of the j-th layer in the N network layers and the folded feature value includes: When determining that the number of layers of the j-th layer in the N network layers is less than the collapsed feature value, obtaining an input for the j-th layer based on an implicit output result of the j-1-th layer and at least one word in the second word-word sequence; or, When it is determined that the number of layers of the j-th layer in the N network layers is greater than or equal to the folded feature value, the input of the i-th position of the j-th layer is obtained based on the implicit output result of the j-1-th layer.

6. The method according to claim 5, wherein: The step of obtaining the input of the jth layer based on the implicit output result of the j-1th layer and at least one word in the second word sequence includes: Based on the implicit output result of the i-th position in the j-1th layer and lexical elements , get the input of the jth layer i position, where the value of i is the same as the The value of and the value of the folding feature value are related, represents the collapsed eigenvalue.

7. The method according to claim 6, wherein: The implicit output result based on the i-th position of the j-1th layer and lexical elements , get the input of the j-th layer i-th position, including: Using the depth folding function, the implicit output result of the i-th position of the j-1th layer is and lexical elements The fusion is performed to use the fused result as the input of the i-th position of the j-th layer.

8. The method according to claim 5, wherein The step of obtaining the input of the i-th position of the j-th layer based on the implicit output result of the j-1-th layer includes: The implicit output result of the i-th position in the j-1th layer , as the input of the i-th position of the j-th layer.

9. The method according to any one of claims 1 to 8, wherein: The method of using each word in the first word-gram sequence as an input to a first layer of N network layers included in a preset model, and using each word in the second word-gram sequence as an input to other layers of the N network layers except the first layer, so as to adjust at least part of the network parameters of the N network layers to obtain a target model, includes: Using each word in the first word-gram sequence as the input of the first layer of N network layers included in a preset model, and using each word in the second word-gram sequence as the input of the other layers of the N network layers except the first layer; Obtaining the output predicted word-gram sequence for predicting the next word-gram of each word-gram in the initial word-gram sequence; Based on the degree of difference between the predicted word sequence and the target sequence, at least part of the network parameters in the N network layers are adjusted to obtain a target model.

10. The method according to claim 9, wherein: The predicted word-gram sequence includes the predicted word-gram output by the last layer of the N network layers included in the preset model, and the predicted word-gram output by some layers of the N network layers except the last layer.

11. The method according to claim 9, wherein Except for the last layer, the number of network layers used to output predicted tokens is related to the folded feature values.

12. The method according to claim 11, wherein The number of network layers used to output predicted tokens is greater than the folded feature value.

13. The method according to claim 12, wherein: When j is greater than or equal to the total number of layers and the folded eigenvalue In the case of the difference, the predicted word unit output at the i-th position of the j-th layer is used to predict: ； Among them, the value of i is the same as that of The value of and the value of the folding feature value are related.

14. A model inference method comprising: Obtain the initial word sequence to be inferred; Based on the folding feature value used to fold the word-gram sequence, folding processing is performed on the initial word-gram sequence to be inferred to obtain at least a first target word-gram sequence to be inferred, including: The initial word-gram sequence to be inferred is folded to obtain the first target word-gram sequence to be inferred and the second target word-gram sequence to be inferred, wherein the initial word-gram sequence to be inferred represents The first target word sequence to be inferred includes the word sequence displayed after folding. word units; the second target word unit sequence to be inferred includes the word units that are covered by the displayed word units after folding. word unit; Input the first target word sequence to be inferred into the target model to obtain the target inference result, including: Each word in the first target word-gram sequence to be inferred is used as the input of the first layer of the N target network layers contained in the target model, and each word in the second target word-gram sequence to be inferred is used as the input of other layers of the N target network layers except the first layer to obtain a target inference result; the target inference result is the predicted next word-gram sequence of the target word-gram sequence to be inferred.

15. The method according to claim 14, wherein The N target network layers are connected in series; the output of the jth layer among the N target network layers serves as the input of the j+1th layer among the N target network layers.

16. The method according to claim 14 or 15, further comprising: Based on the folded feature value and the position of the jth layer in the N target network layers, the input of the jth layer in the N target network layers is determined; wherein the value of j is related to the N.

17. The method according to claim 16, wherein: The determining, based on the folded feature value and a position of the jth layer in the N target network layers, an input of the jth layer in the N target network layers includes: Based on a numerical relationship between the number of layers of the j-th layer in the N target network layers and the folded eigenvalue, an input of the j-th layer in the N target network layers is determined.

18. The method according to claim 17, wherein The determining, based on a numerical relationship between the number of layers of the j-th layer in the N target network layers and the folded eigenvalue, an input of the j-th layer in the N target network layers, includes: When it is determined that the number of layers of the j-th layer in the N target network layers is less than the collapsed feature value, obtaining an input of the j-th layer in the N target network layers based on an implicit output result of the j-1-th layer in the N target network layers and at least one word in the second target word-word sequence to be inferred; or, When it is determined that the number of layers of the j-th layer in the N target network layers is greater than or equal to the folded eigenvalue, the input of the i-th position of the j-th layer in the N target network layers is obtained based on the implicit output result of the j-1-th layer in the N target network layers.

19. The method according to claim 18, wherein The step of obtaining the input of the jth layer among the N target network layers based on the implicit output result of the j-1th layer among the N target network layers and at least one word in the second target word-unit sequence to be inferred includes: Based on the implicit output result of the i-th position of the j-1th layer in the N target network layers and lexical elements , get the input of the jth layer i position in the N target network layers, where the value of i is the same as the The value of and the value of the folding feature value are related, represents the collapsed eigenvalue.

20. The method according to claim 19, wherein The implicit output result based on the j-1th layer i position in the N target network layers and lexical elements , obtaining the input of the j-th layer i-th position of the N target network layers, including: Using the depth folding function, the implicit output result of the i-th position of the j-1th layer in the N target network layers is and lexical elements The fusion is performed to use the fused result as the input of the j-th layer i-th position in the N target network layers.

21. The method according to claim 18, wherein The obtaining, based on the implicit output result of the j-1th layer in the N target network layers, the input of the i-th position of the j-th layer in the N target network layers includes: The implicit output result of the j-1th layer i position in the N target network layers , as the input of the j-th layer i-th position in the N target network layers.

22. The method according to claim 14 or 15, wherein: Obtaining the target reasoning result includes: Obtain a target inference result output by the last position of the last n target network layers among the N target network layers; wherein the target inference result is at least the next word of the last word in the predicted initial word sequence to be inferred.

23. A model training device comprising: The first data processing unit is configured to fold the initial word-gram sequence used for model training based on the folding feature value used for folding the word-gram sequence, so as to obtain at least a first word-gram sequence after folding; wherein the initial word-gram sequence represents A word sequence composed of words; A model training unit, configured to input the first word-gram sequence into a preset model to perform model training on the preset model to obtain a target model; The first data processing unit is specifically configured to: perform folding processing on the initial word sequence to obtain the first word sequence and the second word sequence, wherein the first word sequence includes the words displayed after folding processing. word units, the second word unit sequence includes the word units that are covered by the displayed word units after folding. word unit; The model training unit is specifically used to use each word in the first word sequence as the input of the first layer of the N network layers contained in the preset model, and use each word in the second word sequence as the input of other layers in the N network layers except the first layer, so as to adjust at least part of the network parameters in the N network layers to obtain the target model.

24. The device according to claim 23, wherein The N network layers are connected in series; the output of the jth layer among the N network layers serves as the input of the j+1th layer among the N network layers.

25. The apparatus according to claim 23, wherein The first data processing unit is further used to determine the input of the jth layer based on the folded feature value and the position of the jth layer in the N network layers; wherein the value of j is related to N.

26. The device according to claim 25, wherein The first data processing unit is specifically configured to include: An input of the j-th layer is determined based on a numerical relationship between the number of layers of the j-th layer in the N network layers and the folded feature value.

27. The device according to claim 26, wherein The first data processing unit is specifically configured to: When determining that the number of layers of the j-th layer in the N network layers is less than the collapsed feature value, obtaining an input for the j-th layer based on an implicit output result of the j-1-th layer and at least one word in the second word-word sequence; or, When it is determined that the number of layers of the j-th layer in the N network layers is greater than or equal to the folded feature value, the input of the i-th position of the j-th layer is obtained based on the implicit output result of the j-1-th layer.

28. The apparatus according to claim 27, wherein The first data processing unit is specifically configured to: Based on the implicit output result of the i-th position in the j-1th layer and lexical elements , get the input of the i-th position of the j-th layer, where the value of i is the same as the The value of and the value of the folding feature value are related. represents the collapsed eigenvalue.

29. The apparatus according to claim 28, wherein The first data processing unit is specifically configured to: Using the depth folding function, the implicit output result of the i-th position of the j-1th layer is and lexical elements The fusion is performed to use the fused result as the input of the i-th position of the j-th layer.

30. The apparatus of claim 27, wherein: The first data processing unit is specifically configured to: The implicit output result of the i-th position in the j-1th layer , as the input of the i-th position of the j-th layer.

31. The device according to any one of claims 23 to 30, wherein: The model training unit is specifically used to: Using each word in the first word-gram sequence as the input of the first layer of N network layers included in a preset model, and using each word in the second word-gram sequence as the input of the other layers of the N network layers except the first layer; Obtaining the output predicted word-gram sequence for predicting the next word-gram of each word-gram in the initial word-gram sequence; Based on the degree of difference between the predicted word sequence and the target sequence, at least part of the network parameters in the N network layers are adjusted to obtain a target model.

32. The apparatus according to claim 31, wherein The predicted word-gram sequence includes the predicted word-gram output by the last layer of the N network layers included in the preset model, and the predicted word-gram output by some layers of the N network layers except the last layer.

33. The apparatus according to claim 31, wherein Except for the last layer, the number of network layers used to output predicted tokens is related to the folded feature values.

34. The apparatus according to claim 33, wherein The number of network layers used to output predicted tokens is greater than the folded feature value.

35. The apparatus of claim 34, wherein: When j is greater than or equal to the total number of layers and the folded eigenvalue In the case of the difference, the predicted word unit output at the i-th position of the j-th layer is used to predict: ； Among them, the value of i is the same as that of The value of and the value of the folding feature value are related.

36. A model reasoning device comprising: A second data processing unit is used to obtain an initial word sequence to be inferred; Based on the folding feature value used to fold the word-gram sequence, the initial word-gram sequence to be inferred is folded to obtain at least a first target word-gram sequence to be inferred; wherein the initial word-gram sequence to be inferred represents A word sequence composed of words; A model inference unit, configured to input a first target word-gram sequence to be inferred into a target model to obtain a target inference result, wherein the target inference result is a predicted next word-gram sequence of the target word-gram sequence to be inferred; The second data processing unit is specifically configured to: perform folding processing on the initial word-gram sequence to be inferred to obtain the first target word-gram sequence to be inferred and the second target word-gram sequence to be inferred, wherein the first target word-gram sequence to be inferred includes the word-gram sequence displayed after folding processing. word units, the second target word unit sequence to be inferred includes the word units that are covered by the displayed word units after folding. word unit; The model inference unit is specifically used to include: using each word in the first target word-gram sequence to be inferred as the input of the first layer of the N target network layers contained in the target model, and using each word in the second target word-gram sequence to be inferred as the input of other layers of the N target network layers except the first layer, so as to obtain a target inference result.

37. The apparatus according to claim 36, wherein The N target network layers are connected in series; the output of the jth layer among the N target network layers serves as the input of the j+1th layer among the N target network layers.

38. The apparatus according to claim 36 or 37, wherein The second data processing unit is further used to determine the input of the jth layer in the N target network layers based on the folded feature value and the position of the jth layer in the N target network layers; wherein the value of j is related to N.

39. The apparatus according to claim 38, wherein The second data processing unit is specifically configured to: Based on a numerical relationship between the number of layers of the j-th layer in the N target network layers and the folded eigenvalue, an input of the j-th layer in the N target network layers is determined.

40. The apparatus of claim 39, wherein The second data processing unit is specifically configured to: When it is determined that the number of layers of the j-th layer in the N target network layers is less than the collapsed feature value, obtaining an input of the j-th layer in the N target network layers based on an implicit output result of the j-1-th layer in the N target network layers and at least one word in the second target word-word sequence to be inferred; or, When it is determined that the number of layers of the j-th layer in the N target network layers is greater than or equal to the folded eigenvalue, the input of the i-th position of the j-th layer in the N target network layers is obtained based on the implicit output result of the j-1-th layer in the N target network layers.

41. The apparatus according to claim 40, wherein The second data processing unit is specifically configured to: Based on the implicit output result of the i-th position of the j-1th layer in the N target network layers and lexical elements , get the input of the jth layer i position in the N target network layers, where the value of i is the same as the The value of and the value of the folding feature value are related, represents the collapsed eigenvalue.

42. The apparatus according to claim 41, wherein The second data processing unit is specifically configured to: Using the depth folding function, the implicit output result of the i-th position of the j-1th layer in the N target network layers is and lexical elements The fusion is performed to use the fused result as the input of the j-th layer i-th position in the N target network layers.

43. The apparatus according to claim 40, wherein The second data processing unit is specifically configured to: The implicit output result of the j-1th layer i position in the N target network layers , as the input of the j-th layer i-th position in the N target network layers.

44. The apparatus according to claim 36 or 37, wherein The model inference unit is specifically used to: Obtain a target inference result output by the last position of the last n target network layers among the N target network layers; wherein the target inference result is at least the next word of the last word in the predicted initial word sequence to be inferred.

45. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 22.

46. ​​A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-22.

47. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 22.

Images