Text object coding method and device, computer equipment and storage medium

Abstract

The embodiment of the invention discloses a text object coding method and device, computer equipment and a storage medium. Relates to the technical field of text processing, and mainly aims to solve the problem of poor text processing model training effect caused by low text object coding efficiency. Comprising the steps of obtaining a text identifier sequence; determining each identifier pair and a target text object in the text identifier sequence; and determining an adjacent logarithm and a non-adjacent logarithm corresponding to the target text object based on the identifier pair, and encoding the target text object based on the adjacent logarithm and the non-adjacent logarithm to obtain encoding information.

Description

Technical Field

[0001] The present invention relates to the field of text processing technology, in particular to a text object encoding method, device, computer equipment and storage medium Background Art

[0002] Natural language processing, as an interdisciplinary subject in the fields of artificial intelligence and linguistics, usually refers to the use of machine learning and other technologies to enable computers to process and understand human language. In natural language processing, classic processing tasks mainly include text classification, text matching, sequence labeling, etc. Among them, for text generation tasks, such as summary extraction, machine translation and other scenarios, the processing ability of the language model is the key technology for text generation, which usually requires encoding of the text.

[0003] At present, the existing Transformer decoding-encoding architecture is usually used to generate text identification tokens one by one, that is, the token is encoded using a one-hot encoding method as the vocabulary size. At this time, each token in the vocabulary is encoded only based on the information of the previous token. However, due to the large number of tokens in the text vocabulary, encoding one by one based on the previous token will result in a limited learning range of the token as the learning target in the subsequent model training stage, thereby affecting the accuracy of processing text tasks. Summary of the invention

[0004] The purpose of the present invention is to propose a text object encoding method, device, computer equipment and storage medium. The main purpose is to solve the problem of poor training effect of text processing model caused by low text object encoding efficiency.

[0005] According to one aspect of the present invention, there is provided a method for encoding a text object, comprising:

[0006] Get a sequence of text identifiers;

[0007] Determine each identifier pair in the text identifier sequence and a target text object;

[0008] The number of adjacent pairs and the number of non-adjacent pairs corresponding to the target text object are determined based on the identifier pair, and the target text object is encoded based on the number of adjacent pairs and the number of non-adjacent pairs to obtain encoding information.

[0009] Furthermore, encoding the target text object based on the adjacent pairs and the non-adjacent pairs further includes:

[0010] Perform a first operation based on the adjacent pairs and the non-adjacent pairs to obtain a first number of 2-tuples of the target text object;

[0011] Perform a second operation based on the number of adjacent pairs and the first number of two-tuples to obtain a first probability value of the adjacent identifier;

[0012] Perform a third operation based on the number of non-adjacent pairs and the first number of two-tuples to obtain a second probability value of the non-adjacent identifier;

[0013] The encoding information of the target text object is obtained by combining the first probability value and the second probability value.

[0014] Furthermore, encoding the target text object based on the adjacent pairs and the non-adjacent pairs further includes:

[0015] If the number of adjacent pairs is less than the number of non-adjacent pairs, performing a fourth operation based on the number of adjacent pairs, the number of non-adjacent pairs and the adjustment coefficient to obtain a second number of two-tuples;

[0016] Perform a fifth operation based on the number of adjacent pairs, the number of non-adjacent pairs, and the second number of two-tuples to obtain a third probability value of the adjacent identifier;

[0017] Performing a sixth operation based on the number of non-adjacent pairs and the second number of two-tuples to obtain a fourth probability value of the non-adjacent identifier;

[0018] Based on the combination of the third probability value and the fourth probability value, the adjusted coding information is obtained.

[0019] Furthermore, the method further comprises:

[0020] If the number of non-adjacent pairs is less than a preset threshold, configuring the second probability value as a preset probability value;

[0021] The obtaining the encoding information of the target text object based on the combination of the first probability value and the second probability value includes:

[0022] Based on the combination of the first probability value and the preset probability value, the encoding information of the target text object is obtained.

[0023] Further, the determining each identifier pair in the text identifier sequence and the target text object comprises:

[0024] Based on all identifiers in the text identifier sequence, construct identifier pairs in the form of two tuples one by one, and count the logarithms of the identifier pairs; determine the left identifier in the identifier pair corresponding to the largest value in the logarithms as the target text object; or,

[0025] The left identifier in each of the identifier pairs is determined as the target text object.

[0026] Further, the determining the number of adjacent pairs and the number of non-adjacent pairs corresponding to the target text object based on the identifier pair includes:

[0027] Counting the number of adjacent pairs of adjacent identifiers formed by the target text object and the adjacent identifiers;

[0028] The number of non-adjacent pairs of non-adjacent identifier pairs formed by the target text object and the non-adjacent identifiers is counted.

[0029] Furthermore, after encoding the target text object based on the adjacent pairs and the non-adjacent pairs to obtain encoding information, the method further includes:

[0030] The decoding model is trained based on the target text object and the encoding information to obtain a trained decoding model, wherein the target text object is an input sample, the encoding information is an output sample, and the loss function of the decoding model is constructed based on adjacent identifier probability values, non-adjacent identifier probability values, normalized adjacent identifier probability values, and normalized non-adjacent identifier probability values.

[0031] According to another aspect of the present invention, there is provided a text object encoding device, comprising:

[0032] An acquisition module, used for acquiring a text identifier sequence;

[0033] A determination module, used for determining each identifier pair in the text identifier sequence and a target text object;

[0034] The encoding module is used to determine the number of adjacent pairs and the number of non-adjacent pairs corresponding to the target text object based on the identifier pair, and encode the target text object based on the number of adjacent pairs and the number of non-adjacent pairs to obtain encoding information.

[0035] Furthermore, the encoding module is specifically used to perform a first operation based on the adjacent pairs and the non-adjacent pairs to obtain a first number of bigrams of the target text object; perform a second operation based on the adjacent pairs and the first number of bigrams to obtain a first probability value of the adjacent identifier; perform a third operation based on the non-adjacent pairs and the first number of bigrams to obtain a second probability value of the non-adjacent identifier; and obtain encoding information of the target text object by combining the first probability value and the second probability value.

[0036] Furthermore, the encoding module is specifically used to perform a fourth operation based on the adjacent pairs, the non-adjacent pairs and the adjustment coefficient to obtain the second number of two-tuples if the adjacent pair number is less than the non-adjacent pair number; perform a fifth operation based on the adjacent pairs, the non-adjacent pairs and the second number of two-tuples to obtain the third probability value of the adjacent identifier; perform a sixth operation based on the non-adjacent pairs and the second number of two-tuples to obtain the fourth probability value of the non-adjacent identifier; and perform a combination based on the third probability value and the fourth probability value to obtain the adjusted encoding information.

[0037] Furthermore, the device further comprises: a configuration module,

[0038] The configuration module is configured to configure the second probability value to a preset probability value if the number of non-adjacent pairs is less than a preset threshold;

[0039] The encoding module is specifically used to obtain encoding information of the target text object based on the combination of the first probability value and the preset probability value.

[0040] Furthermore, the determination module is specifically used to construct identifier pairs in the form of tuples based on all identifiers in the text identifier sequence one by one, and count the logarithms of the identifier pairs; determine the left identifier in the identifier pair corresponding to the largest numerical value in the logarithm as the target text object; or, determine the left identifier in each of the identifier pairs as the target text object.

[0041] Furthermore, the encoding module is specifically used to count the number of adjacent pairs of adjacent identifiers formed by the target text object and adjacent identifiers; and count the number of non-adjacent pairs of non-adjacent identifiers formed by the target text object and non-adjacent identifiers.

[0042] Furthermore, the device also includes:

[0043] A training module is used to train a decoding model based on the target text object and the encoding information to obtain a decoding model that has completed training, wherein the target text object is an input sample, the encoding information is an output sample, and the loss function of the decoding model is constructed based on adjacent identifier probability values, non-adjacent identifier probability values, and normalized adjacent identifier probability values ​​and normalized non-adjacent identifier probability values.

[0044] According to another aspect of the present invention, a storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute operations corresponding to the encoding method of the text object as described above.

[0045] According to another aspect of the present invention, there is provided a terminal, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0046] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the encoding method of the text object.

[0047] By means of the above technical solution, the technical solution provided by the embodiment of the present invention has at least the following advantages:

[0048] The present invention provides a text object encoding method, device, computer equipment and storage medium. Compared with the prior art, the embodiments of the present invention determine each identifier pair and a target text object in a text identifier sequence, count the number of adjacent pairs and non-adjacent pairs that appear in the text identifier sequence relative to the target text object, and then encode the target text object based on the adjacent pairs and non-adjacent pairs to obtain encoding information, thereby realizing the fusion of information in multiple tokens in adjacent or non-adjacent forms, avoiding the single learning target problem of one-hot encoding, and thus improving the processing accuracy of text tasks.

[0049] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] in:

[0052] Figure 1 A flow chart of a text object encoding method provided by an embodiment of the present invention is shown;

[0053] Figure 2 A flow chart of another text object encoding method provided by an embodiment of the present invention is shown;

[0054] Figure 3 A block diagram showing a text object encoding device provided by an embodiment of the present invention;

[0055] Figure 4A schematic diagram of the structure of a terminal provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0058] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0059] Since there are a large number of tokens in the text vocabulary, encoding them one by one depending on the previous token will result in a limited learning range of the token as the learning target in the subsequent model training stage, thereby affecting the effectiveness of processing text tasks. The embodiment of the present invention provides a method for encoding text objects, such as Figure 1 As shown, the method includes:

[0060] 101. Get a text identifier sequence.

[0061] In an embodiment of the present invention, when performing a text generation task, the current execution end converts the text content into different text identifiers in advance, and constructs a text identifier sequence for a sentence of text content or a paragraph of text content. Among them, the text language can be Chinese, English or other minority languages. For the text content of different languages, it is subjected to word segmentation conversion to obtain a text identifier for representing the text content, which is constructed as a text identifier sequence, i.e., a token sequence, which is not specifically limited in the embodiment of the present invention. In addition, when performing word segmentation conversion, an n-gram model, an HMM hidden Markov model, a naive Bayes classification, etc. can be used, which is not specifically limited in the embodiment of the present invention.

[0062] It should be noted that the current execution end can be an execution end for performing tasks such as text generation, text processing, text training, etc., including but not limited to terminal devices and server devices, such as computer equipment, tablet computers, mobile phones, servers, etc., wherein the server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0063] 102. Determine each identifier pair in the text identifier sequence and a target text object.

[0064] In the embodiment of the present invention, since the text identifier sequence includes all the identifiers that constitute the text content, an identifier pair is constructed. At this time, the identifier pair is constructed in the form of a tuple, that is, any text identifier in the text identifier sequence is combined with other identifiers to form an identifier pair. For example, the text identifier sequence is X=[[a,b,c,c,d,b],[b,c,d,a,c,a,d,d],[c,b,d,a,d,a]], and the identifier pair includes (a,a), (a,b), (a,c), (a,d), (c,a), (c,b), (c,c), (c,d), (b,a), (b,b), (b,c), (b,d), (d,a), (d,b), (d,c), (d,d), which is not specifically limited in the embodiment of the present invention.

[0065] In addition, in an embodiment of the present invention, in order to realize text encoding in combination with a machine learning model to improve the encoding effect of all text identifiers, the target text object is the encoding object of the specific encoding method selected in step 103 to be executed, which can be any identifier object in the text identifier sequence, and the embodiment of the present invention does not make specific limitations.

[0066] 103. Determine the number of adjacent pairs and the number of non-adjacent pairs corresponding to the target text object based on the identifier pair, and encode the target text object based on the number of adjacent pairs and the number of non-adjacent pairs to obtain encoding information.

[0067] In the embodiment of the present invention, since the identifier pair is constructed based on two text identifiers and is represented in the form of a binary, the positions of the two different text identifiers in the binary are different, and the counted numbers of pairs are also different, for example, (a, b) and (b, a), and the positions of different text identifiers in the text identifier sequence are also different, therefore, the current execution end counts the number of adjacent pairs and the number of non-adjacent pairs belonging to the target text object. Among them, the adjacent pair is an identifier pair composed of the target text object and other text identifiers in the text identifier sequence having an adjacent relationship, and the non-adjacent pair is an identifier pair composed of the target text object and other text identifiers in the text identifier sequence having a non-adjacent relationship, thereby the number of adjacent pairs and the number of non-adjacent pairs of the target text object can be counted, and the embodiment of the present invention does not make specific limitations.

[0068] It should be noted that in order to effectively learn the influence of other text identifiers on the target text object in the text content and improve the encoding effect of the text object, the current execution end encodes the target text object based on adjacent pairs and non-adjacent pairs to obtain encoding information. Among them, the probability value of the target text object and the probability value of other text identifiers can be calculated by operating the adjacent pairs and non-adjacent pairs to obtain the encoding information.

[0069] An embodiment of the present invention provides a method for encoding text objects, by determining each identifier pair and a target text object in a text identifier sequence, and counting the number of adjacent pairs and non-adjacent pairs that appear in the text identifier sequence relative to the target text object, thereby encoding the target text object based on the number of adjacent pairs and the number of non-adjacent pairs to obtain encoding information, thereby realizing the fusion of information in multiple tokens in adjacent or non-adjacent forms, avoiding the single learning target problem of one-hot encoding, and thus improving the processing accuracy of text tasks.

[0070] The embodiment of the present invention provides another encoding method of text objects, such as Figure 2 As shown, the method includes:

[0071] 201. Get a text identifier sequence.

[0072] In the embodiment of the present invention, step 201 is the same as step 101 and will not be described in detail herein.

[0073] 202a. Construct identifier pairs in the form of tuples one by one based on all identifiers in the text identifier sequence, and count the logarithms of the identifier pairs; determine the left identifier in the identifier pair corresponding to the largest value in the logarithm as the target text object.

[0074] In order to effectively encode a text identifier in a text identifier sequence so that the encoded content can cover the relevance of other text identifiers in text expression, when the current execution end determines the target text object, it first constructs an identifier pair containing all text identifiers. Among them, the bigram model Bigram can be used to construct a combination of two consecutive identifiers to obtain a bigram identifier pair. For example, the text identifier sequence is X = [[a,b,c,c,d,b], [b,c,d,a,c,a,d,d], [c,b,d,a,d,a]], and the identifier pair includes (a,a), (a,b), (a,c), (a,d), (c,a), (c,b), (c,c), (c,d), (b,a), (b,b), (b,c), (b,d), (d,a), (d,b), (d,c), (d,d), which is not specifically limited in the embodiment of the present invention. Further, the number of pairs of each identifier pair is counted.

[0075] In order to achieve high coverage of text object encoding, when determining the target text object, the current execution end can determine the left identifier in the identifier pair corresponding to the maximum value in the logarithm obtained by statistics as the target text object. Among them, since the identifier pair is constructed in the form of a tuple, the identifier pair in the form of a tuple includes a left identifier and a right identifier, and after the logarithm of all identifier pairs is obtained by statistics, the identifier pair corresponding to the maximum logarithm can be determined, and at this time, the left identifier is determined as the target text object.

[0076] In the embodiment of the present invention, step 202b, which is parallel to step 202a, determines the left identifier in each of the identifier pairs as the target text object.

[0077] In order to achieve high coverage of text object encoding, when determining the target text object, the current execution end may also randomly select a left identifier of an identifier pair from all identifier pairs as the target text object by random selection.

[0078] 203. Count the number of adjacent pairs of adjacent identifier pairs formed by the target text object and the adjacent identifier.

[0079] 204. Count the number of non-adjacent pairs of non-adjacent identifier pairs formed by the target text object and the non-adjacent identifiers.

[0080] In order to calculate the probability based on the number of pairs of identifiers containing the target text object, so as to serve as the encoding basis of the target text object, the current execution end counts the number of pairs of adjacent identifiers consisting of the target text object and the adjacent identifiers, as well as the number of pairs of non-adjacent identifiers. For example, the text identifier sequence is X=[[a,b,c,c,d,b],[b,c,d,a,c,a,d,d],[c,b,d,a,d,a]], the identifier pairs include (a,a), (a,b), (a,c), (a,d), (c,a), (c,b), (c,c), (c,d), (b,a), (b,b), (b,c), (b,d), (d,a), (d,b), (d,c), (d,d), the target text object is d, then the adjacent identifier pairs include (d,a), (d,b), (d,c), (d,d), the number of adjacent pairs is 3+1+0+1=5, which is not specifically limited in the embodiment of the present invention.

[0081] 205. Perform a first operation based on the adjacent pairs and the non-adjacent pairs to obtain a first number of 2-tuples of the target text object.

[0082] In order to use the occurrence probability of the identifier as the coding basis to achieve the expansion of the identifier coding range, the current execution end first performs a first operation on the adjacent pairs and the non-adjacent pairs. At this time, the first operation is a summation operation to obtain the first number of 2-tuples. The summation operation as the first operation is expressed as C(t left ,t right ) is the number of identifier pairs containing the target text object and the adjacent identifier pairs, C(t left ,other right ) is the number of non-adjacent pairs of target text objects and non-adjacent identifiers, i is a variable, i=1, 2...v, v is the number of identifier types in the identifier table, for example, the word list includes a, b, c, d, a total of 4 words, then v=4. At the same time, t left is the target text object, t right Adjacent identifier, other right are non-adjacent identifiers.

[0083] 206. Perform a second operation based on the number of adjacent pairs and the first number of two-tuples to obtain a first probability value of the adjacent identifier.

[0084] In order to use the occurrence probability of the identifier as the basis for encoding, thereby expanding the encoding range of the identifier, the current execution end performs a second operation on the number of adjacent pairs and the first and second groups of numbers. At this time, the second operation is a probability operation to obtain a first probability value. The probability operation as the second operation is expressed as C is the number of first two-tuples, C(tleft ,t right ) is the number of adjacent pairs including the target text object and the adjacent identifier pairs, which is not specifically limited in the embodiment of the present invention.

[0085] 207. Perform a third operation based on the number of non-adjacent pairs and the first number of two-tuples to obtain a second probability value of the non-adjacent identifier.

[0086] In order to use the occurrence probability of the identifier as the basis for encoding, thereby expanding the encoding range of the identifier, the current execution end performs a third operation on the number of non-adjacent pairs and the number of first two-tuples. At this time, the third operation is a probability operation to obtain a second probability value. The probability operation as the third operation is expressed as In addition, since there may be multiple non-adjacent identifiers, multiple corresponding second probability values ​​may be calculated accordingly, which is not specifically limited in the embodiment of the present invention.

[0087] 208. Obtain encoding information of the target text object based on a combination of the first probability value and the second probability value.

[0088] In order to use the probability of occurrence of the identifier as the basis for encoding, thereby expanding the encoding range of the identifier, the current execution end combines the first probability value and the second probability value as the encoding information when constructing the encoding information of the target text object. For example, for the target text object a, the calculated first probability value is 0.3, and the second probability values ​​are 0.2, 0.4, and 0.1, then the encoding information for a is (0.3, 0.2, 0.4, 0.1).

[0089] In another embodiment of the present invention, for further definition and explanation, encoding the target text object based on the adjacent pairs and the non-adjacent pairs in step 103 further includes:

[0090] If the number of adjacent pairs is less than the number of non-adjacent pairs, performing a fourth operation based on the number of adjacent pairs, the number of non-adjacent pairs and the adjustment coefficient to obtain a second number of two-tuples;

[0091] Perform a fifth operation based on the number of adjacent pairs, the number of non-adjacent pairs, and the second number of two-tuples to obtain a third probability value of the adjacent identifier;

[0092] Performing a sixth operation based on the number of non-adjacent pairs and the second number of two-tuples to obtain a fourth probability value of the non-adjacent identifier;

[0093] Based on the combination of the third probability value and the fourth probability value, the adjusted coding information is obtained.

[0094] In the implementation scenario of randomly selecting the left identifier as the target text object, in order to ensure that the probability value of the adjacent identifier is the largest in the probability distribution, and to retain the probability value of the non-adjacent identifier, so as to improve the statistical validity of the number of bigrams and thus meet the learning effect of the encoding, the current execution end first compares the size of the adjacent pair number and the non-adjacent pair number. If the adjacent pair number is greater than or equal to the non-adjacent pair number, it means that the probability value of the adjacent identifier is the largest in the probability distribution. If the adjacent pair number is less than the non-adjacent pair number, it means that the probability value of the adjacent identifier is not the largest in the probability distribution. Therefore, the current execution end calculates the number of bigrams. Specifically, a fourth operation is performed based on the adjacent pair number, the non-adjacent pair number and the adjustment coefficient to obtain the second number of bigrams. Among them, the fourth operation is a sum operation, and the sum operation as the fourth operation is expressed as C(t left ,t right ) is the number of identifier pairs containing the target text object and the adjacent identifier pairs, C(t left ,other right ) is the number of pairs of identifiers including the target text object and non-adjacent pairs of identifiers, ε is an adjustment coefficient, preferably 2. Further, the current execution end calculates the third probability value of adjacent identifiers by combining the number of adjacent pairs and the number of non-adjacent pairs, wherein the fifth operation is a probability operation, and the fifth operation as a probability operation is expressed as C is the number of second tuples, C(t left ,t right ) is the number of adjacent pairs including the target text object and the adjacent identifier pair, which is not specifically limited in the embodiment of the present invention. Meanwhile, when calculating the fourth probability value of the non-adjacent identifier, the probability operation of the sixth operation and the third operation is expressed as: The same, only the first two-tuple number is adjusted to the second two-tuple number, and the embodiment of the present invention does not make specific limitations. Finally, based on the third probability value and the fourth probability value, the adjusted coding information is obtained, which is the same as the above combination method and is not repeated here.

[0095] In another embodiment of the present invention, for further definition and description, the steps further include:

[0096] If the number of non-adjacent pairs is less than a preset threshold, configuring the second probability value as a preset probability value;

[0097] The obtaining the encoding information of the target text object based on the combination of the first probability value and the second probability value includes:

[0098] Based on the combination of the first probability value and the preset probability value, the encoding information of the target text object is obtained.

[0099] In order to avoid the long tail phenomenon caused by multiple zeros in the construction of binary identifier pairs, thereby improving the learning effectiveness in model training, the current execution end pre-sets a threshold value to determine whether to reconfigure the second probability value of non-adjacent identifiers. Among them, the preset threshold can be set based on the number distribution of binary groups, preferably a value between 7-10. At this time, if the number of non-adjacent pairs obtained by counting non-adjacent identifiers is less than the preset threshold, the second probability value calculated for this non-adjacent identifier is configured as the preset probability value. At this time, the preset probability value is 0. Furthermore, when the first probability value is combined with the second probability value to obtain the coding information, specifically, the first probability value is directly combined with the preset probability value to obtain the coding information. For example, if the number of non-adjacent pairs of the v-3, v-2, and v-1 non-adjacent identifiers is less than the preset threshold, the probability values ​​of the v-3, v-2, and v-1 non-adjacent identifiers are configured to 0, and the target coding information obtained is p=[p(t right ),p(other 1 right ),p(other 2 right )...0 v-3 ,0 v-2 ,0 v-1 ], and the embodiments of the present invention are not specifically limited.

[0100] In another embodiment of the present invention, for further definition and explanation, the step encodes the target text object based on the adjacent pairs and the non-adjacent pairs, and after obtaining the encoding information, the method further includes:

[0101] The decoding model is trained based on the target text object and the encoding information to obtain a trained decoding model.

[0102] In order to use the obtained encoding information to learn in the model training phase, so as to meet the need of learning other text object information while learning the target text object, so as to enhance the ability to understand the text, after the current execution end obtains the encoding information, it trains the decoding model based on the target text object and the encoding information to obtain the decoding model after learning the encoding information. The target text object is the input sample, and the encoding information is the output sample. For example, when learning adjacent pairs (t left ,t right ), t left is the input, and p obtained by alignment and encoding is the output that the model needs to learn, which is not specifically limited in the embodiment of the present invention.

[0103] It should be noted that, in the embodiment of the present invention, the loss function of the decoding model is constructed based on the adjacent identifier probability value, the non-adjacent identifier probability value, and the normalized adjacent identifier probability value and the normalized non-adjacent identifier probability value, that is, the loss function is expressed as Among them, p(t right ) is the probability value of adjacent identifiers, p(t right )' is the normalized probability value of adjacent identifiers, is the probability value of non-adjacent identifiers, is the normalized probability value of non-adjacent identifiers, for example, p(t right )=[p(t right ),p(other 1 right ),p(other 2 right )...0 v-3 ,0 v-2 ,0 v-1 ] is the coded information including the probability values ​​of adjacent identifiers and the probability values ​​of non-adjacent identifiers. The normalized probability distribution is p=[p(t right )',p(other 1 right )',p(other 2 right )'...0' v-3 ,0' v-2 ,0' v-1 ], so that the normalized encoded information contains more semantic content under the premise of unchanged sample size. In a specific implementation scenario, the decoding model is preferably a Transformer decoder as the training model, and the training set X = {x i}, i∈[1,n], at this time, x i is a text identifier obtained by text splitting, which can be a string, etc., and n is the total number of samples in the training set, which is not specifically limited in the embodiment of the present invention. At the same time, when training the entire training set X, the current execution end can average the loss functions of all adjacent identification pairs in a sample, and then use back propagation to update the model parameters of the Transformers decoder to achieve the same degree of learning for all samples during the training process. A custom coefficient can also be configured on each loss function to avoid imbalance in the degree of learning of the samples.

[0104] In a specific implementation scenario of the embodiment of the present invention, the data set X = [[a,b,c,c,d,b],[b,c,d,a,c,a,d,d],[c,b,d,a,d,a]], after constructing the text identifier pair, the number of occurrences of each adjacent identifier and non-adjacent identifier is counted as shown in Table 1 below:

[0105] Table 1

[0106] (a,a) 0 (b,a) 0 (a,b) 1 (b,b) 0 (a,c) 1 (b,c) 1 (a,d) 2 (b,d) 1 (c,a) 0 (d,a) 3 (c,b) 1 (d,b) 1 (c,c) 1 (d,c) 0 (c,d) 2 (d,d) 1

[0107] After selecting token d as the target text object, count the number of bigrams C(d, a). Since there are only 4 tokens [a, b, c, d] in the data set, v = 4, and calculate

[0108] Generate the probability values ​​for tokens a, b, c, and d, so that when the decoding model takes d as input, the target encoding that the decoding model needs to learn is

[0109] An embodiment of the present invention provides another encoding method for text objects, by determining each identifier pair and a target text object in a text identifier sequence, and counting the number of adjacent pairs and non-adjacent pairs that appear in the text identifier sequence relative to the target text object, thereby encoding the target text object based on the number of adjacent pairs and the number of non-adjacent pairs to obtain encoding information, thereby realizing the fusion of information in multiple tokens in adjacent or non-adjacent forms, avoiding the single learning target problem of one-hot encoding, and thus improving the processing accuracy of text tasks.

[0110] Furthermore, as a response to the above Figure 1 The embodiment of the present invention provides a text object encoding device, such as Figure 3 As shown, the device comprises:

[0111] An acquisition module 31, used for acquiring a text identifier sequence;

[0112] A determination module 32, used to determine each identifier pair in the text identifier sequence and a target text object;

[0113] The encoding module 33 is used to determine the number of adjacent pairs and the number of non-adjacent pairs corresponding to the target text object based on the identifier pair, and encode the target text object based on the number of adjacent pairs and the number of non-adjacent pairs to obtain encoding information.

[0114] Furthermore, the encoding module is specifically used to perform a first operation based on the adjacent pairs and the non-adjacent pairs to obtain a first number of bigrams of the target text object; perform a second operation based on the adjacent pairs and the first number of bigrams to obtain a first probability value of the adjacent identifier; perform a third operation based on the non-adjacent pairs and the first number of bigrams to obtain a second probability value of the non-adjacent identifier; and obtain encoding information of the target text object by combining the first probability value and the second probability value.

[0115] Furthermore, the encoding module is specifically used to perform a fourth operation based on the adjacent pairs, the non-adjacent pairs and the adjustment coefficient to obtain the second number of two-tuples if the adjacent pair number is less than the non-adjacent pair number; perform a fifth operation based on the adjacent pairs, the non-adjacent pairs and the second number of two-tuples to obtain the third probability value of the adjacent identifier; perform a sixth operation based on the non-adjacent pairs and the second number of two-tuples to obtain the fourth probability value of the non-adjacent identifier; and perform a combination based on the third probability value and the fourth probability value to obtain the adjusted encoding information.

[0116] Furthermore, the device further comprises: a configuration module,

[0117] The configuration module is configured to configure the second probability value to a preset probability value if the number of non-adjacent pairs is less than a preset threshold;

[0118] The encoding module is specifically used to obtain encoding information of the target text object based on the combination of the first probability value and the preset probability value.

[0119] Furthermore, the determination module is specifically used to construct identifier pairs in the form of tuples based on all identifiers in the text identifier sequence one by one, and count the logarithms of the identifier pairs; determine the left identifier in the identifier pair corresponding to the largest numerical value in the logarithm as the target text object; or, determine the left identifier in each of the identifier pairs as the target text object.

[0120] Furthermore, the encoding module is specifically used to count the number of adjacent pairs of adjacent identifiers formed by the target text object and adjacent identifiers; and count the number of non-adjacent pairs of non-adjacent identifiers formed by the target text object and non-adjacent identifiers.

[0121] Furthermore, the device also includes:

[0122] A training module is used to train a decoding model based on the target text object and the encoding information to obtain a decoding model that has completed training, wherein the target text object is an input sample, the encoding information is an output sample, and the loss function of the decoding model is constructed based on adjacent identifier probability values, non-adjacent identifier probability values, and normalized adjacent identifier probability values ​​and normalized non-adjacent identifier probability values.

[0123] An embodiment of the present invention provides a device for encoding text objects. The device determines each identifier pair and a target text object in a text identifier sequence, and counts the number of adjacent pairs and non-adjacent pairs that appear in the text identifier sequence relative to the target text object, thereby encoding the target text object based on the number of adjacent pairs and the number of non-adjacent pairs to obtain encoding information, thereby realizing the fusion of information in multiple tokens in adjacent or non-adjacent forms, avoiding the single learning target problem of one-hot encoding, and thus improving the processing accuracy of text tasks.

[0124] According to one embodiment of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the computer executable instruction can execute the text object encoding method in any of the above method embodiments.

[0125] Figure 4 A schematic diagram of the structure of a terminal provided according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the terminal.

[0126] like Figure 4 As shown, the terminal may include: a processor (processor) 402 , a communication interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

[0127] The processor 402 , the communication interface 404 , and the memory 406 communicate with each other via a communication bus 408 .

[0128] The communication interface 404 is used to communicate with other devices such as clients or other servers.

[0129] The processor 402 is used to execute the program 410, and specifically can execute the relevant steps in the above-mentioned text object encoding method embodiment.

[0130] Specifically, the program 410 may include program codes, which include computer operation instructions.

[0131] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention. The one or more processors included in the terminal may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0132] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0133] The program 410 may be specifically configured to enable the processor 402 to perform the following operations:

[0134] Get a sequence of text identifiers;

[0135] Determine each identifier pair in the text identifier sequence and a target text object;

[0136] The number of adjacent pairs and the number of non-adjacent pairs corresponding to the target text object are determined based on the identifier pair, and the target text object is encoded based on the number of adjacent pairs and the number of non-adjacent pairs to obtain encoding information.

[0137] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for encoding a text object, characterized in that: The method comprises: Get a sequence of text identifiers; Determine each identifier pair in the text identifier sequence and a target text object; The number of adjacent pairs and the number of non-adjacent pairs corresponding to the target text object are determined based on the identifier pair, and the target text object is encoded based on the number of adjacent pairs and the number of non-adjacent pairs to obtain encoding information.

2. The method according to claim 1, characterized in that The encoding of the target text object based on the adjacent pairs and the non-adjacent pairs further comprises: Perform a first operation based on the adjacent pairs and the non-adjacent pairs to obtain a first number of 2-tuples of the target text object; Perform a second operation based on the number of adjacent pairs and the first number of two-tuples to obtain a first probability value of the adjacent identifier; Perform a third operation based on the number of non-adjacent pairs and the first number of two-tuples to obtain a second probability value of the non-adjacent identifier; The encoding information of the target text object is obtained by combining the first probability value and the second probability value.

3. The method according to claim 2, characterized in that The encoding of the target text object based on the adjacent pairs and the non-adjacent pairs further comprises: If the number of adjacent pairs is less than the number of non-adjacent pairs, performing a fourth operation based on the number of adjacent pairs, the number of non-adjacent pairs and the adjustment coefficient to obtain a second number of two-tuples; Perform a fifth operation based on the number of adjacent pairs, the number of non-adjacent pairs, and the second number of two-tuples to obtain a third probability value of the adjacent identifier; Performing a sixth operation based on the number of non-adjacent pairs and the second number of two-tuples to obtain a fourth probability value of the non-adjacent identifier; Based on the combination of the third probability value and the fourth probability value, the adjusted coding information is obtained.

4. The method according to claim 2, characterized in that: The method further comprises: If the number of non-adjacent pairs is less than a preset threshold, configuring the second probability value as a preset probability value; The obtaining the encoding information of the target text object based on the combination of the first probability value and the second probability value includes: Based on the combination of the first probability value and the preset probability value, the encoding information of the target text object is obtained.

5. The method according to claim 1, characterized in that Determining each identifier pair in the text identifier sequence and the target text object comprises: Based on all identifiers in the text identifier sequence, construct identifier pairs in the form of two tuples one by one, and count the logarithms of the identifier pairs; determine the left identifier in the identifier pair corresponding to the largest value in the logarithms as the target text object; or, The left identifier in each of the identifier pairs is determined as the target text object.

6. The method according to claim 1, characterized in that The determining the number of adjacent pairs and the number of non-adjacent pairs corresponding to the target text object based on the identifier pair comprises: Counting the number of adjacent pairs of adjacent identifiers formed by the target text object and the adjacent identifiers; The number of non-adjacent pairs of non-adjacent identifier pairs formed by the target text object and the non-adjacent identifiers is counted.

7. The method according to any one of claims 1 to 6, characterized in that: After encoding the target text object based on the adjacent pairs and the non-adjacent pairs to obtain encoding information, the method further includes: The decoding model is trained based on the target text object and the encoding information to obtain a trained decoding model, wherein the target text object is an input sample, the encoding information is an output sample, and the loss function of the decoding model is constructed based on adjacent identifier probability values, non-adjacent identifier probability values, normalized adjacent identifier probability values, and normalized non-adjacent identifier probability values.

8. A text object encoding device, characterized in that: The device comprises: An acquisition module, used for acquiring a text identifier sequence; A determination module, used for determining each identifier pair in the text identifier sequence and a target text object; The encoding module is used to determine the number of adjacent pairs and the number of non-adjacent pairs corresponding to the target text object based on the identifier pair, and encode the target text object based on the number of adjacent pairs and the number of non-adjacent pairs to obtain encoding information.

9. A computer device, characterized in that: The device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; A processor, for implementing the steps of the text object encoding method described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for encoding a text object according to any one of claims 1 to 7 are implemented.

Images