Position encoding method and apparatus, and storage medium

Abstract

The present disclosure relates to a position encoding method, device and storage medium. The method comprises: determining position information of each text unit in a text to be processed, encoding the position information according to a first encoding mode to obtain a first position encoding result, encoding the position information according to a second encoding mode to obtain a second position encoding result, and obtaining a target position encoding result based on the first position encoding result and the second position encoding result, wherein the first position encoding mode is different from the second position encoding mode. The present disclosure can simply and accurately enhance the relative position encoding information corresponding to each text unit in the text to be processed without significantly increasing the amount of parameters and the amount of calculation, and effectively improve the accuracy of subsequent model result prediction.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a location encoding method, apparatus and storage medium. Background Technology

[0002] With the continuous development of neural networks, researchers have comprehensively analyzed existing parallel processing models based on Convolutional Neural Networks (CNNs) and gated long short-term memory (LSTM) temporal translation models based on Recurrent Neural Networks (RNNs). They summarized the advantages and disadvantages of both models and proposed a translation model (Transformer) based on a self-attention mechanism. The Transformer model does not use the methods and modules of CNN and RNN models; instead, it pioneers a new approach by using an attention mechanism as the core of the encoder-decoder to construct the translation execution mechanism, contributing entirely new solutions to other areas of Natural Language Processing (NLP). Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides a location encoding method, apparatus and storage medium.

[0004] According to a first aspect of the present disclosure, a position encoding method is provided, comprising:

[0005] Determine the position information of each text unit in the text to be processed;

[0006] The position information is encoded according to the first position encoding method to obtain the first position encoding result;

[0007] The position information is encoded according to the second position encoding method to obtain the second position encoding result;

[0008] Based on the first position encoding result and the second position encoding result, the target position encoding result is obtained;

[0009] The first position encoding method is different from the second position encoding method.

[0010] In some embodiments, determining the position information of each text unit in the text to be processed includes:

[0011] The text to be processed is segmented to obtain individual text units; wherein, each text unit includes at least: characters and words;

[0012] Determine the word segmentation order corresponding to each of the aforementioned text units;

[0013] The position information is determined based on the word segmentation order.

[0014] In some embodiments, encoding the location information according to a first location encoding method to obtain a first location encoding result includes:

[0015] The location information is encoded according to a preset encoding model to obtain an encoding result in a preset format;

[0016] The encoding result of the preset format is multiplied with the custom feature to obtain the first position encoding result;

[0017] The dimension of the custom feature is determined by the number of text units in the text to be processed and the dimension of the position encoding result of the preset format.

[0018] In some embodiments, encoding the location information according to the second location encoding method to obtain the second location encoding result includes:

[0019] Determine the difference between the positional information of each text unit in the text to be processed;

[0020] Based on the difference, the positional difference information corresponding to each text unit is obtained respectively;

[0021] Based on at least two conversion strategies, the positional difference information is transformed by function to obtain the second positional encoding result.

[0022] In some embodiments, the step of performing function transformation on each of the position difference information according to at least two transformation strategies to obtain the second position encoding result includes:

[0023] For elements inserted into even positions in a preset sequence, the first transformation strategy is used to process the position difference information to obtain the value corresponding to the element at the even position;

[0024] For elements inserted into odd positions in the preset sequence, a second transformation strategy is used to process the position difference information to obtain the value corresponding to the element at the odd position;

[0025] The values ​​corresponding to the elements at even positions and the values ​​corresponding to the elements at odd positions are inserted into the preset sequence to obtain the second position encoding result.

[0026] In some embodiments, the first conversion strategy includes: a sine conversion strategy;

[0027] The second conversion strategy includes: cosine conversion strategy.

[0028] In some embodiments, obtaining the target encoding result based on the first position encoding result and the second position encoding result includes:

[0029] The first position encoding result and the second position encoding result are added together to obtain the target position encoding result.

[0030] In some embodiments, the method further includes:

[0031] Obtain the text encoding result corresponding to each text unit in the text to be processed;

[0032] The text encoding result and the target position encoding result are added together to obtain the temporal encoding result corresponding to each text unit in the text to be processed;

[0033] The temporal coding results corresponding to each text unit in the text to be processed are input into the generation model to obtain the response text corresponding to the text to be processed.

[0034] According to a second aspect of the present disclosure, a position encoding device is provided, comprising:

[0035] The first determining module is configured to determine the position information of each text unit in the text to be processed.

[0036] The first encoding module is configured to encode the position information according to a first position encoding method to obtain a first position encoding result;

[0037] The second encoding module is configured to encode the position information according to the second position encoding method to obtain the second position encoding result;

[0038] The second determining module is configured to obtain the target location encoding result based on the first location encoding result and the second location encoding result;

[0039] The first position encoding method is different from the second position encoding method.

[0040] In some embodiments, the first determining module is configured to:

[0041] The text to be processed is segmented to obtain individual text units; wherein, each text unit includes at least: characters and words;

[0042] Determine the word segmentation order corresponding to each of the aforementioned text units;

[0043] The position information is determined based on the word segmentation order.

[0044] In some embodiments, the first encoding module is configured as follows:

[0045] The location information is encoded according to a preset encoding model to obtain an encoding result in a preset format;

[0046] The encoding result of the preset format is multiplied with the custom feature to obtain the first position encoding result;

[0047] The dimension of the custom feature is determined by the number of text units in the text to be processed and the dimension of the position encoding result of the preset format.

[0048] In some embodiments, the second encoding module is configured to:

[0049] Determine the difference between the positional information of each text unit in the text to be processed;

[0050] Based on the difference, the positional difference information corresponding to each text unit is obtained respectively;

[0051] Based on at least two conversion strategies, the positional difference information is transformed by function to obtain the second positional encoding result.

[0052] In some embodiments, the second encoding module is configured to:

[0053] For elements inserted into even positions in a preset sequence, the first transformation strategy is used to process the position difference information to obtain the value corresponding to the element at the even position;

[0054] For elements inserted into odd positions in the preset sequence, a second transformation strategy is used to process the position difference information to obtain the value corresponding to the element at the odd position;

[0055] The values ​​corresponding to the elements at even positions and the values ​​corresponding to the elements at odd positions are inserted into the preset sequence to obtain the second position encoding result.

[0056] In some embodiments, the first conversion strategy includes: a sine conversion strategy;

[0057] The second conversion strategy includes: cosine conversion strategy.

[0058] In some embodiments, the second determining module is configured to:

[0059] The first position encoding result and the second position encoding result are added together to obtain the target position encoding result.

[0060] In some embodiments, the apparatus further includes:

[0061] The acquisition module is configured to acquire the text encoding results corresponding to each text unit in the text to be processed;

[0062] The addition module is configured to add the text encoding result and the target position encoding result to obtain the temporal encoding result corresponding to each text unit in the text to be processed;

[0063] The input module is configured to input the temporal coding results corresponding to each text unit in the text to be processed into the generation model to obtain the response text corresponding to the text to be processed.

[0064] According to a third aspect of the present disclosure, a position encoding device is provided, comprising:

[0065] processor;

[0066] Memory configured to store processor-executable instructions;

[0067] The processor is configured to implement the steps of any of the position encoding methods in the first aspect above during execution.

[0068] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which, when instructions in the storage medium are executed by a processor of a position encoding device, enables the device to perform the steps of any of the position encoding methods in the first aspect described above.

[0069] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0070] In this embodiment of the disclosure, the position information of each text unit in the determined text to be processed can be encoded according to a first position encoding method to obtain a first position encoding result, and the position information can be encoded according to a second position encoding method different from the first position encoding method to obtain a second position encoding result. Then, based on the first position encoding result and the second position encoding result, the target position encoding result corresponding to the text to be processed can be obtained.

[0071] This disclosure obtains a target position encoding result containing the absolute and relative position information of each text unit in the text by acquiring multiple different position encoding results corresponding to the text to be processed. Without significantly increasing the number of parameters and computational load, it can simply and accurately enhance the relative position encoding information of each text unit in the text to be processed during the prediction process of the generative model. Compared with related technologies that only use absolute position information to determine the target position encoding result, it effectively improves the accuracy of the prediction result of the generative model.

[0072] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0073] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0074] Figure 1 This is a flowchart illustrating a position encoding method according to an exemplary embodiment of the present disclosure.

[0075] Figure 2 This is a schematic diagram illustrating a location encoding application scenario according to an exemplary embodiment of the present disclosure.

[0076] Figure 3 This is a block diagram of a position encoding device according to an exemplary embodiment of the present disclosure.

[0077] Figure 4 This is a hardware structure block diagram of a position encoding device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0078] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0079] Figure 1 This is a flowchart illustrating a position encoding method according to an exemplary embodiment, such as... Figure 1 As shown, this method is applied to electronic devices and mainly includes the following steps:

[0080] In step 101, the position information of each text unit in the text to be processed is determined;

[0081] In step 102, the position information is encoded according to the first position encoding method to obtain the first position encoding result;

[0082] In step 103, the position information is encoded according to the second position encoding method to obtain the second position encoding result;

[0083] In step 104, the target location encoding result is obtained based on the first location encoding result and the second location encoding result.

[0084] Here, electronic devices can include terminal devices, such as mobile terminals or fixed terminals. Mobile terminals can include devices such as mobile phones, tablets, laptops, or wearable devices, and can also include smart home devices such as smart speakers. Fixed terminals can include desktop computers or smart TVs.

[0085] It's important to note that Natural Language Processing (NLP) is a crucial area within computer science and artificial intelligence. NLP is closely related to linguistics, integrating linguistics, computer science, and mathematics. Its main focus is on the methods and theories of effective human-computer interaction through natural language. In other words, through NLP technologies, electronic devices can directly recognize natural language provided by users in the form of speech or text and respond accordingly, such as performing specific operations or providing responses in natural language based on the user's intent.

[0086] Natural Language Processing (NLP) encompasses processing languages ​​commonly used by users, such as Chinese, English, and Russian, including machine translation, sentiment monitoring, automatic summarization, opinion extraction, text classification, question answering, knowledge graphs, sentiment analysis, text semantic comparison, and speech recognition. To implement these NLP technologies, an essential step is to decompose the user-provided text into multiple text units, and then convert each text unit into a corresponding word vector. This transforms the user-input text into a sequence of word vectors. Only then can the electronic device process this word vector sequence.

[0087] In this embodiment of the disclosure, the text to be processed can refer to the text that needs to be processed. The text can be a sentence or a combination of sentences with complete and systematic meaning. A text can be a sentence, a paragraph, or a chapter, such as short texts like short sentences, proverbs, maxims, and titles, or long texts like articles and document texts. The text to be processed can be represented in the form of vector sequences (e.g., but not limited to [I do not like the story of the movie]), and this embodiment of the disclosure does not specifically limit this. A text unit can refer to a single character or word in the text to be processed. For example, if the text to be processed is a sentence "Zhang San took the high-speed rail to City A", then the text units contained in the text to be processed are "Zhang San", "take", "high-speed rail", "go to", and "City A" in sequence. The location information can refer to information such as the order attribute of the text unit relative to the text to be processed. If the location information of a certain text unit in the text to be processed is changed, the semantics of the text to be processed may be completely changed. Therefore, in the natural language processing process, the location information of each text unit can be determined first. For example, the location information corresponding to "Zhang San" can be 1, the location information corresponding to "Cheng" can be 2, the location information corresponding to "A City" can be 5, etc.

[0088] In one possible embodiment, the electronic device can directly receive location information manually marked by the user. For example, the electronic device can display the text to be processed, "The animal did not cross the street because it was too tired," through a display screen or other device. Then, the user can directly mark the location information corresponding to each text unit using a mouse, touch screen, or other components (e.g., but not limited to marking "The" with a value of 1, "animal" with a value of 2, etc.).

[0089] In one possible embodiment, the electronic device can also determine the position information by recognizing the typing order of each text unit in the text to be processed. The typing order refers to the sequence in which the text units are entered into the electronic device. Due to user habits, users usually input the text units in sequence to form a complete text to be processed. For example, the electronic device can use the "Enter" button on a mouse, keyboard, or touch screen to type the first text unit "The", and then type the second text unit "animal". Based on the corresponding typing order, the electronic device can determine the position information of the text unit "The" as 1, the position information of the text unit "animal" as 2, and so on. Determining the position information of each text unit in the above way helps the electronic device to easily and accurately determine the position information of each text unit in the text to be processed, reducing the amount of computation and improving the operating efficiency of the electronic device.

[0090] In some embodiments, the representation of location information can take many forms, including numerical values ​​and vectors, and this disclosure does not impose specific limitations. In one possible embodiment, the electronic device can first determine the number of text units corresponding to the text to be processed, and then set a sequence of a corresponding size for the location information. For example, if the electronic device determines that the number of text units in the text to be processed is 5, then the electronic device can determine that the location information can be represented by the sequence [0,1,2,3,4]. The electronic device can linearly assign a value to each text unit to represent the relationship between the order of different text units, that is, 0 is assigned to the first text unit in the text to be processed, 1 is assigned to the second text unit in the text to be processed, and so on. The electronic device can preset a maximum number of text units (e.g., but not limited to 192) to set the sequence corresponding to the location information to prevent the sequence from exceeding the range.

[0091] In this embodiment of the disclosure, after the electronic device determines the position information of each text unit in the text to be processed, it can encode the position information according to different encoding methods to obtain different position encoding results. That is, the first position encoding method and the second position encoding method are different. The encoding method can refer to the conversion of position information in a certain form into another data that meets preset conditions or formats through a set conversion technology. The preset conditions may include operations such as feature extraction, filtering enhancement, fitting, and time-domain-frequency-domain transformation. The position encoding result can refer to the secondary representation result after processing the position information according to the corresponding encoding method. There is a corresponding mapping relationship between different encoding methods and different position encoding results.

[0092] There are various encoding methods, and this disclosure does not impose specific limitations on the embodiments. For example, an electronic device can pre-set a neural network model, then use position information as input to the neural network model, and determine that the maximum length of the neural network model output is 512 and the dimension is 768. This yields a 512*768 matrix as the position encoding result, which can be updated during the training process. Alternatively, the electronic device can also determine the corresponding position encoding result using encoding methods such as trigonometric function processing, recursive processing, multiplication processing, negative number processing, or sine and cosine processing. The position encoding result can be represented in matrix form. The position encoding result for a single text unit can be a one-dimensional matrix, while the position encoding result for the text to be processed, which can be combined from multiple text units, can be a multi-dimensional matrix. Combining multiple text units into a multi-dimensional matrix form facilitates parallel computing in the electronic device, improving computational efficiency.

[0093] In one possible embodiment, the first positional encoding result can refer to an absolute positional encoding result (also known as parametric absolute positional encoding, etc.), and the second positional encoding result can refer to a relative positional encoding result (also known as functional positional encoding, etc.). The absolute positional encoding result can be used to determine the absolute information of each text unit relative to the text to be processed, while the relative positional encoding result can be used to determine the relative information between different text units. For example, in the text to be processed, "Zhang San took the high-speed train to City A," the absolute positional encoding result for the text unit "Zhang San" can be 1, and the relative positional encoding result between the text unit "Zhang San" and the text unit "high-speed train" can be 2, etc. Because the position or arrangement order of a text unit in the text to be processed may differ, the meaning of the entire text may be distorted. By determining multiple different positional encoding results, the electronic device can more accurately perform corresponding natural language processing on the text to be processed.

[0094] In this embodiment, the electronic device encodes the location information according to different encoding methods to obtain different location encoding results. Then, a target location encoding result can be determined based on these different location encoding results. Here, the target location encoding result can refer to a comprehensive location encoding result determined by the text unit based on the location information, containing features of multiple location encoding results. The method for determining the target location encoding result in this embodiment is not specifically limited. For example, if the electronic device determines the first location encoding result as [1,2,3] and the second location encoding result as [4,5,6], then the electronic device can perform convolution processing on the different location encoding results to obtain the corresponding target location encoding result, such as 32.

[0095] In this embodiment of the disclosure, the location encoding method can be applied to various practical scenarios, such as intelligent question-answering systems in intelligent robots. Figure 2The diagram illustrates an application scenario of positional encoding in natural language processing. During voice interaction with an intelligent robot, the robot first converts the received voice signal into corresponding text to be processed. Then, it determines the interaction type (e.g., but not limited to voice control, casual conversation, etc.) based on the text. When determining that the current voice interaction belongs to casual conversation, the intelligent robot first identifies the corresponding question entity 201 (Query) in the text to be processed. It then searches the retrieval database 202 to obtain multiple candidate entities 203 (Candidates) corresponding to question entity 201. Based on question entity 201 and candidate entities 203, multiple triplets in the form of <question entity (Query), attribute (Post), candidate entity (Candidates)> can be generated.

[0096] Then, the intelligent robot can use ranking model 204 (BertRank) to sort all the triples generated from question entity 201 and candidate entity 203, obtaining multiple candidate sequences. The retrieval model 205 (Selector) then processes these candidate sequences, determining the answer entity 207 (Reply) corresponding to question entity 201 by obtaining the candidate sequence with the highest accuracy. If the retrieval model 205 does not determine the answer entity 207 corresponding to question entity 201, the intelligent robot can choose to use generation model 206 (Generator) to determine the accurate answer entity 207. Generation model 206 can be a Transformer model trained based on the positional encoding method described in this embodiment. Researchers found that since the Transformer model does not include recursive and convolutional processing mechanisms, in order to enable the Transformer model to utilize the order information of the input sequence to further improve the accuracy of model prediction, the Transformer model needs to import some information about the relative or absolute positions of the markers in the input sequence. In practical use, it has been found that the location features constructed through the technical solutions in related technologies can easily cause the self-attention layer in the Transformer model to lose relative location feature information during the calculation process, thereby affecting the accuracy of the Transformer model's prediction results.

[0097] In this embodiment, based on the determined positional information of each text unit in the text to be processed, the positional information is encoded according to a first positional encoding method to obtain a first positional encoding result. Then, the positional information is encoded according to a second positional encoding method different from the first positional encoding method to obtain a second positional encoding result. Finally, based on the first and second positional encoding results, a target positional encoding result corresponding to the text to be processed is obtained. This disclosure obtains a target positional encoding result containing the absolute and relative positional information of each text unit in the text to be processed by acquiring multiple different positional encoding results corresponding to the text to be processed. Without significantly increasing the number of parameters or computational load, it can simply and accurately enhance the relative positional encoding information of each text unit in the text to be processed during the prediction process of the generative model, effectively improving the accuracy of the prediction results of the generative model.

[0098] In some embodiments, determining the position information of each text unit in the text to be processed includes:

[0099] The text to be processed is segmented to obtain individual text units; wherein, each text unit includes at least: characters and words;

[0100] Determine the word segmentation order corresponding to each of the aforementioned text units;

[0101] The position information is determined based on the word segmentation order.

[0102] In this embodiment, word segmentation is fundamental to natural language processing, and its accuracy directly determines the quality of subsequent part-of-speech tagging, syntactic analysis, word vectorization, and text analysis. English sentences use spaces to separate words, and except for certain specific words such as "how many" and "New York," word segmentation is generally unnecessary. However, Chinese is different. Due to the lack of delimiters (such as spaces), word segmentation and sentence breaking need to be performed beforehand. Therefore, electronic devices can perform word segmentation before performing Chinese natural language processing. In this embodiment, the electronic device can perform word segmentation on the text to be processed using dictionary-based rule matching methods or statistical machine learning methods, which helps the electronic device accurately and quickly determine location information.

[0103] The electronic device performs word segmentation on the text to be processed, obtaining individual text units. It can then determine the segmentation order of each text unit and, based on this order, determine the positional information. The segmentation order refers to the sequential order of the text units. For example, the electronic device can use a dictionary-based rule matching method to segment the text "married and unmarried colleagues," obtaining text units such as "married," "and," "unmarried," "of," and "colleague." The electronic device can then determine that the segmentation order for the text unit "married" is 1, the segmentation order for the text unit "and" is 2, and so on. Therefore, it can determine the positional information for the text unit "married" as 1, and the positional information for the text unit "and" as 2, and so on.

[0104] In this embodiment, the text to be processed can first be segmented to obtain various text units, wherein each text unit includes at least characters and words. Then, the segmentation order corresponding to each text unit is determined. Finally, the position information can be determined based on the segmentation order. This disclosure can easily and accurately determine position information and improve operating efficiency by segmenting the text to be processed.

[0105] In some embodiments, encoding the location information according to a first location encoding method to obtain a first location encoding result includes:

[0106] The location information is encoded according to a preset encoding model to obtain an encoding result in a preset format;

[0107] The encoding result of the preset format is multiplied with the custom feature to obtain the first position encoding result;

[0108] The dimension of the custom feature is determined by the number of text units in the text to be processed and the dimension of the position encoding result of the preset format.

[0109] In this embodiment of the disclosure, after determining the location information, the electronic device can obtain a first encoding result based on the location information. The preset encoding model can refer to a model that converts the format of the location information, such as a one-hot encoding model, also known as one-bit valid encoding. The preset format can include the dimension size or representation of the encoding result, such as the electronic device converting the location information from a numerical format to a vector format, which helps reduce the complexity of subsequent calculations. For example, if the electronic device determines that the location information corresponding to the text to be processed, "Zhang San takes the high-speed train to City A," can be [0,1,2,3,4], then the encoding result of the preset format corresponding to the text to be processed can be [1,0,0,0,0;0,1,0,0,0;0,0,1,0,0;0,0,0,1,0;0,0,0,0,1]. That is, the preset format encoding result corresponding to the text unit "Zhang San," whose location information can be 1, is [1,0,0,0,0]. The 1*5 dimension of the matrix can be determined by the number of 5 text units, or a default length can be preset. Then, the encoding results of all text units in the preset format are combined into the encoding result of the text to be processed in the preset format, which helps electronic devices to perform parallel computing and reduce the computation time.

[0110] In some embodiments, the electronic device can multiply the encoding result of the preset format with a custom feature to obtain the first positional encoding result. The custom feature matrix can be a matrix with randomly initialized element values, where the dimension of the custom feature is determined by the number of text units in the text to be processed and the dimension of the positional encoding result of the preset format. For example, if the number of text units in the text to be processed is 10, and the dimension of the positional encoding result of the text unit in the preset format is 1*3, then the dimension of the custom feature can be 10*3, etc. The multiplication of the encoding result of the preset format with the custom feature can be understood as feature extraction. For example, if the dimension of the encoding result of the text to be processed is 10*5, and the dimension of the custom feature is 5*3, then the dimension of the first positional encoding result of the text to be processed is 10*3. The electronic device can transform the encoding result of the preset format from a sparse state to a dense state of the first positional encoding result, and turn mutually independent vectors into relational vectors with inherent connections, thus improving subsequent computational efficiency while representing the absolute positional information corresponding to each text unit.

[0111] In this embodiment, the location information can first be encoded according to a preset encoding model to obtain an encoding result in a preset format. Then, the encoding result in the preset format is multiplied by a custom feature to obtain the first location encoding result. The dimension of the custom feature is determined by the number of text units in the text to be processed and the dimension of the location encoding result in the preset format. This disclosure can easily and quickly obtain the first location encoding result carrying absolute location information, and reduces computational complexity.

[0112] In some embodiments, encoding the location information according to the second location encoding method to obtain the second location encoding result includes:

[0113] Determine the difference between the positional information of each text unit in the text to be processed;

[0114] Based on the difference, the positional difference information corresponding to each text unit is obtained respectively;

[0115] Based on at least two conversion strategies, the positional difference information is transformed by function to obtain the second positional encoding result.

[0116] In this embodiment of the disclosure, after obtaining the first position encoding result, the electronic device can first determine the difference between the position information of each text unit in the text to be processed. For example, the difference between "Zhang San" and "multiply" is 1, and the difference between "Zhang San" and "high-speed rail" is 2, etc. Then, the electronic device can obtain the position difference information corresponding to each text unit according to the difference. The position difference information can refer to the difference information between the absolute position information of a certain text unit and other text units. For example, the position difference information corresponding to "multiply" can be -1, 0, 1, 2, 3, etc. Then, the electronic device can perform function transformation on each position difference information according to at least two transformation strategies to obtain the second position encoding result. The transformation strategy can refer to the function method used to transform the representation of the position difference information. For example: absolute value transformation strategy, that is, performing absolute value calculation on the position difference information; reciprocal transformation strategy, that is, performing reciprocal calculation on the position difference information; binary transformation strategy, that is, performing binary calculation on the position difference information, etc. Electronic devices may employ two, three, or four conversion strategies, or they may employ two conversion strategies separately or sequentially. This embodiment does not impose any specific limitations.

[0117] In one possible embodiment, if the text to be processed corresponds to 5 text units, the electronic device can use the first conversion strategy to calculate the second position encoding result for each text unit at odd positions 1, 3, and 5, and use the second conversion strategy to calculate the second position encoding result for each text unit at even positions 2 and 4. Finally, a combination process is performed to obtain the second position encoding result for all text units. Alternatively, the electronic device can use the first conversion strategy to calculate result A for each text unit, and use the second conversion strategy to calculate result B for each text unit. Finally, a combination process such as addition and subtraction can be performed to obtain the second position encoding result. In this embodiment, the function transformation method corresponding to at least two conversion strategies is not specifically limited and can be customized according to the user's actual needs.

[0118] In this embodiment, the difference between the positional information of each text unit in the text to be processed can be determined first. Then, based on the difference, the positional difference information corresponding to each text unit can be obtained. Finally, the positional difference information can be transformed by functions according to at least two transformation strategies to obtain the second positional encoding result. This disclosure can obtain the second positional encoding result carrying relative positional information simply and accurately, increasing the richness of the positional information corresponding to each text unit.

[0119] In some embodiments, the step of performing function transformation on each of the position difference information according to at least two transformation strategies to obtain the second position encoding result includes:

[0120] For elements inserted into even positions in a preset sequence, the first transformation strategy is used to process the position difference information to obtain the value corresponding to the element at the even position;

[0121] For elements inserted into odd positions in the preset sequence, a second transformation strategy is used to process the position difference information to obtain the value corresponding to the element at the odd position;

[0122] The values ​​corresponding to the elements at even positions and the values ​​corresponding to the elements at odd positions are inserted into the preset sequence to obtain the second position encoding result.

[0123] In this embodiment, the electronic device can first process the position difference information of the elements to be inserted into the even-numbered positions of the preset sequence using a first conversion strategy to obtain the values ​​corresponding to the even-numbered positions. The preset sequence can refer to a container used to store intermediate calculation results. For example, the preset sequence can be a one-dimensional matrix 1*N with N elements from 1 to N, where 1, 3, 5... are odd-numbered positions, and 2, 4, 6... are even-numbered positions. Then, the electronic device can process the position difference information using the first conversion strategy to obtain the values ​​corresponding to the even-numbered positions 2, 4, 6... Then, it can process the position difference information using a second conversion strategy to obtain the values ​​corresponding to the odd-numbered positions 1, 3, 5... Finally, the electronic device can combine the values ​​corresponding to the odd-numbered positions and the even-numbered positions to obtain the values ​​corresponding to all positions 1, 2, 3... N in the preset sequence 1*N. The interpolated preset sequence can then be used as the second position encoding result. That is, the electronic device can insert the values ​​corresponding to the even-numbered positions and the values ​​corresponding to the odd-numbered positions into the preset sequence to obtain the second position encoding result.

[0124] In one possible embodiment, the electronic device processes the position difference information of the element to be inserted into position A of the preset sequence using a first conversion strategy to obtain the value corresponding to the element at position A. For the element to be inserted into position B of the preset sequence, a second conversion strategy is used to process the position difference information to obtain the value corresponding to the element at position B. For the element to be inserted into position C of the preset sequence, a third conversion strategy is used to process the position difference information to obtain the value corresponding to the element at position C. Then, the values ​​corresponding to the elements at positions A, B, and C are inserted into the preset sequence to obtain the second position encoding result. The number of conversion strategies used and the specified positions of each element in the corresponding preset sequence are not specifically limited and can be customized according to user needs.

[0125] In this embodiment, the element to be inserted into the even-numbered position of the preset sequence is first processed using a first transformation strategy to obtain the value corresponding to the element at the even-numbered position. Then, the element to be inserted into the odd-numbered position of the preset sequence is processed using a second transformation strategy to obtain the value corresponding to the element at the odd-numbered position. Finally, the values ​​corresponding to the even-numbered and odd-numbered positions are inserted into the preset sequence to obtain the second positional encoding result. This disclosure can obtain the second positional encoding result through different transformation strategies, accurately and effectively extracting the features of the positional difference information, and increasing the richness of the relative positional information carried in the second positional encoding result.

[0126] In some embodiments, the first conversion strategy includes: a sine conversion strategy;

[0127] The second conversion strategy includes: cosine conversion strategy.

[0128] In this embodiment of the disclosure, the sine conversion strategy can refer to a conversion strategy based on the sine function, and the cosine conversion strategy can refer to a conversion strategy based on the cosine function. The calculation formula for the sine conversion strategy is as follows:

[0129] ; (1)

[0130] The calculation formula for the cosine transformation strategy is as follows:

[0131] ; (2)

[0132] In formulas (1) and (2), This represents the value corresponding to the element in the preset sequence. This represents the value corresponding to the even-numbered element in the preset sequence. This represents the value corresponding to the element at the odd position in the preset sequence. This represents the dimension number corresponding to each element in the preset sequence. and This indicates the positional information corresponding to different text units in the text to be processed. This indicates the location difference information. It is a positive integer (e.g., but not limited to 10000). This indicates the dimension corresponding to the second position encoding result. Indicates sine conversion, This indicates the cosine transformation.

[0133] In this embodiment of the disclosure, by setting the first conversion strategy to a sine conversion strategy and the second conversion strategy to a cosine conversion strategy, it is possible to effectively ensure that the conversion can be performed accurately even when the sequence or vector corresponding to the second position encoding result is too long.

[0134] In some embodiments, obtaining the target encoding result based on the first position encoding result and the second position encoding result includes:

[0135] The first position encoding result and the second position encoding result are added together to obtain the target position encoding result.

[0136] In this embodiment of the disclosure, after the electronic device obtains the first position encoding result and the second position encoding result, it can obtain the target encoding result. For example, the electronic device can add the first position encoding result and the second position encoding result to obtain the target position encoding result. For example, if the electronic device obtains the first position encoding result corresponding to the text unit as [1,2,3,4,5] and the second position encoding result as [6,7,8,9,10], then the target position encoding result can be [7,9,11,13,15], etc.

[0137] In this embodiment of the disclosure, the first position encoding result and the second position encoding result can be added together, which can quickly and easily determine the target position encoding result.

[0138] In some embodiments, the method further includes:

[0139] Obtain the text encoding result corresponding to each text unit in the text to be processed;

[0140] The text encoding result and the target position encoding result are added together to obtain the temporal encoding result corresponding to each text unit in the text to be processed;

[0141] The temporal coding results corresponding to each text unit in the text to be processed are input into the generation model to obtain the response text corresponding to the text to be processed.

[0142] In this embodiment, the text encoding result can refer to a vector of real numbers mapped from text units in the form of words or phrases from a vocabulary, or it can be called a word vector result. The electronic device can use methods such as continuous skip elements and continuous bag-of-words, singular value decomposition, and Braun clustering to obtain the text encoding result corresponding to each text unit in the text to be processed. Then, the electronic device can add the text encoding result to the target position encoding result to obtain the temporal encoding result corresponding to each text unit in the text to be processed. The temporal encoding result can refer to comprehensive feature information containing word vector feature information and position information corresponding to the text to be processed, and can be represented in matrix form. For example, if the electronic device determines that the text encoding result corresponding to a text unit is [9,8,7,6,5], and the corresponding target position encoding is [0.5,0.2,0.7,0.1,0.6], then the corresponding temporal encoding result can be [9.5,8.2,7.7,6.1,5.6], etc.

[0143] The electronic device can then input the temporal coding results corresponding to each text unit in the text to be processed into the generative model to obtain the response text corresponding to the text to be processed. The generative model can refer to a Transformer model trained using historical temporal coding results and manually labeled historical response texts. The Transformer model is commonly used in the field of natural language processing, which involves technologies including, but not limited to, semantic understanding, machine translation, machine question answering, knowledge graphs, and sentiment analysis. The response text can refer to the output text of the generative model after processing the text to be processed. For example, if the generative model is used for machine translation and the text to be processed is "I am a student," then the corresponding response text could be "I am a student," etc.; if the generative model is used for knowledge graphs and the text to be processed is "XXX is tall," then the corresponding response text could be "2.26m," etc.

[0144] In this embodiment, the text encoding results corresponding to each text unit in the text to be processed can be obtained first. Then, the text encoding results are added to the target position encoding results to obtain the temporal encoding results corresponding to each text unit in the text to be processed. Finally, the temporal encoding results corresponding to each text unit in the text to be processed are input into the generation model to obtain the response text corresponding to the text to be processed. This disclosure, by combining the text encoding results and the target position encoding results and then inputting them into the generation model, can accurately and effectively improve the prediction results of the generation model.

[0145] The technical solution disclosed herein enables the encoding of positional information of each text unit in a given text to be processed, according to a first positional encoding method, to obtain a first positional encoding result. Then, the positional information is encoded according to a second positional encoding method, different from the first, to obtain a second positional encoding result. Finally, based on the first and second positional encoding results, a target positional encoding result corresponding to the text to be processed is obtained. This disclosure obtains a target positional encoding result containing both absolute and relative positional information of each text unit in the text to be processed by acquiring multiple different positional encoding results corresponding to the text to be processed. Without significantly increasing the number of parameters or computational load, it can simply and accurately enhance the relative positional encoding information of each text unit in the text to be processed during the prediction process of the generative model, effectively improving the accuracy of the generative model's prediction results.

[0146] Figure 3 This is a block diagram illustrating a position encoding device according to an exemplary embodiment. Figure 3 As shown, the device is applied to an electronic device, and the location encoding device 300 mainly includes:

[0147] The first determining module 301 is configured to determine the position information of each text unit in the text to be processed;

[0148] The first encoding module 302 is configured to encode the position information according to a first position encoding method to obtain a first position encoding result;

[0149] The second encoding module 303 is configured to encode the position information according to the second position encoding method to obtain the second position encoding result;

[0150] The second determining module 304 is configured to obtain the target position encoding result based on the first position encoding result and the second position encoding result;

[0151] The first position encoding method is different from the second position encoding method.

[0152] In some embodiments, the first determining module 301 is configured as follows:

[0153] The text to be processed is segmented to obtain individual text units; wherein, each text unit includes at least: characters and words;

[0154] Determine the word segmentation order corresponding to each of the aforementioned text units;

[0155] The position information is determined based on the word segmentation order.

[0156] In some embodiments, the first encoding module 302 is configured as follows:

[0157] The location information is encoded according to a preset encoding model to obtain an encoding result in a preset format;

[0158] The encoding result of the preset format is multiplied with the custom feature to obtain the first position encoding result;

[0159] The dimension of the custom feature is determined by the number of text units in the text to be processed and the dimension of the position encoding result of the preset format.

[0160] In some embodiments, the second encoding module 303 is configured as follows:

[0161] Determine the difference between the positional information of each text unit in the text to be processed;

[0162] Based on the difference, the positional difference information corresponding to each text unit is obtained respectively;

[0163] Based on at least two conversion strategies, the positional difference information is transformed by function to obtain the second positional encoding result.

[0164] In some embodiments, the second encoding module 303 is configured as follows:

[0165] For elements inserted into even positions in a preset sequence, the first transformation strategy is used to process the position difference information to obtain the value corresponding to the element at the even position;

[0166] For elements inserted into odd positions in the preset sequence, a second transformation strategy is used to process the position difference information to obtain the value corresponding to the element at the odd position;

[0167] The values ​​corresponding to the elements at even positions and the values ​​corresponding to the elements at odd positions are inserted into the preset sequence to obtain the second position encoding result.

[0168] In some embodiments, the first conversion strategy includes: a sine conversion strategy;

[0169] The second conversion strategy includes: cosine conversion strategy.

[0170] In some embodiments, the second determining module 304 is configured to:

[0171] The first position encoding result and the second position encoding result are added together to obtain the target position encoding result.

[0172] In some embodiments, the device 300 further includes:

[0173] The acquisition module is configured to acquire the text encoding results corresponding to each text unit in the text to be processed;

[0174] The addition module is configured to add the text encoding result and the target position encoding result to obtain the temporal encoding result corresponding to each text unit in the text to be processed;

[0175] The input module is configured to input the temporal coding results corresponding to each text unit in the text to be processed into the generation model to obtain the response text corresponding to the text to be processed.

[0176] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0177] Figure 4This is a hardware structure block diagram illustrating a location encoding device according to an exemplary embodiment. For example, device 400 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0178] Reference Figure 4 The device 400 may include one or more of the following components: a processing component 402, a memory 404, a power component 406, a multimedia component 408, an audio component 410, an input / output (I / O) interface 412, a sensor component 414, and a communication component 416.

[0179] Processing component 402 typically controls the overall operation of device 400, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 402 may include one or more processors 420 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 402 may include one or more modules to facilitate interaction between processing component 402 and other components. For example, processing component 402 may include a multimedia module to facilitate interaction between multimedia component 408 and processing component 402.

[0180] Memory 404 is configured to store various types of data to support the operation of device 400. Examples of such data include instructions for any application or method operating on device 400, contact data, phonebook data, messages, pictures, videos, etc. Memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0181] The power supply component 406 provides power to the various components of the device 400. The power supply component 406 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 400.

[0182] Multimedia component 408 includes a screen that provides an output interface between the device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 408 includes a front-facing camera and / or a rear-facing camera. When the device 400 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0183] Audio component 410 is configured to output and / or input audio signals. For example, audio component 410 includes a microphone (MIC) configured to receive external audio signals when device 400 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 404 or transmitted via communication component 416. In some embodiments, audio component 410 also includes a speaker for outputting audio signals.

[0184] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0185] Sensor assembly 414 includes one or more sensors for providing status assessments of various aspects of device 400. For example, sensor assembly 414 may detect the on / off state of device 400, the relative positioning of components such as the display and keypad of device 400, changes in the position of device 400 or a component of device 400, the presence or absence of user contact with device 400, the orientation or acceleration / deceleration of device 400, and temperature changes of device 400. Sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 414 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 414 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0186] Communication component 416 is configured to facilitate wired or wireless communication between device 400 and other devices. Device 400 can access wireless networks based on communication standards, such as Wi-Fi, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 416 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 416 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0187] In an exemplary embodiment, the apparatus 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0188] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, which can be executed by a processor 420 of the device 400 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0189] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a position encoding device, enables the position encoding device to perform a position encoding method applied to an electronic device, comprising:

[0190] Determine the position information of each text unit in the text to be processed;

[0191] The position information is encoded according to the first position encoding method to obtain the first position encoding result;

[0192] The position information is encoded according to the second position encoding method to obtain the second position encoding result;

[0193] Based on the first position encoding result and the second position encoding result, the target position encoding result is obtained;

[0194] The first position encoding method is different from the second position encoding method.

[0195] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0196] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A position encoding method, characterized in that, The method includes: Determine the position information of each text unit in the text to be processed; The positional information is encoded according to a first positional encoding method to obtain a first positional encoding result; the encoding of the positional information according to the first positional encoding method to obtain the first positional encoding result includes: encoding the positional information according to a preset encoding model to obtain an encoding result in a preset format; multiplying the encoding result in the preset format with a custom feature to obtain the first positional encoding result; wherein, the dimension of the custom feature is determined by the number of text units in the text to be processed and the dimension of the positional encoding result in the preset format; the first positional encoding result includes an absolute positional encoding result, which is used to determine the absolute information of each text unit relative to the text to be processed; The positional information is encoded according to a second positional encoding method to obtain a second positional encoding result; the process of encoding the positional information according to the second positional encoding method to obtain a second positional encoding result includes: determining the difference between the positional information of each text unit in the text to be processed; obtaining the positional difference information corresponding to each text unit based on the difference; and performing function transformation on each positional difference information according to at least two transformation strategies to obtain the second positional encoding result; the second positional encoding result includes a relative positional encoding result, which is used to determine the relative information between different text units; Based on the first position encoding result and the second position encoding result, the target position encoding result is obtained; The first position encoding method is different from the second position encoding method.

2. The method according to claim 1, characterized in that, Determining the position information of each text unit in the text to be processed includes: The text to be processed is segmented to obtain individual text units; wherein, each text unit includes at least: characters and words; Determine the word segmentation order corresponding to each of the aforementioned text units; The position information is determined based on the word segmentation order.

3. The method according to claim 1, characterized in that, The step of performing function transformations on each of the positional difference information according to at least two transformation strategies to obtain the second positional encoding result includes: For elements inserted into even positions in a preset sequence, the first transformation strategy is used to process the position difference information to obtain the value corresponding to the element at the even position; For elements inserted into odd positions in the preset sequence, a second transformation strategy is used to process the position difference information to obtain the value corresponding to the element at the odd position; The values ​​corresponding to the elements at even positions and the values ​​corresponding to the elements at odd positions are inserted into the preset sequence to obtain the second position encoding result.

4. The method according to claim 3, characterized in that, The first conversion strategy includes: a sine conversion strategy; The second conversion strategy includes: cosine conversion strategy.

5. The method according to claim 1, characterized in that, The step of obtaining the target encoding result based on the first position encoding result and the second position encoding result includes: The first position encoding result and the second position encoding result are added together to obtain the target position encoding result.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the text encoding result corresponding to each text unit in the text to be processed; The text encoding result and the target position encoding result are added together to obtain the temporal encoding result corresponding to each text unit in the text to be processed; The temporal coding results corresponding to each text unit in the text to be processed are input into the generation model to obtain the response text corresponding to the text to be processed.

7. A position encoding device, characterized in that, include: The first determining module is configured to determine the position information of each text unit in the text to be processed. The first encoding module is configured to encode the position information according to a first position encoding method to obtain a first position encoding result; the first encoding module is further configured to encode the position information according to a preset encoding model to obtain an encoding result in a preset format; and to multiply the encoding result in the preset format with a custom feature to obtain the first position encoding result; wherein, the dimension of the custom feature is determined by the number of text units in the text to be processed and the dimension of the position encoding result in the preset format; the first position encoding result includes an absolute position encoding result, which is used to determine the absolute information of each text unit relative to the text to be processed; The second encoding module is configured to encode the position information according to a second position encoding method to obtain a second position encoding result; the second encoding module is further configured to determine the difference between the position information of each text unit in the text to be processed; obtain the position difference information corresponding to each text unit according to the difference; and perform function transformation on each position difference information according to at least two transformation strategies to obtain the second position encoding result; the second position encoding result includes a relative position encoding result, which is used to determine the relative information between different text units; The second determining module is configured to obtain the target location encoding result based on the first location encoding result and the second location encoding result; The first position encoding method is different from the second position encoding method.

8. A position encoding device, characterized in that, include: processor; Memory configured to store processor-executable instructions; The processor is configured to implement the steps of any one of the position encoding methods in claims 1 to 6 during execution.

9. A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a position encoding device, the device is enabled to perform the steps of any of the position encoding methods of claims 1 to 6.

Images