Language model output messy code processing method and device and storage medium
By detecting the garbled word elements in the language model output sequence and adjusting its encoding format using the transfer matrix generated by historical text, the problem of garbled Chinese output in language model is solved, and efficient garbled processing and user experience improvement is achieved.
Patent Information
- Application Number
- CN202510055918.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-14
- Publication Date
- 2025-05-02
AI Technical Summary
Language models often have garbled problems when outputting Chinese results, which affects the user experience. Although existing solutions such as adding training corpus can reduce garbled code, they are costly and have limited results.
By obtaining the original output sequence of the target language model, the encoding format of its code body is detected for each word element. If it does not conform to the preset format, it is determined to be a garbled word element. Use the transfer matrix generated by historical text to adjust the garbled code body to the preset format, generate the target output sequence and perform decoding processing.
It effectively avoids the generation of garbled code, saves the cost of solving garbled code problems by expanding the corpus, and improves the processing effect of garbled code output in the language model, ensuring the integrity and user experience of the output results.
Smart Images
Figure CN119918504A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a method, device and storage medium for processing garbled code output by a language model. Background Art
[0002] Language models provide great convenience for people through their powerful language understanding and generation capabilities, efficient data processing speed, and extensive knowledge coverage. They can assist people in data consultation, literature analysis, language content creation, etc. However, when language models output Chinese results, garbled characters may appear, which seriously affects the user experience.
[0003] At present, the way to deal with garbled output of language models is usually to increase the training corpus when training the language model to improve the language model's generation ability, thereby reducing the possibility of generating garbled output. However, there are usually many reasons for the garbled output of language models. In addition to insufficient training corpus, there are also many other reasons, such as inconsistency between training encoding and generation encoding, incorrect display encoding, text segmentation errors, incomplete word list coverage, etc. Therefore, this method of expanding the corpus will inevitably increase the training cost, and the effect of dealing with the garbled output problem of language models is limited. Summary of the invention
[0004] In order to solve the above technical problems, the present application provides a method, device and storage medium for processing garbled output of a language model, which can better solve the problem of garbled output of a language model.
[0005] In a first aspect, the present application provides a method for processing garbled code output by a language model, comprising: obtaining an original output sequence of a target language model; the original output sequence is composed of at least one word unit, a word unit is composed of a terminal, or a word unit is composed of a terminal and at least one code body; for each word unit, when the word unit includes a code body and the encoding format of any code body of the word unit is not a preset format, the code body is determined to be a garbled code body, and the word unit is determined to be a garbled word unit; wherein, in the preset format, the target position of the code body encoding is the target string; the target position is the position of the first two characters of the first byte of the code body; the garbled code body is adjusted to the preset format according to the garbled word unit and the transfer matrix to obtain a target output sequence; the transfer matrix is generated according to the historical text input to the target language model; the target output sequence is decoded to generate the output result of the target language model.
[0006] In some embodiments, before matching the target code body in the transfer matrix based on the garbled word, the method also includes: obtaining historical text input into the target language model within a historical time period; determining special characters in the historical text, and generating a transfer matrix based on the special characters; the encoding length of the special characters is greater than a preset length, or the encoding result of the special characters is within a preset encoding range.
[0007] In some embodiments, determining special characters in historical text includes: decoding the historical text into UTF-8 encoded text, and determining characters represented by two bytes and three bytes in the text as special characters; or decoding the historical text into Unicode character text, and determining characters in the text that fall into a target extension area as special characters; the target extension area includes Chinese, Japanese, and Korean CJK Extension A area and CJK Extension B area; CJK Extension A area includes characters with Unicode encoding within U+3400 to U+4DBF, and CJK Extension B area includes characters with Unicode encoding within U+20000 to U+2A6DD.
[0008] In some embodiments, generating a transfer matrix based on special characters includes: encoding the special characters by UTF-8 to obtain a special character code; a special character code includes a terminal and at least one code body; counting a combination list in the special character code in order of appearance, and generating a transfer matrix based on the combination list; the combination list includes at least one code combination, and a code combination consists of a code header and at least one code tail, the code header is the terminal in the special character code or the first code body of the three-byte code, and the code tail is the code body in the special character code.
[0009] In some embodiments, generating a transfer matrix according to special characters includes: encoding the special characters in UTF-8 to obtain special character codes, and directly composing the special character codes into a transfer matrix.
[0010] In some embodiments, the garbled code body is adjusted to a preset format according to the garbled word unit and the transfer matrix to obtain a target output sequence, including: matching the target code body in the transfer matrix according to the garbled word unit, and using the target code body to replace the garbled code body to obtain the target output sequence; the encoding of the target code body is in a preset format.
[0011] In some embodiments, matching the target code body in the transition matrix according to the garbled word unit includes: matching the target code body from the transition matrix according to the position of the garbled code body in the garbled word unit.
[0012] In some embodiments, the target code body is matched from the transition matrix according to the position of the garbled code body in the garbled word unit, including: when the garbled code body is a two-byte code body, the target code body is matched in the transition matrix according to the terminal of the garbled word unit; when the garbled code body is a three-byte code body, the target code body is matched in the transition matrix according to the terminal of the garbled word unit and the remaining code body; when there are multiple matched code bodies, any one of the multiple matched code bodies is determined as the target code body, or, the code body that appears the most times in the historical text among the multiple code bodies is determined as the target code body.
[0013] In a second aspect, the present application provides a processing device for garbled code output by a language model, comprising: an acquisition module, used to acquire the original output sequence of a target language model; the original output sequence is composed of at least one word unit, a word unit is composed of a terminal, or a word unit is composed of a terminal and at least one code body; a determination module, used to, for each word unit, when the word unit includes a code body and the encoding format of any code body of the word unit is not a preset format, determine the code body as a garbled code body and the word unit as a garbled word unit; wherein, in the preset format, the target position of the code body encoding is the target character string; the target position is the position of the first two characters of the first byte of the code body; a processing module, used to adjust the garbled code body to a preset format according to the garbled word unit and the transfer matrix to obtain a target output sequence; the transfer matrix is generated according to the historical text input to the target language model; a generation module, used to decode the target output sequence to generate the output result of the target language model.
[0014] In some embodiments, the acquisition module is also used to acquire historical text input into the target language model within a historical time period before matching the target code body in the transfer matrix based on the garbled word element; the determination module is also used to determine the special characters in the historical text and generate a transfer matrix based on the special characters; the encoding length of the special character is greater than the preset length, or the encoding result of the special character is within the preset encoding range.
[0015] In some embodiments, the determination module is specifically used to: decode the historical text into UTF-8 encoded text, and determine the characters represented by two bytes and three bytes in the text as special characters; or decode the historical text into Unicode character text, and determine the characters in the text that fall into the target extension area as special characters; the target extension area includes Chinese, Japanese, and Korean CJK extension A area and CJK extension B area; CJK extension A area includes characters with Unicode encoding within U+3400 to U+4DBF, and CJK extension B area includes characters with Unicode encoding within U+20000 to U+2A6DD.
[0016] In some embodiments, the determination module is specifically used to: perform UTF-8 encoding on special characters to obtain special character codes; a special character code includes a terminal and at least one code body; a combination list in the special character code is counted in order of appearance, and a transfer matrix is generated based on the combination list; the combination list includes at least one coding combination, a coding combination consists of a coding header and at least one coding tail, the coding header is the terminal in the special character code or the first code body of the three-byte code, and the coding tail is the code body in the special character code.
[0017] In some embodiments, the determination module is specifically used to: perform utf-8 encoding on special characters to obtain special character codes, and directly form a transfer matrix with the special character codes.
[0018] In some embodiments, the processing module is specifically used to match the target code body in the transfer matrix according to the garbled code word, and use the target code body to replace the garbled code body to obtain a target output sequence; the encoding of the target code body is a preset format.
[0019] In some embodiments, the processing module is specifically configured to match the target code body from the transition matrix according to the position of the garbled code body in the garbled word unit.
[0020] In some embodiments, the processing module is specifically used to: when the garbled code body is a two-byte code body, match the target code body in the transfer matrix according to the terminal of the garbled word unit; when the garbled code body is a three-byte code body, match the target code body in the transfer matrix according to the terminal of the garbled word unit and the remaining code body; when there are multiple matched code bodies, determine any one of the multiple matched code bodies as the target code body, or determine the code body that appears the most times in the historical text among the multiple code bodies as the target code body.
[0021] In a third aspect, the present application provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, a method for processing garbled output of a language model as in any embodiment of the first aspect is implemented.
[0022] In a fourth aspect, the present application provides a computer-readable storage medium, including: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, a method for processing garbled output of a language model as in any embodiment of the first aspect is implemented.
[0023] In a fifth aspect, the present application provides a computer program product, comprising: when the computer program product runs on a computer, the computer implements a method for processing garbled code output by a language model as in any embodiment of the first aspect.
[0024] The technical solution provided by the present application has the following advantages compared with the prior art: First, the original output sequence of the target language model is obtained. Wherein, the original output sequence is composed of at least one word unit, a word unit is composed of a terminal, or a word unit is composed of a terminal and at least one code body. Afterwards, for each word unit, when the word unit includes a code body and the encoding format of any code body of the word unit is not a preset format, the code body is determined to be a garbled code body, and the word unit is determined to be a garbled word unit. Wherein, the target position of the code body encoding in the preset format is the target string; the target position is the position of the first two characters of the first byte of the code body. Afterwards, the garbled code body is adjusted to the preset format according to the garbled word unit and the transfer matrix to obtain the target output sequence. Wherein, the transfer matrix is generated according to the historical text input to the target language model. Finally, the target output sequence is decoded to generate the output result of the target language model. In this way, on the one hand, the output of the target language model can be detected, the generation of garbled code can be avoided on the output side of the model, and the garbled code problem can be solved by expanding the corpus, which greatly saves costs and improves the processing effect of garbled code output by the language model. On the other hand, when garbled characters are found in the output results, the garbled characters can be replaced with historical text content, thereby ensuring the integrity of the output results as much as possible and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0027] Figure 1 One of the flowcharts of the method for processing garbled code output by the language model provided in the embodiment of the present application;
[0028] Figure 2 A second flowchart of a method for processing garbled code output by a language model provided in an embodiment of the present application;
[0029] Figure 3 A schematic diagram of the structure of a device for processing garbled code output by a language model provided in an embodiment of the present application;
[0030] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0031] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0032] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.
[0033] As the accuracy of large language models (also called models) increases, their application scenarios in Chinese tasks are increasing. For example, reading and analyzing documents (such as reading and analyzing massive documents and reports), creating and editing content (such as creating novels and scripts). However, when large language models output Chinese results, garbled characters may appear sometimes, such as special symbols such as "?" and "*" are output after decoding. Such problems not only affect the accuracy of the output results of large language models, but also seriously affect the user experience.
[0034] At present, the way to deal with garbled output of language models is usually to increase the training corpus when training the language model to improve the language model's generation ability, thereby reducing the possibility of generating garbled output. However, the reasons for the garbled output of language models are usually varied, and can generally be divided into encoding problems, model tokenizer problems, and model training problems. Encoding problems include model training encoding problems, program code encoding problems, and display encoding problems. Model Tokenizer problems include Tokenizer segmentation errors and incomplete vocabulary coverage. Model training problems include insufficient training corpus.
[0035] Among them, the training encoding problem refers to the fact that the data set used in model training may use different character encoding methods, such as 8-bit Unicode Transformation Format (UTF-8), Chinese Internal Code Specification (GBK), etc. If the encoding used by the model when generating text is inconsistent with the training data (or exceeds the normal encoding distribution of the training data), garbled characters will appear. The program code encoding problem refers to the fact that the program code that calls the large language model may also use different character encodings. If it is inconsistent with the model output, it will also cause garbled characters. The display encoding problem means that the terminal or web page that finally displays the text also needs to set the correct character encoding, otherwise even if the model output is correct, it will display garbled characters due to decoding errors. Tokenizer cutting errors refer to the fact that some large language models use character-based Tokenizers, which may incorrectly cut Chinese characters into multiple tokens, resulting in incomplete Chinese encoding information after decoding, which will cause garbled characters. Incomplete vocabulary coverage refers to the fact that if the vocabulary of the model cannot completely cover all Chinese characters, garbled characters may be output when encountering words that have not been encountered. Insufficient training corpus means that if the model training data contains insufficient Chinese corpus, the model's ability to understand and generate Chinese will be relatively weak, and it will easily output incoherent text or text containing garbled characters.
[0036] Therefore, the above method of expanding the corpus will definitely increase the training cost, and the effect of dealing with the garbled output problem of the language model is limited.
[0037] In response to the above problems, an embodiment of the present application provides a method for processing garbled output of a language model, including first, obtaining the original output sequence of the target language model. Wherein, the original output sequence is composed of at least one word unit, a word unit is composed of a terminal, or a word unit is composed of a terminal and at least one code body. Afterwards, for each word unit, when the word unit includes a code body and the encoding format of any code body of the word unit is not a preset format, the code body is determined to be a garbled code body, and the word unit is determined to be a garbled word unit. Wherein, the target position of the code body encoding in the preset format is the target string; the target position is the position of the first two characters of the first byte of the code body. Afterwards, the garbled code body is adjusted to the preset format according to the garbled word unit and the transfer matrix to obtain the target output sequence. Wherein, the transfer matrix is generated according to the historical text input to the target language model. Finally, the target output sequence is decoded to generate the output result of the target language model. In this way, on the one hand, the output of the target language model can be detected, and the generation of garbled characters can be avoided on the output side of the model, avoiding the need to solve the garbled character problem by expanding the corpus, greatly saving costs, and improving the processing effect of garbled characters output by the language model. On the other hand, when garbled characters are found in the output results, the garbled characters can be replaced with historical text content, thus ensuring the integrity of the output results as much as possible, thereby improving the user experience.
[0038] The method for processing garbled output of a language model provided in the embodiment of the present application is mainly applicable to the processing scenario of garbled output of a language model based on utf-8 encoding. The method for processing garbled output of a language model provided in the embodiment of the present application can be executed by a processing device for garbled output of a language model, which can be implemented by software and / or hardware, and the device can be integrated in an electronic device with a function of processing garbled output of a language model, such as a server, a television, a mobile phone, a PDA, a tablet computer, a laptop computer or a desktop computer.
[0039] Figure 1 A flow chart of a method for processing garbled output of a language model provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method for processing garbled characters output by the language model may include the following steps:
[0040] S10. Obtain the original output sequence of the target language model.
[0041] The target language model is a language model that can process Chinese data, such as BERT, ERNIE, GPT, etc. In some embodiments, the target language model is a large language model based on utf-8 encoding. The original output sequence consists of at least one word unit, a word unit consists of a terminal, or a word unit consists of a terminal and at least one code body.
[0042] To ensure the compatibility of the language model, when processing various types of data, the language model usually first converts the data into a more general Unicode code that can be recognized by the device (for example, when the input data is in Chinese, it first goes through the processes of Chinese, utf-8 transcoding, and Unicode transcoding), and then generates a processing result based on the converted data (represented as a string of Unicode codes). Then, the processing result is transcoded into utf-8 encoding to obtain a string of utf-8 encoding sequences. This utf-8 encoding sequence is the original output sequence composed of at least one token referred to in the embodiments of the present application, and the tokens in the original output sequence are all represented as utf-8 encodings. Then, the language model decodes the utf-8 encoding into Chinese to finally obtain the Chinese output result of the language model. Therefore, the way to obtain the original output sequence of the target language model can be to directly obtain it when the target language model converts the generated processing result into utf-8 encoding without performing Chinese decoding first.
[0043] S11. For each token, when the token includes a code body and the encoding format of any code body of the token is not the preset format, determine the code body as a garbled code body and the token as a garbled token.
[0044] First, determine whether the token includes a code body.
[0045] Specifically, the tokenizer of the language model uses the SentencePiece method for word segmentation. First, the sentence is treated as a sequence of Unicode codes for processing, and then the final processing result is transcoded into the currently most common utf-8 standard encoding. Since utf-8 is a variable-length code consisting of 1 - 4 bytes, a small number of Chinese characters are split into multiple Unicode codes in units of bytes and form a token with multiple tokens. For example, for the character "彧", the token identifier (Identifier, id) after tokenization is composed of [534, 494, 472] to form a token. This results in the need for 3 ordered tokens to correctly decode when generating such Chinese characters, which is relatively difficult. If the generated Unicode code does not conform to the utf-8 encoding standard, it will lead to the appearance of garbled problems. Therefore, when solving the garbled problem, post-processing needs to be performed during the inference process of the target language model to limit the tokenid sequence so that it can be decoded into normal Chinese and ensure that the continuously generated utf-8 tokens can be normally decoded. Find the token sequence of the multi-byte encoding and modify it so that the large model can only generate utf-8 code bodies, thereby ensuring the encoding legality of the continuous utf-8 tokens. Here, the token is the so-called dock or code body in this article and is also the basic unit that makes up the token.
[0046] Specifically, when converting Unicode code to UTF-8, the following rules (1)-(4) must be met:
[0047] (1) Single-byte code (0xxx ~): The Unicode range corresponding to the single-byte character in UTF-8 encoding is 0x00~0x7F. Among them, the highest bit in the single-byte code is 0.
[0048] (2) Double-byte code terminal (110x�~�): The identifier of the double-byte character in the UTF-8 encoding. The double-byte encoding is applicable to characters in the Unicode range from 0x80 to 0x7FF. In the UTF-8 encoding, the first byte starts with 110 and the second byte starts with 10.
[0049] (3) Three-byte code terminal (1110 � ~ <0xEF): The identifier of the three-byte character in the UTF-8 encoding. The three-byte encoding is applicable to characters in the Unicode range from 0x800 to 0xFFFF. The first byte of the UTF-8 encoding starts with 1110, and the next two bytes start with 10.
[0050] (4) Code body (10xx�~�): The code body part in UTF-8 encoding, indicating the structure of subsequent bytes in double-byte, three-byte or four-byte encoding. All code body bytes start with 10, indicating that they are "continuation bytes" that make up multi-byte characters.
[0051] As can be seen from the above, UTF-8 encoding is divided into single-byte code and multi-byte code (two-byte, three-byte), and most of the rare Chinese characters with problems correspond to three-byte codes. Multi-byte code is composed of a terminal and n (n = 1, 2) code bodies. You can know whether the encoding is a two-byte code or a three-byte code just by the terminal. The first byte of the two-byte terminal starts with 110, and the code body starts with 10. The first byte of the three-byte terminal starts with 1110, and the code body starts with 10. Therefore, the way to determine whether a word element includes a code body can be to determine that the word element includes a code body when the first byte of the terminal of the word element starts with 110 or 1110.
[0052] Secondly, when the target position of any code body of a word unit is not the target string, the code body is determined as a garbled code body, and the word unit is determined as a garbled word unit. Among them, the target position of the code body encoding in the preset format is the target string; the target position is the position of the first two characters of the first byte of the code body, and the target string is 10.
[0053] Specifically, when the first two characters of the first byte of any code body of a word unit are not 10, the code body is determined to be a garbled code body, and the word unit where the code body is located is a garbled word unit.
[0054] In some embodiments, when a word does not have a code body, or when a word has a code body but the target position of each code body is the target string, it is determined that there is no garbled word, and the target output sequence is directly decoded to generate the output result of the target language model.
[0055] S12, adjusting the garbled code body to a preset format according to the garbled code word unit and the transfer matrix to obtain a target output sequence.
[0056] The transfer matrix is generated based on the historical text input into the target language model. The historical text is the text input into the target model within a preset time period, or all the historical text input by the user in the current session of the target language model, or the text input within a preset time period in the current session of the target language model. The transfer matrix includes the utf-8 encoding set of special characters in the historical text, or the special characters in the historical text.
[0057] Specifically, the method of adjusting the garbled code body to a preset format according to the garbled word unit and the transfer matrix can be to match the target code body in the transfer matrix according to the garbled word unit, and use the target code body to replace the garbled code body to obtain a target output sequence; wherein the encoding of the target code body is in a preset format.
[0058] First, match the target code body in the transfer matrix according to the garbled code word.
[0059] Specifically, the target code body is matched from the transfer matrix according to the way the garbled word unit is matched in the transfer matrix according to the position of the garbled code body in the garbled word unit. When the garbled code body is a two-byte code body, the target code body is matched in the transfer matrix according to the terminal of the garbled word unit; when the garbled code body is a three-byte code body, the target code body is matched in the transfer matrix according to the terminal of the garbled word unit and the remaining code body; when there are multiple matched code bodies, any one of the multiple matched code bodies is determined as the target code body, or the code body that appears the most times in the historical text among the multiple code bodies is determined as the target code body.
[0060] For example, in the following garbled word element: 111000001110000010000001, it can be seen from the first 4 bits of its terminal that the word element is a three-byte code, and its second byte should be the first code body of the three bytes, but its target position is 11, not 10 as the code body should have. Therefore, the code body (11100000) is determined as the garbled code body, and the word element (111000001110000010000001) is determined as the garbled word element, and the corresponding UTF-8 encoded hexadecimal representation is 0xE00xE00x81. At this time, the three-byte encoded word element with the terminal 0xE0 and the second code body 0x81 is matched in the transfer matrix, and the matched word element is determined as the target word element, and the first code body of the target word element is determined as the target code body.
[0061] For another example, in the garbled word element as follows: 1110000011100000, it can be seen from the first 4 bits of its terminal that the word element is a two-byte code, and its second byte should be the two-byte code body, but its target position is 11, not the 10 that the code body should have. Therefore, the code body (11100000) is determined as the garbled code body, and the word element (1110000011100000) is determined as the garbled word element, and the corresponding UTF-8 encoding hexadecimal representation is 0xE00xE0. At this time, the two-byte coded word element with a terminal of 0xE0 is matched in the transfer matrix, and the matched word element is determined as the target word element, and the code body of the target word element is determined as the target code body. When multiple code bodies are matched, any one of the matched multiple code bodies is determined as the target code body, or the code body that appears the most times in the historical text among the multiple code bodies is determined as the target code body.
[0062] Secondly, the target code body is used to replace the garbled code body to obtain the target output sequence.
[0063] Specifically, all the garbled words in the original output sequence are matched with corresponding target code bodies, and the target code bodies are used to replace the corresponding garbled code bodies in the original output sequence to obtain the final target output sequence.
[0064] S13. Decode the target output sequence to generate an output result of the target language model.
[0065] Specifically, the target output sequence is a UTF-8 encoded sequence, and the UTF-8 encoded sequence is decoded to obtain the output result of the target language model. The decoding method of the UTF-8 encoding can be a currently commonly used decoding method, and this application does not limit this.
[0066] In the above scheme, first, the original output sequence of the target language model is obtained. Wherein, the original output sequence is composed of at least one word unit, a word unit is composed of a terminal, or a word unit is composed of a terminal and at least one code body. Afterwards, for each word unit, when the word unit includes a code body and the encoding format of any code body of the word unit is not a preset format, the code body is determined to be a garbled code body, and the word unit is determined to be a garbled word unit. Wherein, the target position of the code body encoding in the preset format is the target string; the target position is the position of the first two characters of the first byte of the code body. Afterwards, the garbled code body is adjusted to the preset format according to the garbled word unit and the transfer matrix to obtain the target output sequence. Wherein, the transfer matrix is generated according to the historical text input to the target language model. Finally, the target output sequence is decoded to generate the output result of the target language model. In this way, on the one hand, the output of the target language model can be detected, the generation of garbled code can be avoided on the output side of the model, and the garbled code problem can be solved by expanding the corpus, which greatly saves costs and improves the processing effect of garbled code output by the language model. On the other hand, when garbled characters are found in the output results, the garbled characters can be replaced with historical text content, thereby ensuring the integrity of the output results as much as possible and improving the user experience.
[0067] In some embodiments, Figure 2 As shown, before matching the target code body in the transfer matrix according to the garbled code word unit, the method for processing the garbled code output by the language model also includes the following steps:
[0068] S1. Obtain historical texts input into the target language model within a historical time period.
[0069] Among them, the historical time period is preset, for example, a default value, or is set by relevant personnel according to actual conditions.
[0070] Specifically, the most common garbled characters in Chinese text are often the names of people or objects. Because they are rarely used in daily life, the model is not exposed to them much during the training process. Based on this, it is possible to consider strengthening restrictions during the model reasoning process, and outputting as many characters that have appeared in the input text as possible on the basis of ensuring the output of legal continuous utf-8 tokens, thereby alleviating the name change problem. Therefore, the embodiment of the present application establishes a transfer matrix based on historical text.
[0071] S2. Determine the special characters in the historical text and generate a transfer matrix based on the special characters.
[0072] First, determine the special characters in the historical text. Among them, the encoding length of the special characters is greater than the preset length, or the encoding result of the special characters is within the preset encoding range. The preset length and the preset encoding range are both preset values in advance. For example, the preset length can be a one-byte length or a two-byte length; the preset encoding range can be the range corresponding to the Chinese, Japanese, and Korean (CJK) extended area A and the CJK extended area B; CJK extended area A and CJK extended area B are areas where rare characters are included in Unicode. CJK extended area A includes characters with Unicode encodings of U+3400 to U+4DBF, and CJK extended area B includes characters with Unicode encodings of U+20000 to U+2A6DD.
[0073] Specifically, the method of determining the special characters in the historical text may be to decode the historical text into a text encoded in UTF-8, and determine the characters represented by two bytes and three bytes in the text as special characters. Alternatively, the historical text may be decoded into a text of Unicode characters, and the characters in the text falling into the target extension area are determined as special characters. The target extension area includes CJK extension area A and CJK extension area B.
[0074] Secondly, a transition matrix is generated based on the special characters.
[0075] Specifically, the transfer matrix may be generated according to the special characters by performing UTF-8 encoding on the special characters to obtain the special character encoding, and the special character encoding is directly used to form the transfer matrix.
[0076] In some embodiments, according to the method of generating a transfer matrix based on special characters, the special characters may be first encoded by UTF-8 to obtain a special character code. A special character code includes a terminal and at least one code body. Then, a combination list in the special character code is counted in order of appearance, and a transfer matrix is generated based on the combination list. The combination list includes at least one coding combination, and a coding combination consists of a coding header and at least one coding tail. The coding header is the terminal in the special character code or the first code body of the three-byte code, and the coding tail is the code body in the special character code.
[0077] For example, the UTF-8 encoding of the historical text is as follows: xxxxxxxxxx 0xE0 0x80 0x81 xxxxx, xxxxxxxxxxx 0xE1 0x83 0x84, xxxxxxx 0xE0 0x82 0x84 xxxx, where x represents a non-special character in the historical text, and the special character encodings in the historical text include: 0xE0 0x80 0x81, 0xE1 0x83 0x84, and 0xE0 0x82 0x84. At this time, after statistics, the combination list in the historical text includes: (1) 0xE0: 0x80, 0x82; (2) 0x80: 0x81; (3) 0xE1: 0x83; (4) 0x83: 0x84; (5) 0x82: 0x84. Therefore, the generated transfer matrix is as follows: {�:[�,�],�:[�],�:[�],�:[�],�:[�]}.
[0078] In some embodiments, when the transfer matrix is a transfer matrix generated according to a statistical method, the target code body can be matched from the transfer matrix according to the position of the garbled code body in the garbled code word.
[0079] For example, the garbled word element is as follows: 111000001110000010000001. It can be seen from the first 4 bits of its terminal that the word element is a three-byte code, and its second byte should be the first code body of the three bytes, but its target position is 11, not 10 as the code body should have. Therefore, the code body (11100000) is determined as the garbled code body, and the word element (111000001110000010000001) is determined as the garbled word element. The corresponding UTF-8 encoding hexadecimal representation is 0xE00xE00x81. At this time, 0xE0 can be followed by 0x80 and 0x82 in the transfer matrix, and 0x80 can be followed before 0x81. They have a common element 0x80, so the target code body is determined to be 0x80.
[0080] Similar to step S12, if multiple code bodies are matched in the transfer matrix, any one of the matched code bodies is determined as the target code body, or the code body that appears most times in the historical text is determined as the target code body.
[0081] In the above scheme, a transfer matrix is generated according to the historical text input into the target language model, so that the garbled words can be repaired according to the content in the transfer matrix. The garbled words can be corrected more accurately, which improves the accuracy and consistency of the text generated by the target language model and improves the reliability of text repair.
[0082] The embodiment of the present application can divide the functional modules of the processing device for outputting garbled characters of the language model according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing unit. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.
[0083] like Figure 3 As shown, it is a structural diagram of a device for processing garbled code output by a language model provided in an embodiment of the present application. The device for processing garbled code output by a language model includes an acquisition module 51, a determination module 52, a processing module 53, and a generation module 54.
[0084] The acquisition module 51 is used to obtain the original output sequence of the target language model; the original output sequence is composed of at least one word unit, a word unit is composed of a terminal, or a word unit is composed of a terminal and at least one code body; the determination module 52 is used for, for each word unit, when the word unit includes a code body and the encoding format of any code body of the word unit is not a preset format, determining the code body as a garbled code body and the word unit as a garbled word unit; wherein the target position of the code body encoding in the preset format is the target string; the target position is the position of the first two characters of the first byte of the code body; the processing module 53 is used to adjust the garbled code body to a preset format according to the garbled word unit and the transfer matrix to obtain a target output sequence; the transfer matrix is generated according to the historical text input to the target language model; the generation module 54 is used to decode the target output sequence to generate the output result of the target language model.
[0085] In the above scheme, first, the original output sequence of the target language model is obtained. Wherein, the original output sequence is composed of at least one word unit, a word unit is composed of a terminal, or a word unit is composed of a terminal and at least one code body. Afterwards, for each word unit, when the word unit includes a code body and the encoding format of any code body of the word unit is not a preset format, the code body is determined to be a garbled code body, and the word unit is determined to be a garbled word unit. Wherein, the target position of the code body encoding in the preset format is the target string; the target position is the position of the first two characters of the first byte of the code body. Afterwards, the garbled code body is adjusted to the preset format according to the garbled word unit and the transfer matrix to obtain the target output sequence. Wherein, the transfer matrix is generated according to the historical text input to the target language model. Finally, the target output sequence is decoded to generate the output result of the target language model. In this way, on the one hand, the output of the target language model can be detected, the generation of garbled code can be avoided on the output side of the model, and the garbled code problem can be solved by expanding the corpus, which greatly saves costs and improves the processing effect of garbled code output by the language model. On the other hand, when garbled characters are found in the output results, the garbled characters can be replaced with historical text content, thereby ensuring the integrity of the output results as much as possible and improving the user experience.
[0086] In some embodiments, the acquisition module 51 is also used to obtain historical text input into the target language model within a historical time period before matching the target code body in the transfer matrix according to the garbled word element; the determination module 52 is also used to determine the special characters in the historical text and generate a transfer matrix based on the special characters; the encoding length of the special character is greater than the preset length, or the encoding result of the special character is within the preset encoding range.
[0087] In some embodiments, the determination module 52 is specifically used to: decode the historical text into UTF-8 encoded text, and determine the characters represented by two bytes and three bytes in the text as special characters; or decode the historical text into Unicode character text, and determine the characters in the text that fall into the target extension area as special characters; the target extension area includes Chinese, Japanese, and Korean CJK extension area A and CJK extension area B; CJK extension area A includes characters with Unicode encoding within U+3400 to U+4DBF, and CJK extension area B includes characters with Unicode encoding within U+20000 to U+2A6DD.
[0088] In some embodiments, the determination module 52 is specifically used to: perform UTF-8 encoding on the special character to obtain a special character code; a special character code includes a terminal and at least one code body; a combination list in the special character code is counted in order of appearance, and a transfer matrix is generated according to the combination list; the combination list includes at least one coding combination, and a coding combination consists of a coding header and at least one coding tail, the coding header is the terminal in the special character code or the first code body of the three-byte code, and the coding tail is the code body in the special character code.
[0089] In some embodiments, the determination module 52 is specifically used to: perform utf-8 encoding on the special characters to obtain the special character codes, and directly form the transfer matrix with the special character codes.
[0090] In some embodiments, the processing module 53 is specifically used to match the target code body in the transfer matrix according to the garbled code word, and use the target code body to replace the garbled code body to obtain a target output sequence; the encoding of the target code body is a preset format.
[0091] In some embodiments, the processing module 53 is specifically configured to match the target code body from the transition matrix according to the position of the garbled code body in the garbled word unit.
[0092] In some embodiments, the processing module 53 is specifically used to: when the garbled code body is a two-byte code body, match the target code body in the transfer matrix according to the terminal of the garbled word unit; when the garbled code body is a three-byte code body, match the target code body in the transfer matrix according to the terminal of the garbled word unit and the remaining code body; when there are multiple matched code bodies, determine any one of the multiple matched code bodies as the target code body, or determine the code body that appears the most times in the historical text among the multiple code bodies as the target code body.
[0093] The device for processing garbled output of a language model provided in this embodiment can execute the method for processing garbled output of a language model provided in the above method embodiment. Its implementation principle and technical effect are similar to those of the above method and will not be repeated here.
[0094] Figure 4 An electronic device according to an exemplary embodiment may include a processor 902, and the processor 902 is used to execute application code, thereby implementing the method for processing garbled code output by a language model in the present application.
[0095] The processor 902 may be a central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.
[0096] like Figure 4 As shown, the electronic device may further include a memory 903. The memory 903 is used to store application program codes for executing the solution of the present application, and the execution is controlled by the processor 902.
[0097] The memory 903 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 903 may exist independently and be connected to the processor 902 via the bus 904. The memory 903 may also be integrated with the processor 902.
[0098] like Figure 4 As shown, the electronic device may further include a communication interface 901, wherein the communication interface 901, the processor 902, and the memory 903 may be coupled to each other, for example, via a bus 904. The communication interface 901 is used to exchange information with other devices, for example, to support information exchange between the electronic device and other devices.
[0099] It should be pointed out that Figure 4 The device structure shown in the figure does not constitute a limitation on the electronic device, except Figure 4 In addition to the components shown, the electronic device may include more or fewer components than shown, or combine certain components, or arrange the components differently. The electronic device provided in this embodiment can execute the method for processing garbled output of the language model provided in the above method embodiment, and its implementation principle and technical effect are similar to the above method, which will not be repeated here.
[0100] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each process of the method for processing garbled code output by a language model in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0101] The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0102] An embodiment of the present application provides a computer program product, which stores a computer program. When the computer program is executed by a processor, each process of the method for processing garbled code output by the language model in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0103] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.
[0104] In this application, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0105] In this application, computer-readable media includes permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data and carrier waves.
[0106] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0107] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for processing garbled code output by a language model, characterized in that: include: Get the original output sequence of the target language model; The original output sequence consists of at least one word unit, a word unit consists of a terminal, or a word unit consists of a terminal and at least one code body; For each word unit, when the word unit includes a code body and the encoding format of any code body of the word unit is not a preset format, the code body is determined to be a garbled code body, and the word unit is determined to be a garbled word unit; wherein the target position of the code body encoding in the preset format is the target character string; the target position is the position of the first two characters of the first byte of the code body; The garbled code body is adjusted to the preset format according to the garbled code word unit and the transfer matrix to obtain a target output sequence; the transfer matrix is generated according to the historical text input into the target language model; The target output sequence is decoded to generate an output result of the target language model.
2. The processing method according to claim 1, characterized in that: Before matching the target code body in the transfer matrix according to the garbled code word, the method further includes: Acquire historical text input into the target language model within a historical time period; Determine the special characters in the historical text, and generate a transfer matrix according to the special characters; the encoding length of the special characters is greater than a preset length, or the encoding result of the special characters is within a preset encoding range.
3. The processing method according to claim 2, characterized in that: The determining of the special characters in the historical text includes: Decoding the historical text into a utf-8 encoded text, and determining characters represented by two bytes and three bytes in the text as special characters; or, The historical text is decoded into a text of Unicode characters, and characters in the text that fall into a target extension area are determined as special characters; the target extension area includes a CJK extension A area and a CJK extension B area for Chinese, Japanese, and Korean; the CJK extension A area includes characters with Unicode encodings ranging from U+3400 to U+4DBF, and the CJK extension B area includes characters with Unicode encodings ranging from U+20000 to U+2A6DD.
4. The processing method according to claim 2, characterized in that: The step of generating a transfer matrix according to the special character comprises: The special character is encoded by UTF-8 to obtain a special character code; a special character code includes a terminal and at least one code body; A combination list in the special character code is counted in order of appearance, and a transfer matrix is generated according to the combination list; the combination list includes at least one code combination, and a code combination consists of a code header and at least one code tail, the code header is the terminal in the special character code or the first code body of the three-byte code, and the code tail is the code body in the special character code.
5. The processing method according to claim 2, characterized in that: The step of generating a transfer matrix according to the special character comprises: The special characters are encoded by UTF-8 to obtain special character codes, and the special character codes are directly used to form a transfer matrix.
6. The processing method according to claim 4 or 5, characterized in that: The step of adjusting the garbled code body to the preset format according to the garbled code word element and the transfer matrix to obtain a target output sequence includes: A target code body is matched in a transfer matrix according to the garbled code word, and the garbled code body is replaced with the target code body to obtain a target output sequence; the encoding of the target code body is the preset format.
7. The processing method according to claim 6, characterized in that: The matching of the target code body in the transfer matrix according to the garbled code word element includes: According to the position of the garbled code body in the garbled word unit, a target code body is matched from the transition matrix.
8. The processing method according to claim 7, characterized in that: According to the position of the garbled code body in the garbled word unit, matching the target code body from the transfer matrix includes: When the garbled code body is a two-byte code body, matching a target code body in the transfer matrix according to the terminal of the garbled word unit; When the garbled code body is a three-byte code body, matching a target code body in the transfer matrix according to the terminal and the remaining code body of the garbled word element; When there are multiple matched code bodies, any one of the multiple matched code bodies is determined as the target code body, or the code body that appears most times in the historical text among the multiple code bodies is determined as the target code body.
9. A device for processing garbled code output by a language model, characterized in that: include: An acquisition module, used to obtain the original output sequence of the target language model; The original output sequence consists of at least one word unit, a word unit consists of a terminal, or a word unit consists of a terminal and at least one code body; A determination module is used for, for each word unit, when the word unit includes a code body and the encoding format of any code body of the word unit is not a preset format, determining the code body as a garbled code body and the word unit as a garbled word unit; wherein the target position of the code body encoding in the preset format is a target string; and the target position is the position of the first two characters of the first byte of the code body; A processing module, configured to adjust the garbled code body to the preset format according to the garbled code word unit and the transfer matrix to obtain a target output sequence; the transfer matrix is generated according to the historical text input into the target language model; A generation module is used to decode the target output sequence to generate an output result of the target language model.
10. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements a method for processing garbled output of a language model as described in any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for processing garbled code output by a language model according to any one of claims 1 to 8 is implemented.
12. A computer program product, characterized in that When the computer program product runs on a computer, the computer is enabled to implement the method for processing garbled code output by a language model according to any one of claims 1 to 8.