Model training method, text processing method, device and electronic device

Abstract

The present application discloses a model training method, a text processing method, an apparatus, and an electronic device, belonging to the field of computer technology. The model training method includes: obtaining a training text, where the training text includes punctuation characters that are misused; generating a target sequence according to the training text, where the target sequence is a sequence obtained after correcting the punctuation characters that are misused in the training text; generating a first initial mask matrix based on the training text; hiding a first target sub-text in the first initial mask matrix to obtain a first target mask matrix, where the training text includes the first target sub-text; training an initial model according to the training text, the target sequence, and the first target mask matrix to obtain a target model, where the target model is used to correct punctuation characters that are misused in the text input to the target model.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and particularly relates to a model training method, a text processing method, a device, and an electronic device. Background Art

[0002] Currently, the method for correcting punctuation marks in a text is to first remove the punctuation marks in the text, and then use some model methods such as machine learning and deep learning to predict the positions and types of punctuation marks that appear in the text, and then compare them with the punctuation marks previously marked manually to confirm the punctuation marks with differences between the two. Finally, manual verification is performed.

[0003] It can be seen that the current symbol correction method has a cumbersome process, a high degree of manual participation, and low correction efficiency. Summary of the Invention

[0004] The purpose of the embodiments of this application is to provide a model training method, a text processing method, a device, and an electronic device, which can solve the problem of low correction efficiency when the current technology corrects the symbols in a text.

[0005] In a first aspect, the embodiments of this application provide a model training method, which includes:

[0006] Obtain a training text, where the training text includes punctuation characters that are misused;

[0007] Generate a target sequence according to the training text, where the target sequence is a sequence obtained after correcting the punctuation characters that are misused in the training text;

[0008] Generate a first initial mask matrix based on the training text;

[0009] Hide a first target sub-text in the first initial mask matrix to obtain a first target mask matrix, where the training text includes the first target sub-text;

[0010] Train an initial model according to the training text, the target sequence, and the first target mask matrix to obtain a target model, where the target model is used to correct the punctuation characters that are misused in the text input to the target model.

[0011] In a second aspect, the embodiments of this application provide a text processing method, which includes:

[0012] Obtain a text to be processed, where the text to be processed includes punctuation characters;

[0013] Generate a second initial mask matrix according to the text to be processed;

[0014] Hide the third target sub - text in the second initial mask matrix to obtain a second target mask matrix, where the text to be processed includes the third target sub - text;

[0015] According to the text to be processed and the second target mask matrix, use the target model to correct the punctuation characters misused in the text to be processed, and obtain the target text.

[0016] In a third aspect, an embodiment of the present application provides a model training device, including:

[0017] A first acquisition module, configured to acquire training text, where the training text includes misused punctuation characters;

[0018] A first generation module, configured to generate a target sequence according to the training text, where the target sequence is a sequence obtained after correcting the misused punctuation characters in the training text;

[0019] A second generation module, configured to generate a first initial mask matrix based on the training text;

[0020] A second acquisition module, configured to hide the first target sub - text in the first initial mask matrix to obtain a first target mask matrix, where the training text includes the first target sub - text;

[0021] A third acquisition module, configured to train an initial model according to the training text, the target sequence, and the first target mask matrix to obtain a target model, where the target model is used to correct the misused punctuation characters in the text input to the target model.

[0022] In a fourth aspect, an embodiment of the present application provides a text processing device, where the device includes:

[0023] A first acquisition module, configured to acquire text to be processed, where the text to be processed includes punctuation characters;

[0024] A generation module, configured to generate a second initial mask matrix according to the text to be processed;

[0025] A second acquisition module, configured to hide the third target sub - text in the second initial mask matrix to obtain a second target mask matrix, where the text to be processed includes the third target sub - text;

[0026] A third acquisition module, configured to correct the misused punctuation characters in the text to be processed according to the text to be processed and the second target mask matrix by using the target model, and obtain the target text.

[0027] Fifth aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0028] Sixth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0029] Seventh aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect or the second aspect.

[0030] In an embodiment of the present application, a training text is obtained, and the training text includes punctuation characters that are misused; according to the training text, a target sequence is generated, and the target sequence is a sequence obtained after correcting the punctuation characters that are misused in the training text; based on the training text, a first initial mask matrix is generated; a first target sub-text in the first initial mask matrix is hidden to obtain a first target mask matrix, and the training text includes the first target sub-text; the initial model is trained according to the training text, the target sequence, and the first target mask matrix to obtain a target model, and the target model is used to correct the punctuation characters that are misused in the text input to the target model. Through the above method, a target model for correcting the punctuation characters that are misused in the input text, that is, the text input to the target model, can be obtained. When correcting the punctuation marks of the text, since the target model is used for correction, the manual participation degree is low, and the correction efficiency can be improved. Description of the Drawings

[0031] Figure 1 is a flowchart of a model training method provided by an embodiment of the present application;

[0032] Figure 2a is a schematic structural diagram of an encoding block provided by an embodiment of the present application;

[0033] Figure 2b is a schematic diagram of a mask processing process provided by an embodiment of the present application;

[0034] Figure 2c is another schematic diagram of a mask processing process provided by an embodiment of the present application;

[0035] Figure 3 is a flowchart of a text processing method provided by an embodiment of the present application;

[0036] Figure 4 is another flowchart of the text processing method provided by the embodiments of the present application;

[0037] Figure 5 is a structural diagram of the model training device provided by the embodiments of the present application;

[0038] Figure 6 is a structural diagram of the text processing device provided by the embodiments of the present application;

[0039] Figure 7 is a structural diagram of an electronic device provided by the embodiments of the present application;

[0040] Figure 8 is a schematic diagram of the hardware structure of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0041] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0042] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. generally belong to the same category, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally means an "or" relationship between the associated objects before and after.

[0043] Next, in conjunction with the accompanying drawings, the model training method provided by the embodiments of the present application will be described in detail through specific embodiments and their application scenarios.

[0044] Figure 1 is a flowchart of the model training method provided by the embodiments of the present application. As Figure 1 shown, the model training method provided in this embodiment includes the following steps:

[0045] Step 101, obtain training text, where the training text includes punctuation characters that are misused.

[0046] The number of training texts can be multiple. By crawling the information news on the website, filter out the information texts with less than i (for example, 5) punctuation marks (which can also be called punctuation characters) and less than j (for example, 50) characters in text length, where both i and j are positive integers. Randomly select some texts from the filtered information texts. For the selected texts, randomly replace the punctuation marks in the texts with incorrect punctuation marks to obtain the training texts.

[0047] For example, for the information text "On July 1st, Beijing time, the draw for the group stage of the 12th round of the Asian Qualifiers for the 2022 FIFA World Cup in Qatar was held. The Chinese men's national football team was drawn into the same group as Vietnam, Oman, Saudi Arabia, Australia, and Japan.", replace the last punctuation mark "." in this information text with the incorrect punctuation mark "?", and the obtained training text is "On July 1st, Beijing time, the draw for the group stage of the 12th round of the Asian Qualifiers for the 2022 FIFA World Cup in Qatar was held. The Chinese men's national football team was drawn into the same group as Vietnam, Oman, Saudi Arabia, Australia, and Japan?".

[0048] Step 102: Generate a target sequence according to the training text.

[0049] Based on predefined conversion rules, a target sequence can be generated according to the training text. The target sequence can be a digital sequence. The target sequence is the sequence obtained after correcting the incorrectly used punctuation characters in the training text. That is to say, the target sequence corresponds to the sequence of the text with correct punctuation character usage.

[0050] Step 103: Generate a first initial mask matrix based on the training text.

[0051] The first initial mask matrix includes N rows and N columns, where N is the sum of the number of literal characters and punctuation characters in the training text, and N is a positive integer. For example, if the training text is "Hello, I'm fine, everyone is fine!", the sum of the number of literal characters and punctuation characters in the training text is 10, then the first initial mask matrix is 10 rows and 10 columns. Then, "Hello" corresponds to the first row and the first column, "I'm fine" corresponds to the second row and the second column, "," corresponds to the third row and the third column, and so on.

[0052] Furthermore, the values at each position in the first initial mask matrix can be initialized to 0.

[0053] Step 104: Hide the first target sub-text in the first initial mask matrix to obtain a first target mask matrix, and the training text includes the first target sub-text.

[0054] For the punctuation marks in the text, whether the punctuation mark is correct is only related to the M sentences that are arranged in front of it in the word order and are the closest in position. Other sentences can be hidden to avoid interference. M is a positive integer, and preferably takes values of 1, 2, or 3. For ease of description, in the embodiments of the present application, M takes the value of 1.

[0055] For example, in the training text "Hello, I'm fine, everyone is fine!", for the second punctuation mark ",", whether this punctuation mark is correct is only related to the closest sentence in front of it, that is, related to "I'm fine". Then, "Hello," and "everyone is fine!" need to be hidden. For the second punctuation mark ",", the first target sub-text includes "Hello," and "everyone is fine!"; for the third punctuation mark "!", what needs to be hidden is "Hello, I'm fine,", so for the "!" in the training text, the first target sub-text includes "Hello, I'm fine,". That is to say, for the punctuation marks in the text, the text located before the punctuation mark and with a distance exceeding M sentences from the punctuation mark can be hidden, and the text after the punctuation mark can be hidden.

[0056] In the above, hiding the first target sub-text means setting the positions corresponding to the first target sub-text in the first initial mask matrix to a preset value. For example, the values of the positions corresponding to the first target sub-text can be set to -inf (negative infinity). By setting the positions corresponding to the first target sub-text in the first initial mask matrix to -inf, in subsequent operations, the values of the positions corresponding to the first target sub-text do not participate in the operations, thereby achieving the hiding of the first target sub-text and reducing the interference of the first target sub-text.

[0057] Step 105: Train the initial model according to the training text, the target sequence, and the first target mask matrix to obtain a target model, where the target model is used to correct the punctuation characters misused in the text input to the target model.

[0058] The target sequence can be regarded as a reference sequence. The initial model is trained according to the training text and the target sequence to obtain a target model. The initial model can adopt an encoder-decoder structure, and the encoder can use a Bert model. The target model obtained after training is used to correct the punctuation characters misused in the text input to the target model. That is to say, the output text with correct punctuation character usage can be obtained through the target model.

[0059] In this embodiment, a training text is obtained, and the training text includes punctuation characters that are misused; according to the training text, a target sequence is generated, and the target sequence is a sequence obtained after correcting the misused punctuation characters in the training text; based on the training text, a first initial mask matrix is generated; a first target sub-text in the first initial mask matrix is hidden to obtain a first target mask matrix, and the training text includes the first target sub-text; the initial model is trained according to the training text, the target sequence, and the first target mask matrix to obtain a target model, and the target model is used to correct the misused punctuation characters in the text input to the target model. Through the above method, a target model for correcting the misused punctuation characters in the text input to the target model can be obtained. When correcting the punctuation of the text, since the target model is used for correction, the manual participation degree is low, and the correction efficiency can be improved.

[0060] In the above, the generating the target sequence according to the training text includes:

[0061] A vocabulary is obtained, and the vocabulary includes the correspondence between characters and first labels, and the correspondence between conversion operations and second labels. The characters include literal characters and punctuation characters;

[0062] According to the vocabulary, the training text is converted into the target sequence.

[0063] Specifically, the vocabulary may include the correspondence between characters and first labels, and the characters can include literal characters and punctuation characters. As shown in the vocabulary in Table 1, the label corresponding to the literal character (i.e., the non-punctuation symbol in Table 1) is 0, and the label corresponding to "," is 3. In addition, the vocabulary also includes the correspondence between conversion operations and second labels. As shown in Table 1, the conversion operation "keep" corresponds to the label 1, and the conversion operation "delete" corresponds to the label 2.

[0064] Table 1

[0065] Label Character (or conversion operation) 0 Non-punctuation symbol 1 Reserved 2 Delete 3 Comma (,) 4 Full stop (.) 5 Question mark (? 6 Exclamation mark (!) 7 Colon (:) … …

[0066] Based on the vocabulary, the training text can be converted into the target sequence. When converting, it is necessary to convert the misused punctuation characters in the training text into correct characters. That is to say, the target sequence corresponds to the sequence of the text with correct punctuation characters.

[0067] In an embodiment of the present application, the first label includes a first sub-label and the second sub-label, and the second label includes the third sub-label;

[0068] Correspondingly, the converting the training text into the target sequence according to the vocabulary includes:

[0069] Convert the literal characters in the training text into the first sub-label according to the vocabulary list;

[0070] Convert the punctuation characters misused in the training text into the second sub-label to obtain the target sequence. The label corresponding to the first punctuation character in the vocabulary list is the second sub-label, and the first punctuation character is the character obtained after correcting the misused punctuation character;

[0071] Convert the punctuation characters correctly used in the training text into the third sub-label, where the third sub-label is the label corresponding to the target conversion operation in the vocabulary list, and the punctuation characters correctly used correspond to the target conversion operation.

[0072] As shown in Table 1, the label "0" corresponding to non-punctuation symbols in Table 1 is the first sub-label, and the literal characters in the training text correspond to the same label. Taking the training text "On July 1st, Beijing time, the draw for the group stage of the 12th round of the Asian Qualifiers for the 2022 FIFA World Cup in Qatar was held. The Chinese men's national football team was grouped with Vietnam, Oman, Saudi Arabia, Australia, and Japan?" as an example, convert the Chinese characters and other non-punctuation symbols into the serial number "0".

[0073] When converting the punctuation characters misused in the training text, it should be converted into the label corresponding to the correct punctuation character after correcting the misused punctuation character. For example, in the above example, the last character "?" should be a period ".", and when converting, it should be converted into the label "4" corresponding to the period ".".

[0074] When converting the punctuation characters correctly used in the training text, it is converted into the label corresponding to the "keep" operation, and the "keep" operation is the operation corresponding to the punctuation characters correctly used. For example, in the above example, the first ',', is converted into the label "1", indicating that the correct punctuation character is kept.

[0075] Further, add the serial number "0" at the beginning and end of the target sequence respectively as the start symbol and end symbol, and input it into the initial model.

[0076] In an embodiment of the present application, the initial model includes an encoder and a decoder, and the decoder uses the first target mask matrix for decoding.

[0077] Specifically, the initial model adopts an encoder-decoder structure. Among them, the encoder uses the Bert model to extract semantic features in the text. The decoder uses a six-layer Transformer Decoder model. Compared with the encoder in the encoder, the Transformer block has one more encoder-decoder attention. As Figure 2a shown in the schematic diagram of the decoding block (Decoder Block) structure, which includes three layers, namely the masked self-attention layer (Masked Self-Attention), the encoder-decoder self-attention (Encoder-Decoder Self-Attention) layer, and the feed-forward neural network (Feed Forward Neural Network) layer.

[0078] The Masked Self-Attention in the Decoder Block can hide future token information, and an Attention mask is added inside. The Attention mask can be implemented by generating a vector with infinitesimal values in the upper triangular region. The upper triangular region corresponds to the part that needs to be masked. The specific process is as follows Figure 2b shown in the flow chart of the Attention mask processing.

[0079] The above Attention mask process is to mask the future information behind this position during inference. In the scenario of symbol error correction, the information of each symbol is basically only related to one or more sentences in front of it. Therefore, when generating the first target mask matrix according to the training text, the following steps can be specifically carried out:

[0080] (1) Obtain punctuation position information, where the punctuation position information includes the positions of each punctuation character in the training text in the training text;

[0081] (2) Determine a first initial mask matrix according to the training text. The first initial mask matrix includes N rows and N columns, and N is the sum of the number of text characters and punctuation characters in the training text;

[0082] (3) Hide the first target sub - text in the first sequence vector according to the punctuation position information to obtain the first target mask matrix. The value at the first target position corresponding to the first target sub - text in the first target mask matrix is the first target value, such as - inf (negative infinity). The first target sub - text is the text in the training text except the second target sub - text. The second target sub - text is the target character before the second punctuation character and the second punctuation character in the training text. The second punctuation character is any punctuation character in the training text;

[0083] The value at the second target position corresponding to the second target sub - text in the first target mask matrix is the second target value, such as 0.

[0084] Among them, the value at the second target position in the first target mask matrix is the second target value. The second target position is the position corresponding to the target character before the second punctuation character and the second punctuation character in the training text. The second punctuation character is any punctuation character in the punctuation characters included in the training text. The row where the second target position is located is the same as the row corresponding to the second punctuation character; the value at the first target position in the first target mask matrix is the first target value. The first target position is the position except the second target position in the row corresponding to the second punctuation character.

[0085] Specifically, traverse the training text and record the position of each punctuation character. Due to the existence of the start symbol in the target sequence corresponding to the training text, the position of each punctuation character needs to be incremented by 1. Taking the training text "On July 1st, Beijing time, the draw for the group stage of the 12 - team round of the 2022 FIFA World Cup Asian Qualifiers was held. The Chinese men's national football team was grouped with Vietnam, Oman, Saudi Arabia, Australia, and Japan?" as an example, the position of the first symbol ',' is 8 (the position of the first character is 0), and 1 needs to be added when recording. So 9 is added to the punctuation position array. Finally, the punctuation position array of this training text is [9, 33, 41, 44, 47, 52, 59], and thus the punctuation position information of the training text is obtained.

[0086] Construct a first initial mask matrix according to the training text. For example, if the sum of the number of text characters and punctuation characters included in the training text is 10, the first initial mask matrix is 10 rows and 10 columns. For example, if the training text is "Hello, I'm fine, everyone is fine!", then "Hello" corresponds to the first row and the first column, "I'm fine" corresponds to the second row and the second column, and so on. Further, the values at each position in the first initial mask matrix can be initialized to - inf (negative infinity), that is, the first target value.

[0087] Based on the punctuation position information, determine the value corresponding to the target position in the first initial mask matrix. For any punctuation character in the punctuation characters included in the training text (for ease of description, this arbitrary punctuation character may be referred to as the second punctuation character), determine the row corresponding to the second punctuation character in the first initial mask matrix, and then determine the target characters (which can be one character or multiple characters) before the second punctuation character in the training text and the position corresponding to the second punctuation character as the second target position. Further, the target characters can be the characters included in one or more sentences adjacent to and before the second punctuation character in the training text.

[0088] For ease of illustration, a simple example is given below for explanation. If the punctuation position information is [1, 3], that is, the positions of the punctuation marks in the training text are 1 and 3. If the information of each symbol is only related to the sentence immediately preceding it (for ease of illustration, a sentence in the example only includes one character, in fact, a sentence can include multiple characters), then the punctuation mark at position "1" is only related to the characters from position 0 to 1, and the punctuation mark at position "3" is only related to the characters from position 2 to 3. As Figure 2c Shown is the improved Attention Mask (i.e., the first target mask matrix) of the present application. In the figure, for the second row (the row number of the first row is 0, and the row number of the second row is 1, that is, the row corresponding to the first punctuation mark), only the columns with column numbers from 0 to 1 are reserved as the number 0 (i.e., the second target value), and the remaining columns are set to -inf (negative infinity). The positions set to -inf can achieve hiding, and -inf is the first target value. Similarly, for the fourth row (the row number of the fourth row is 3, that is, the row corresponding to the second punctuation mark), only the columns with column numbers from 2 to 3 are reserved as the number 0, and the rest are set to -inf. The improved Attention Mask is as Figure 2c shown. If the information of two sentences is to be retained, then the fourth row will retain the information of the columns with column numbers from 0 to 3.

[0089] The decoder improved in the above manner can obtain the vector information output at each position, and then infer which word in the vocabulary is at each position according to the vector information and the vocabulary. The word at each position output by the model is a character such as 0, 1, 2, 3... Then, the loss is calculated between the output result and the target sequence to update the model parameters of the initial model, and finally the target model is obtained.

[0090] Figure 3 The flowchart of the text processing method provided by the embodiment of the present application is as Figure 3 shown. The text processing method provided in this embodiment can be executed by a second electronic device, and includes the following steps:

[0091] Step 301, obtain the text to be processed, where the text to be processed includes punctuation characters.

[0092] Step 302: Generate a second initial mask matrix according to the text to be processed.

[0093] The second initial mask matrix includes U rows and U columns, where U is the sum of the number of literal characters and punctuation characters in the text to be processed, and U is a positive integer. For example, if the training text is "Hello, I'm fine, everyone is fine!", and the sum of the number of literal characters and punctuation characters in the text to be processed is 10, then the second initial mask matrix is 10 rows and 10 columns. Then, "Hello" corresponds to the first row and the first column, "fine" corresponds to the second row and the second column, ",", corresponds to the third row and the third column, and so on.

[0094] The values at each position in the second initial mask matrix can be initialized to 0.

[0095] Step 303: Hide the third target sub-text in the second initial mask matrix to obtain a second target mask matrix, where the text to be processed includes the third target sub-text;

[0096] For the punctuation marks in the text, whether the punctuation mark is correct only depends on the M sentences that are in front of it in the word order and are the closest in position, where M is a positive integer, and preferably takes values of 1, 2, or 3. For the convenience of description, in the embodiments of the present application, M takes the value of 1.

[0097] For example, in the text to be processed "Hello, I'm fine, everyone is fine!", for the second punctuation mark ",", whether the punctuation mark is correct only depends on the sentence closest to it in front, that is, related to "I'm fine". Then, "Hello," and "everyone is fine!" need to be hidden. For the second punctuation mark ",", the third target sub-text includes "Hello," and "everyone is fine!"; for the third punctuation mark "!", what needs to be hidden is "Hello, I'm fine,". Then, for the "!" in the text to be processed, the third target sub-text includes "Hello, I'm fine,". For the text that needs to be hidden in the second initial mask matrix, the value at the corresponding position of the text can be set to -inf (negative infinity).

[0098] Step 304: According to the text to be processed and the second target mask matrix, use the target model to correct the punctuation characters misused in the text to be processed to obtain the target text. The target model is used to correct the punctuation characters misused in the text input to the target model, and the target model is the target model obtained according to the Figure 1 model training method shown in the embodiments.

[0099] Taking the text to be processed "On July 1st, Beijing time, the draw for the group stage of the 12th round of the Asian qualifiers for the 2022 FIFA World Cup in Qatar was held. The Chinese men's national football team was grouped with Vietnam, Oman, Saudi Arabia, Australia, and Japan?" as an example, after inputting it into the target model, the result [0, 0……,0,1,0,……,4,0] is obtained, and the punctuation position information is [9, 33, 41, 44, 47, 52, 59]. The punctuation symbol results [1,1,1,1,1,1,4] can be extracted from the output result. According to the vocabulary table, it can be found that as long as the result is greater than 1, it means that the punctuation symbol is incorrect, and the incorrect punctuation symbol has been corrected to 4 (i.e., a full stop).

[0100] In this embodiment, a text to be corrected is obtained, and the text to be corrected includes punctuation characters; a second initial mask matrix is generated according to the text to be corrected; a third target sub-text in the second initial mask matrix is hidden to obtain a second target mask matrix, and the text to be corrected includes the third target sub-text; according to the text to be corrected and the second target mask matrix, a target model is used to correct the punctuation characters misused in the text to be corrected to obtain a target text. The target model can correct the punctuation characters misused in the text to be processed to obtain a text with corrected punctuation characters. The correction process has a low degree of manual participation and can improve the correction efficiency.

[0101] The following gives an example of the model training method and text processing method provided in this application. As Figure 4 shown, it includes:

[0102] Step 401: Construct a training data set. Crawl news. Screen out the news with less than 5 punctuation marks and a text length of less than 50. Randomly select some data from the training set, traverse this part of the data, and randomly replace the punctuation marks in the text with other incorrect punctuation marks.

[0103] Step 402: Generate a target sequence. According to the custom vocabulary table, as shown in Table 1, the characters in the text are converted into their corresponding serial numbers to form a target sequence. Taking the text "On July 1st, Beijing time, the draw for the group stage of the 12th round of the Asian qualifiers for the 2022 FIFA World Cup in Qatar was held. The Chinese men's national football team was grouped with Vietnam, Oman, Saudi Arabia, Australia, and Japan?" as an example, the non-punctuation characters such as Chinese characters in it are converted into the serial number "0", and if it is a correct punctuation mark, such as the first ',', it is converted into the serial number "1", which means that the correct punctuation usage can be retained. If it is an incorrect punctuation mark, such as the last character "?", the correct usage should be a full stop ".", so it is converted into the serial number "4". Since start and end symbols will be added when the text is passed into the model, the serial numbers "0" are added at the beginning and end of the target sequence respectively.

[0104] Step 403: Record the positions of punctuation marks in the text. Traverse each piece of text, record the positions of each punctuation mark therein, and due to the starting symbol, add 1 to the position of each punctuation mark. Taking the sentence in Step 402 as an example, the position of the first character is recorded as 0, so the position of the first symbol ",", is 8, and 1 needs to be added during recording. Thus, 9 is added to the punctuation position array. Finally, the recorded punctuation position array for this sentence is [9, 33, 41, 44, 47, 52, 59].

[0105] Step 404: Build a generation model. The model adopts an encoder-decoder structure. Among them, the encoder uses the Bert model to extract semantic features in the text. The decoder uses a six-layer Transformer Decoder model. Compared with the encoder in the encoder, the Transformer block in the decoder has an additional encoder-decoder attention, as follows Figure 2a shown. In the encoder-decoder attention, Q comes from the previous output of the decoder, and K and V come from the output of the encoder.

[0106] In the first self-attention in the decoder, future token information is hidden and an attention mask is added. The attention mask can be implemented by generating a vector with infinitesimal values in the upper triangle. The upper triangular region corresponds to the part to be masked. The specific process is as follows Figure 2b shown.

[0107] Figure 2b The future information after this position is masked by the Attention mask shown above, which conforms to the normal reasoning logic of people. In the scenario of symbol error correction, the information of each symbol is basically only related to one or two sentences in front of it. More previous information will interfere with the model's reasoning. While masking the future information after this position, the information of more than one or two sentences in front also needs to be masked. Whether to retain the information of one or two sentences can be configured as a hyperparameter.

[0108] Here, taking the example of retaining the information of one sentence for simplicity, if the punctuation position information of the text is [1, 3], it can be understood that the punctuation information at position "1" is only related to the characters from position 0 to 1, and the punctuation information at position "3" is only related to the characters from position 2 to 3. Figure 2b In the second row (row number 1, i.e., the first punctuation) of the Attention Mask, only columns 0 to 1 are retained as the number 0, and the remaining columns are set to -inf (negative infinity). Similarly, in the fourth row, only columns 2 to 3 are retained as the number 0, and the rest are set to -inf. Thus, the improved Attention Mask process is as followsFigure 2c As shown. If the information of two sentences is retained, then the information from column 0 to column 3 will be retained in the fourth line here.

[0109] After the Decoder improved through the above process, the vector information output at each position can be obtained, and then based on the vector information and the vocabulary, it can be inferred which word in the vocabulary each position is. It should be noted that the vocabulary here still uses the vocabulary in step 402. In this way, the word at each position output by the model is a character such as 0, 1, 2, 3... Then, the output result is used to calculate the loss with the target sequence marked in step 402 to update the model parameters.

[0110] Step 405: Input the training data and train the generation model. Input the training data set constructed in step 401 and the target sequence obtained in step 402 into the model built in step 404. After training, a model for correcting the incorrect punctuation marks in the text can be obtained.

[0111] Step 406: After obtaining the trained model, the target model can be used to correct the punctuation marks in the text. Taking the text "On July 1st, Beijing time, the draw for the group stage of the 12th round of the Asian qualifiers for the 2022 FIFA World Cup in Qatar was held. The Chinese men's national football team was drawn into the same group as Vietnam, Oman, Saudi Arabia, Australia, and Japan?" as an example, after inputting it into the model, the result [0, 0……,0,1,0,……,4,0] is obtained. Then, through the recorded punctuation position information [9, 33, 41, 44, 47, 52, 59], the punctuation mark result [1, 1, 1, 1, 1, 1, 4] can be extracted from this result. As long as the result is greater than 1, it indicates that the punctuation mark is incorrect and has been replaced with 4 (i.e., a full stop).

[0112] The target model in this application can directly combine the error judgment stage and the inference stage to improve the inference efficiency; and it can target each punctuation mark in a long text at one time, which is more in line with the real-life scenario and has higher application value. The improved Attention Mask strategy not only blocks the future information behind but also blocks the interference of the information in front, ensuring that the result at each position during inference only retains the information of the previous one or two sentences, thus improving the accuracy of the target model inference.

[0113] It should be noted that for the model training method provided in the embodiments of this application, the execution subject can be a model training device, or a control module in the model training device for executing the model training method. In the embodiments of this application, the method of the model training device executing the model training is taken as an example to illustrate the model training device provided in the embodiments of this application.

[0114] As Figure 5 As shown, the model training device 500 provided in the embodiments of this application includes:

[0115] The first acquisition module 501 is configured to acquire a training text, where the training text includes punctuation characters that are misused.

[0116] The first generation module 502 is configured to generate a target sequence according to the training text, where the target sequence is a sequence obtained by correcting the punctuation characters that are misused in the training text.

[0117] The second generation module 503 is configured to generate a first initial mask matrix based on the training text.

[0118] The second acquisition module 504 is configured to hide a first target sub-text in the first initial mask matrix to obtain a first target mask matrix, where the training text includes the first target sub-text.

[0119] The third acquisition module 505 is configured to train an initial model according to the training text, the target sequence, and the first target mask matrix to obtain a target model, where the target model is used to correct the punctuation characters that are misused in the text input to the target model.

[0120] Optionally, the first generation module 502 includes:

[0121] The first acquisition sub-module is configured to acquire a vocabulary, where the vocabulary includes the correspondence between characters and first labels, and the correspondence between conversion operations and second labels, and the characters include literal characters and punctuation characters.

[0122] The conversion sub-module is configured to convert the training text into the target sequence according to the vocabulary.

[0123] Optionally, the first label includes a first sub-label and a second sub-label, and the second label includes a third sub-label.

[0124] The conversion sub-module includes:

[0125] The first conversion unit is configured to convert the literal characters in the training text into the first sub-label according to the vocabulary.

[0126] The second conversion unit is configured to convert the punctuation characters that are misused in the training text into the second sub-label to obtain the target sequence, where the label corresponding to the first punctuation character in the vocabulary is the second sub-label, and the first punctuation character is the character obtained by correcting the misused punctuation character.

[0127] A third conversion unit, configured to convert the punctuation characters correctly used in the training text into the third sub-labels, where the third sub-labels are the labels corresponding to the target conversion operations in the vocabulary, and the punctuation characters correctly used correspond to the target conversion operations.

[0128] Optionally, the initial model includes an encoder and a decoder, and the decoder decodes using a first target mask matrix.

[0129] Optionally, the first initial mask matrix includes N rows and N columns, where N is the sum of the number of literal characters and punctuation characters in the training text, and N is a positive integer;

[0130] The second acquisition module includes:

[0131] A second acquisition sub-module, configured to acquire punctuation position information, where the punctuation position information includes the positions of each punctuation character in the training text in the training text;

[0132] A third acquisition sub-module, configured to hide a first target sub-text in the first initial mask matrix according to the punctuation position information to obtain the first target mask matrix;

[0133] Wherein, the value of the first target position corresponding to the first target sub-text in the first target mask matrix is a first target value, the first target sub-text is the text in the training text except the second target sub-text, and the second target sub-text is the target characters before the second punctuation character and the second punctuation character in the training text, and the second punctuation character is any punctuation character in the training text;

[0134] The value of the second target position corresponding to the second target sub-text in the first target mask matrix is a second target value.

[0135] The model training device 500 provided by the embodiments of the present application can implement Figure 1 each process implemented by the method embodiments and obtain the same technical effects. To avoid repetition, details are not described herein again.

[0136] As Figure 6 shown, the embodiments of the present application further provide a text processing device 600, including:

[0137] A first acquisition module 601, configured to acquire a text to be processed, where the text to be processed includes punctuation characters;

[0138] A generation module 602, configured to generate a second initial mask matrix according to the text to be processed;

[0139] A second acquisition module 603, configured to hide a third target sub-text in the second initial mask matrix to obtain a second target mask matrix, where the text to be processed includes the third target sub-text;

[0140] A third acquisition module 604, configured to correct misused punctuation characters in the text to be processed by using a target model according to the text to be processed and the second target mask matrix, so as to obtain a target text. The target model is used to correct misused punctuation characters in the text input to the target model, and the target model is obtained by the above model training method.

[0141] The text processing device 600 provided in the embodiments of the present application can implement Figure 3 each process implemented by the method embodiments and obtain the same technical effects. To avoid repetition, details are not described herein again.

[0142] The above model training device 500 and text processing device 600 may be a device, or a component, an integrated circuit, or a chip in a terminal. The device may be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a handheld computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device may be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0143] The model training device 500 and the text processing device 600 in the embodiments of the present application may be devices with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0144] Optionally, as Figure 7 shown, the embodiments of the present application further provide an electronic device 700, including a processor 701, a memory 702, and a program or instruction stored in the memory 702 and executable on the processor 701. When the program or instruction is executed by the processor 701, it implements each process of the above model training method embodiment or text processing method embodiment and can achieve the same technical effects. To avoid repetition, details are not described herein again.

[0145] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0146] Figure 8 Schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application.

[0147] The electronic device 1000 includes, but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010, etc.

[0148] Those skilled in the art can understand that the electronic device 1000 may further include a power source (such as a battery) for supplying power to each component. The power source can be logically connected to the processor 1010 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. Figure 8 The structure of the electronic device shown does not limit the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0149] Among them, in an embodiment of the present application, the input unit 1004 is used to obtain a training text, and the training text includes punctuation characters that are misused.

[0150] The processor 1010 is used to generate a target sequence according to the training text, and the target sequence is a sequence obtained after correcting the punctuation characters misused in the training text; generate a first initial mask matrix based on the training text; hide a first target sub-text in the first initial mask matrix to obtain a first target mask matrix, and the training text includes the first target sub-text; train an initial model according to the training text, the target sequence, and the first target mask matrix to obtain a target model, and the target model is used to correct the punctuation characters misused in the text input to the target model.

[0151] Optionally, the processor 1010 is further used to obtain a vocabulary, where the vocabulary includes the correspondence between characters and first labels, and the correspondence between conversion operations and second labels, and the characters include literal characters and punctuation characters; convert the training text into the target sequence according to the vocabulary.

[0152] Optionally, the first label includes a first sub-label and a second sub-label, and the second label includes a third sub-label;

[0153] The processor 1010 is further configured to convert the literal characters in the training text into the first sub-labels according to the vocabulary; convert the punctuation characters misused in the training text into the second sub-labels to obtain the target sequence, where the label corresponding to the first punctuation character in the vocabulary is the second sub-label, and the first punctuation character is the character obtained after correcting the misused punctuation character; convert the punctuation characters correctly used in the training text into the third sub-labels, where the third sub-label is the label corresponding to the target conversion operation in the vocabulary, and the correctly used punctuation character corresponds to the target conversion operation.

[0154] Optionally, the initial model includes an encoder and a decoder, and the decoder decodes using a first target mask matrix.

[0155] Optionally, the first initial mask matrix includes N rows and N columns, where N is the sum of the number of literal characters and punctuation characters in the training text, and N is a positive integer; the processor 1010 is further configured to obtain punctuation position information, where the punctuation position information includes the position of each punctuation character in the training text in the training text;

[0156] Hide the first target sub-text in the first initial mask matrix according to the punctuation position information to obtain the first target mask matrix;

[0157] Wherein, the value of the first target position corresponding to the first target sub-text in the first target mask matrix is a first target value, the first target sub-text is the text in the training text except the second target sub-text, the second target sub-text is the target characters before the second punctuation character and the second punctuation character in the training text, and the second punctuation character is any punctuation character in the training text;

[0158] The value of the second target position corresponding to the second target sub-text in the first target mask matrix is a second target value.

[0159] In an embodiment of the present application, the input unit 1004 is configured to obtain a text to be processed, where the text to be processed includes punctuation characters;

[0160] The processor 1010 is further configured to generate a second initial mask matrix according to the text to be processed; hide a third target sub - text in the second initial mask matrix to obtain a second target mask matrix, where the text to be processed includes the third target sub - text; and correct misused punctuation characters in the text to be processed according to the text to be processed and the second target mask matrix by using a target model, so as to obtain a target text, where the target model is used to correct misused punctuation characters in the text input to the target model, and the target model is obtained by the above - mentioned model training method.

[0161] It should be understood that, in the embodiments of the present application, the input unit 1004 may include a Graphics Processing Unit (GPU) 10041 and a microphone 10042. The graphics processor 10041 processes image data of static pictures or videos obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light - emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. The other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be elaborated here. The memory 1009 may be used to store software programs and various data, including but not limited to application programs and operating systems. The processor 1010 may integrate an application processor and a modulation and demodulation processor. Among them, the application processor mainly processes the operating system, user interface, and application programs, and the modulation and demodulation processor mainly processes wireless communication. It can be understood that the above - mentioned modulation and demodulation processor may not be integrated into the processor 1010.

[0162] The embodiments of the present application further provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements each process of the above - mentioned model training method embodiment or the text processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0163] Among them, the processor is the processor in the electronic device described in the above - mentioned embodiment. The readable storage medium includes a computer - readable storage medium, such as a computer Read - Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disc, etc.

[0164] Another embodiment of the present application further provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is configured to run programs or instructions to implement the various processes of the above-mentioned embodiment of the model training method or the embodiment of the text processing method, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0165] It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0166] It should be noted that in this article, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the existence of additional identical elements in the process, method, article or device comprising such element. In addition, it should be pointed out that the methods and devices in the embodiments of the present application are not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, the features described with reference to certain examples may be combined in other examples.

[0167] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions for causing a first terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present application.

[0168] The embodiments of the present application have been described above with reference to the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection scope of the present application.

Claims

1. A model training method, characterized in that, The method includes: Obtaining a training text, where the training text includes punctuation characters that are misused; Generating a target sequence according to the training text, where the target sequence is a sequence obtained after correcting the punctuation characters that are misused in the training text; Generating a first initial mask matrix based on the training text; Hiding a first target sub-text in the first initial mask matrix to obtain a first target mask matrix, where the training text includes the first target sub-text, and hiding the first target sub-text in the first initial mask matrix means setting the positions corresponding to the first target sub-text in the first initial mask matrix to a preset value; Training an initial model according to the training text, the target sequence, and the first target mask matrix to obtain a target model, where the target model is used to correct the punctuation characters that are misused in the text input to the target model; The initial model includes an encoder and a decoder, and the decoder uses the first target mask matrix for decoding; The first initial mask matrix includes N rows and N columns, where N is the sum of the number of literal characters and punctuation characters in the training text, and N is a positive integer; The hiding the first target sub-text in the first initial mask matrix to obtain a first target mask matrix includes: Obtaining punctuation position information, where the punctuation position information includes the positions of each punctuation character in the training text in the training text; Hiding the first target sub-text in the first initial mask matrix according to the punctuation position information to obtain the first target mask matrix; Wherein, the value of the first target position corresponding to the first target sub-text in the first target mask matrix is a first target value, the first target sub-text is the text in the training text except for the second target sub-text, the second target sub-text is the target character before the second punctuation character and the second punctuation character in the training text, and the second punctuation character is any punctuation character in the training text; The value of the second target position corresponding to the second target sub-text in the first target mask matrix is a second target value.

2. The model training method according to claim 1, wherein The generating a target sequence according to the training text includes: Obtaining a vocabulary, where the vocabulary includes the correspondence between characters and first labels, and the correspondence between conversion operations and second labels, and the characters include literal characters and punctuation characters; Converting the training text into the target sequence according to the vocabulary.

3. The model training method according to claim 2, wherein The first label includes a first sub-label and a second sub-label, and the second label includes a third sub-label; The converting the training text into the target sequence according to the vocabulary includes: Converting the literal characters in the training text into the first sub-label according to the vocabulary; Converting the punctuation characters that are misused in the training text into the second sub-label to obtain the target sequence, where the label corresponding to the first punctuation character in the vocabulary is the second sub-label, and the first punctuation character is the character obtained after correcting the misused punctuation character; Convert the punctuation characters correctly used in the training text into the third sub-label, where the third sub-label is the label corresponding to the target conversion operation in the vocabulary, and the punctuation characters correctly used correspond to the target conversion operation.

4. A text processing method, characterized in that, The method includes: Obtain a text to be processed, where the text to be processed includes punctuation characters; Generate a second initial mask matrix according to the text to be processed; Hide the third target sub-text in the second initial mask matrix to obtain a second target mask matrix. The text to be processed includes the third target sub-text. Hiding the third target sub-text in the second initial mask matrix means setting the position corresponding to the third target sub-text in the second initial mask matrix to a preset value; According to the text to be processed and the second target mask matrix, use the target model according to any one of claims 1-3 to correct the punctuation characters misused in the text to be processed to obtain a target text.

5. A model training device, characterized in that, The device includes: A first acquisition module for acquiring a training text, where the training text includes punctuation characters misused; A first generation module for generating a target sequence according to the training text, where the target sequence is a sequence obtained after correcting the punctuation characters misused in the training text; A second generation module for generating a first initial mask matrix based on the training text; A second acquisition module for hiding the first target sub-text in the first initial mask matrix to obtain a first target mask matrix. The training text includes the first target sub-text. Hiding the first target sub-text in the first initial mask matrix means setting the position corresponding to the first target sub-text in the first initial mask matrix to a preset value; A third acquisition module for training an initial model according to the training text, the target sequence, and the first target mask matrix to obtain a target model, where the target model is used to correct the punctuation characters misused in the text input to the target model; The initial model includes an encoder and a decoder, and the decoder uses the first target mask matrix for decoding; The first initial mask matrix includes N rows and N columns, where N is the sum of the number of literal characters and punctuation characters in the training text, and N is a positive integer; The second acquisition module includes: A second acquisition sub-module for acquiring punctuation position information, where the punctuation position information includes the position of each punctuation character in the training text in the training text; A third acquisition sub-module for hiding the first target sub-text in the first initial mask matrix according to the punctuation position information to obtain the first target mask matrix; Among them, the value at the first target position corresponding to the first target sub - text in the first target mask matrix is the first target value. The first target sub - text is the text in the training text except the second target sub - text. The second target sub - text is the target character before the second punctuation character and the second punctuation character in the training text, and the second punctuation character is any punctuation character in the training text; The value at the second target position corresponding to the second target sub - text in the first target mask matrix is the second target value.

6. The model training device according to claim 5, wherein The first generation module includes: The first acquisition sub - module is used to acquire a vocabulary table, which includes the correspondence between characters and the first label, and the correspondence between conversion operations and the second label. The characters include literal characters and punctuation characters; The conversion sub - module is used to convert the training text into the target sequence according to the vocabulary table.

7. The model training device according to claim 6, wherein The first label includes a first sub - label and a second sub - label, and the second label includes a third sub - label; The conversion sub - module includes: The first conversion unit is used to convert the literal characters in the training text into the first sub - label according to the vocabulary table; The second conversion unit is used to convert the punctuation characters misused in the training text into the second sub - label to obtain the target sequence. The label corresponding to the first punctuation character in the vocabulary table is the second sub - label, and the first punctuation character is the character obtained after correcting the misused punctuation character; The third conversion unit is used to convert the punctuation characters correctly used in the training text into the third sub - label, where the third sub - label is the label corresponding to the target conversion operation in the vocabulary table, and the punctuation characters correctly used correspond to the target conversion operation.

8. A text processing device, characterized in that, The device includes: The first acquisition module is used to acquire the text to be processed, and the text to be processed includes punctuation characters; The generation module is used to generate a second initial mask matrix according to the text to be processed; The second acquisition module is used to hide the third target sub - text in the second initial mask matrix to obtain a second target mask matrix. The text to be processed includes the third target sub - text. Hiding the third target sub - text in the second initial mask matrix means setting the position corresponding to the third target sub - text in the second initial mask matrix to a preset value; The third acquisition module is used to correct the punctuation characters misused in the text to be processed according to the text to be processed and the second target mask matrix by using the target model according to any one of claims 1 - 3 to obtain the target text.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, it implements the steps of the model training method according to any one of claims 1 - 3, or when the program or instruction is executed by the processor, it implements the steps of the text processing method according to claim 4.

10. A readable storage medium, characterized in that, The program or instructions are stored on the readable storage medium, and when the program or instructions are executed by the processor, the steps of the model training method described in any one of claims 1-3 are implemented, or when the program or instructions are executed by the processor, the steps of the text processing method described in claim 4 are implemented.

Images