Text processing method and device, storage medium and processor
By combining feature extraction and error correction with a multi-task word segmentation model, and selecting target word segmentation labels, the problem of low recognition accuracy of word segmentation models caused by typos in training samples is solved, thereby improving the accuracy of word segmentation models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DUXIAOMAN TECH (BEIJING) CO LTD
- Filing Date
- 2022-03-29
- Publication Date
- 2026-04-14
AI Technical Summary
The word segmentation model had a low recognition accuracy due to typos in the training samples.
A multi-task word segmentation model is adopted. Through feature extraction and error correction, the first word segmentation model and the second word segmentation model are combined to select the target word segmentation label to determine the target word segmentation result of the input text.
This improved the recognition accuracy of the word segmentation model and solved the problem of low recognition accuracy caused by typos in the training samples.
Smart Images
Figure CN114781370B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of semantic recognition technology, and more specifically, to a text processing method, apparatus, storage medium, and processor. Background Technology
[0002] With the development of artificial intelligence technology, the application of network models such as semantic recognition is becoming increasingly widespread. In some application scenarios, the text used for semantic recognition may come from OCR (Optical Character Recognition) and manual input, which may contain a certain percentage of typos. If the word segmentation model is trained with such text containing a certain percentage of typos, the recognition accuracy of the resulting word segmentation model will be relatively low.
[0003] There is currently no effective solution to the above problems. Summary of the Invention
[0004] This invention provides a text processing method, apparatus, storage medium, and processor to at least address the technical problem of low accuracy in word segmentation model recognition caused by typos in training samples.
[0005] According to one aspect of the present invention, a text processing method is provided, comprising: acquiring input text, wherein the input text includes a plurality of input characters; performing word segmentation on the input text to obtain an initial word segmentation result of the input text, wherein the initial word segmentation result includes: word segmentation probabilities of the plurality of input characters, a first word segmentation label and a second word segmentation label, wherein the first word segmentation label is used to characterize the word segmentation label of the input text, and the second word segmentation label is used to characterize the word segmentation label obtained after correcting typos contained in the input text; selecting a target word segmentation label from the first word segmentation label and the second word segmentation label based on the word segmentation probabilities; and determining a target word segmentation result of the input text through the target word segmentation label.
[0006] Optionally, the input text is segmented to obtain the initial segmentation result, which includes: extracting features from the input text using the feature extraction module in the multi-task segmentation model to obtain target feature vectors for multiple input characters; segmenting the target feature vectors using the first segmentation model in the multi-task segmentation model to obtain the first segmentation label, wherein the first segmentation model is trained on a first dataset that does not contain misspelled characters; identifying the target feature vectors using the sub-task module in the multi-task segmentation model to obtain the segmentation probability of multiple input characters; and segmenting the target feature vectors using the second segmentation model in the multi-task segmentation model to obtain the second segmentation label, wherein the second segmentation model is trained on a second dataset that contains misspelled characters.
[0007] Optionally, the method further includes: acquiring a first dataset, wherein the first dataset contains: multiple first characters and multiple preset word segmentation labels for the first characters; generating a second dataset based on the first dataset, wherein the second dataset contains: multiple second characters and preset word segmentation labels, the multiple second characters being generated by replacing target characters among the multiple first characters; performing feature extraction on the first dataset and the second dataset respectively using a feature extraction module to obtain first feature vectors for the multiple first characters and second feature vectors for the multiple second characters; performing word segmentation processing on the first feature vectors using a first word segmentation model to obtain first predicted labels for the multiple first characters; performing recognition on the second feature vectors using a subtask module to obtain predicted probabilities for the multiple second characters; performing word segmentation processing on the second feature vectors using a second word segmentation model to obtain second predicted labels for the multiple second characters; and training a multi-task word segmentation model based on the preset word segmentation labels, predicted probabilities, first predicted labels, and second predicted labels.
[0008] Optionally, generating a second dataset based on a first dataset includes: determining a target character according to a preset probability; replacing the target character to obtain a replaced character; generating a target dataset based on other characters in the first dataset besides the target character and the replaced character; and determining the target dataset as a second dataset in response to the difference between the target dataset and the first dataset.
[0009] Optionally, a first preset identifier is added to the target location of the target dataset, wherein the first preset identifier is used to indicate that the target dataset contains typos.
[0010] Optionally, training the multi-task word segmentation model based on preset word segmentation labels, prediction probabilities, a first prediction label, and a second prediction label includes: determining a first loss function based on the preset probabilities; determining a second loss function based on the preset word segmentation labels and the first preset label; determining a third loss function based on the preset word segmentation labels and the second preset label; obtaining the sum of the first loss function, the second loss function, and the third loss function to obtain a target loss function; and training the multi-task word segmentation model based on the target loss function.
[0011] Optionally, selecting a target word segmentation tag from the first word segmentation tag and the second word segmentation tag includes: determining the first word segmentation tag as the target word segmentation tag in response to a word segmentation probability less than or equal to a preset threshold; and determining the second word segmentation tag as the target word segmentation tag in response to a word segmentation probability greater than the preset threshold.
[0012] Optionally, the input text is segmented to obtain the initial segmentation result of the input text, which includes: adding a first preset identifier and a second preset identifier to the target position of the input text to obtain a first target text and a second target text, wherein the first preset identifier is used to indicate that the first target text contains typos and the second preset identifier is used to indicate that the second target text does not contain typos; and the first target text and the second target text are identified to obtain the initial segmentation result.
[0013] According to another aspect of the present invention, a text processing apparatus is also provided, comprising: an acquisition module for acquiring input text, wherein the input text includes a plurality of input characters; a word segmentation module for segmenting the input text to obtain an initial word segmentation result of the input text, wherein the initial word segmentation result includes: word segmentation probabilities of the plurality of input characters, a first word segmentation label and a second word segmentation label, wherein the first word segmentation label is used to characterize the word segmentation label of the input text, and the second word segmentation label is used to characterize the word segmentation label obtained after correcting typos contained in the input text; a selection module for selecting a target word segmentation label from the first word segmentation label and the second word segmentation label based on the word segmentation probabilities; and a processing module for processing the target word segmentation result of the input text through the target word segmentation label.
[0014] Optionally, the word segmentation module includes: a feature extraction unit, used to extract features from the input text using the feature extraction module in the multi-task word segmentation model to obtain target feature vectors for multiple input characters; a first label unit, used to segment the target feature vectors using the first word segmentation model in the multi-task word segmentation model to obtain first word segmentation labels, wherein the first word segmentation model is trained using a first dataset that does not contain misspelled characters; a probability unit, used to identify the target feature vectors using the sub-task module in the multi-task word segmentation model to obtain the word segmentation probabilities for multiple input characters; and a second label unit, used to segment the target feature vectors using the second word segmentation model in the multi-task word segmentation model to obtain second word segmentation labels, wherein the second word segmentation model is trained using a second dataset that contains misspelled characters.
[0015] Optionally, the device further includes: a dataset acquisition module for acquiring a first dataset, wherein the first dataset contains: multiple first characters and multiple preset word segmentation labels for the first characters; a dataset generation module for generating a second dataset based on the first dataset, wherein the second dataset contains: multiple second characters and preset word segmentation labels, the multiple second characters being generated by replacing target characters among the multiple first characters; a feature extraction module for performing feature extraction on the first dataset and the second dataset respectively using the feature extraction module to obtain first feature vectors for the multiple first characters and second feature vectors for the multiple second characters; a first label module for performing word segmentation processing on the first feature vectors using a first word segmentation model to obtain first predicted labels for the multiple first characters; a first probability module for recognizing the second feature vectors using a subtask module to obtain predicted probabilities for the multiple second characters; a second label module for performing word segmentation processing on the second feature vectors using a second word segmentation model to obtain second predicted labels for the multiple second characters; and a training module for training a multi-task word segmentation model based on preset word segmentation labels, predicted probabilities, first predicted labels, and second predicted labels.
[0016] Optionally, the dataset generation module includes: a target character module for determining a target character according to a preset probability; a replacement module for replacing the target character to obtain a replaced character; a target generation unit for generating a target dataset based on other characters in the first dataset besides the target character and the replaced character; and a determination unit for determining the target dataset as a second dataset in response to the target dataset being different from the first dataset.
[0017] Optionally, the device is also used to add a first preset identifier to the target location of the target dataset, wherein the first preset identifier is used to indicate that the target dataset contains typos.
[0018] Optionally, the training module includes: a first function unit for determining a first loss function based on a preset probability; a second function unit for determining a second loss function based on preset word segmentation labels and a first preset label; a third function unit for determining a third loss function based on preset word segmentation labels and a second preset label; a target function unit for obtaining the sum of the first, second, and third loss functions to obtain a target loss function; and a model training unit for training a multi-task word segmentation model based on the target loss function.
[0019] Optionally, the selection module includes: a first response unit, used to determine the first segmentation tag as the target segmentation tag in response to a segmentation probability less than or equal to a preset threshold; and a second response unit, used to determine the second segmentation tag as the target segmentation tag in response to a segmentation probability greater than the preset threshold.
[0020] Optionally, the word segmentation module includes: an identifier unit, used to add a first preset identifier and a second preset identifier to the target position of the input text respectively to obtain a first target text and a second target text, wherein the first preset identifier is used to indicate that the first target text contains typos, and the second preset identifier is used to indicate that the second target text does not contain typos; and an identification unit, used to identify the first target text and the second target text to obtain an initial word segmentation result.
[0021] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform any one of the above-described text processing methods.
[0022] According to another aspect of the present invention, a processor is also provided for running a program, wherein the program executes any one of the above-described text processing methods during runtime.
[0023] In this embodiment of the invention, input text is obtained, wherein the input text includes multiple input characters; the input text is segmented into words to obtain an initial segmentation result, wherein the initial segmentation result includes: segmentation probabilities of multiple input characters, a first segmentation label, and a second segmentation label. The first segmentation label is used to characterize the segmentation label of the input text, and the second segmentation label is used to characterize the segmentation label obtained after correcting typos contained in the input text; based on the segmentation probabilities, a target segmentation label is selected from the first and second segmentation labels; the target segmentation result of the input text is determined by the target segmentation label. It is noteworthy that this application combines the first and second segmentation methods to segment the input text, rather than directly segmenting the input text, thereby solving the technical problem of low accuracy in word segmentation model recognition caused by typos in the training samples, and achieving the technical effect of improving the accuracy of word segmentation. Attached Figure Description
[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:
[0025] Figure 1 This is a flowchart illustrating a text processing method according to an embodiment of the present invention;
[0026] Figure 2 This is a flowchart illustrating a text processing method according to a preferred embodiment of the present invention;
[0027] Figure 3 This is a schematic diagram of the structure of a text processing device according to an embodiment of this application. Detailed Implementation
[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0030] According to an embodiment of the present invention, an embodiment of a text processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0031] Figure 1 This is a flowchart illustrating a text processing method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:
[0032] Step S102: Obtain the input text, wherein the input text includes multiple input characters.
[0033] Specifically, the input text mentioned above is the text used for subsequent word segmentation processing. It can be Chinese text, English text, or text in other languages. The input text includes multiple input characters. If the input text is Chinese text, the input character can be a single Chinese character; if the input text is English text, the input character can be a single word.
[0034] Step S104: Segment the input text to obtain the initial segmentation result of the input text. The initial segmentation result includes: segmentation probability of multiple input characters, first segmentation label and second segmentation label. The first segmentation label is used to represent the segmentation label of the input text, and the second segmentation label is used to represent the segmentation label obtained after correcting the typos contained in the input text.
[0035] Specifically, before segmenting the input text using a pre-trained multi-task word segmentation model, the multi-task recognition model needs to be trained. This includes: acquiring a first dataset and a second dataset. The first dataset includes a sample dataset of multiple first characters and their preset word segmentation labels. The second dataset can be a dataset obtained by determining the target character from the first dataset according to a preset probability, replacing the target character with a phonetically or visually similar character, and adding a first preset identifier to the beginning of the text containing the target character. The second dataset includes multiple second characters and preset word segmentation labels. A pre-trained language model is then used to extract features from the first and second datasets, resulting in first feature vectors for multiple first characters and second feature vectors for multiple second characters.
[0036] Subsequently, the first feature vector is segmented using a Segmentation FFN (Segmentation Full-Forward Network) to obtain multiple first-character prediction labels. The second feature vector is then identified using a Correction FFN (Correction Full-Forward Network) to obtain the prediction probability of the second character. Finally, the second feature vector is identified using a Correction Segmentation FFN (Correction Segmentation Full-Forward Network) to obtain multiple second-character prediction labels.
[0037] After obtaining the preset word segmentation labels, prediction probabilities, first prediction labels and second prediction labels, the multi-task model is trained using the preset word segmentation labels, prediction probabilities, first prediction labels and second prediction labels to obtain the trained multi-task recognition model.
[0038] Then, add a first preset identifier and a second preset identifier to the head of the input text respectively to obtain a first target text and a second target text, where the first preset identifier is used to indicate that the first target text contains typos, and the second preset identifier is used to indicate that the second target text does not contain typos. Then use the trained multi-task recognition model above to recognize the first target text and the second target text to obtain an initial word segmentation result.
[0039] It should be noted that the above word segmentation tags all adopt the BMES (Begin Middle End Single) method. B represents the first position value of a word in the input text, M represents the middle position of a word, E represents the end position of a word, and S represents a single word or character. For example, if the input text is "I am from Guangdong", after being labeled with BMES, the label of "I" is "S", the label of "am" is "S", the label of "Guang" is "B", the label of "dong" is "M", and the label of "person" is "E".
[0040] Step S106, select a target word segmentation tag from the first word segmentation tag and the second word segmentation tag based on the word segmentation probability.
[0041] Specifically, before selecting the target word segmentation tag, it is necessary to obtain a preset threshold. If the word segmentation probability is less than or equal to the preset threshold, determine the first word segmentation tag as the target word segmentation tag. If the word segmentation probability is greater than the preset threshold, determine the second word segmentation tag as the target word segmentation tag.
[0042] Step S108, determine the target word segmentation result of the input text through the target word segmentation tag.
[0043] Specifically, the target word segmentation tag can represent the division of the input text to obtain the target word segmentation result.
[0044] In the embodiment of the present invention, an input text is obtained, where the input text includes multiple input characters; the input text is segmented to obtain an initial word segmentation result of the input text, where the initial word segmentation result includes: the word segmentation probabilities of multiple input characters, a first word segmentation tag and a second word segmentation tag, the first word segmentation tag is used to represent the word segmentation tag of the input text, and the second word segmentation tag is used to represent the word segmentation tag obtained after correcting the typos included in the input text; based on the word segmentation probability, select a target word segmentation tag from the first word segmentation tag and the second word segmentation tag; determine the target word segmentation result of the input text through the target word segmentation tag. It is easy to notice that in this application, the first word segmentation and the second word segmentation are combined to segment the input text, rather than directly segmenting the input text, thus solving the technical problem that the recognition accuracy rate of the word segmentation model is relatively low due to the existence of typos in the training samples, and achieving the technical effect of improving the word segmentation accuracy rate.
[0045] Optionally, the input text is segmented to obtain the initial segmentation result, which includes: extracting features from the input text using the feature extraction module in the multi-task segmentation model to obtain target feature vectors for multiple input characters; segmenting the target feature vectors using the first segmentation model in the multi-task segmentation model to obtain the first segmentation label, wherein the first segmentation model is trained on a first dataset that does not contain misspelled characters; identifying the target feature vectors using the sub-task module in the multi-task segmentation model to obtain the segmentation probability of multiple input characters; and segmenting the target feature vectors using the second segmentation model in the multi-task segmentation model to obtain the second segmentation label, wherein the second segmentation model is trained on a second dataset that contains misspelled characters.
[0046] Specifically, the feature extraction module in the aforementioned multi-task word segmentation model can be a pre-trained language model, such as the BERT model or the Roberta model. The target feature vector includes a first feature vector and a second feature vector. The first feature vector is obtained by the feature extraction model extracting features from the first dataset, and the second feature vector is obtained by the feature extraction model extracting features from the second dataset. It should be noted that the first dataset is a dataset without typos, and the second dataset is a dataset obtained by replacing some characters in the first dataset with typos according to a preset probability. The first word segmentation model can be Segmentation FFN. The sub-task module can be Correction FFN. The second word segmentation model can be Correction Segmentation FFN. The word segmentation probability can be the probability of whether the input character is a correct token (text). Furthermore, the word segmentation probability needs to be normalized using Softmax before the loss function is calculated using the processed word segmentation probability.
[0047] Optionally, the method further includes: acquiring a first dataset, wherein the first dataset contains: multiple first characters and multiple preset word segmentation labels for the first characters; generating a second dataset based on the first dataset, wherein the second dataset contains: multiple second characters and preset word segmentation labels, the multiple second characters being generated by replacing target characters among the multiple first characters; performing feature extraction on the first dataset and the second dataset respectively using a feature extraction module to obtain first feature vectors for the multiple first characters and second feature vectors for the multiple second characters; performing word segmentation processing on the first feature vectors using a first word segmentation model to obtain first predicted labels for the multiple first characters; performing recognition on the second feature vectors using a subtask module to obtain predicted probabilities for the multiple second characters; performing word segmentation processing on the second feature vectors using a second word segmentation model to obtain second predicted labels for the multiple second characters; and training a multi-task word segmentation model based on the preset word segmentation labels, predicted probabilities, first predicted labels, and second predicted labels.
[0048] Specifically, the preset word segmentation labels for the multiple first characters mentioned above can be the word segmentation labels obtained after correctly processing the multiple first characters separately. It should be noted that these word segmentation labels can be BMES labels. According to a certain proportion, a portion of the first characters in the first dataset are selected as target characters. The target characters in the first dataset are replaced with homophones or similar-looking characters to obtain a new dataset, namely the aforementioned second dataset. Then, a feature extraction model is used to extract features from the first dataset and the second dataset respectively, obtaining first feature vectors for the multiple first characters and second feature vectors for the multiple second characters. The first word segmentation model is then used to process the first feature vectors to obtain the predicted probabilities for the multiple second characters. The first predicted label can be the label obtained by the first word segmentation model recognizing the first dataset, and the second predicted label can be the label obtained by the second word segmentation model recognizing the second dataset. The multi-task word segmentation model is trained based on the preset word segmentation labels, predicted probabilities, first predicted labels, and second predicted labels.
[0049] Optionally, generating a second dataset based on a first dataset includes: determining a target character according to a preset probability; replacing the target character to obtain a replaced character; generating a target dataset based on other characters in the first dataset besides the target character and the replaced character; and determining the target dataset as a second dataset in response to the difference between the target dataset and the first dataset.
[0050] Specifically, the first dataset includes multiple texts, each consisting of multiple first characters. For each text, at least one target character is selected according to a preset probability from these first characters, and the target character is replaced with a homophone or similar-looking character. Then, the replaced character is merged with all other characters in the first dataset except the target character to obtain the target dataset. It should be noted that if, after replacing the target character, the replaced character is still the same as the target character, the replaced character is not added to the target dataset, but the target character is added. Then, in response to the target dataset being different from the first dataset, the target dataset is determined to be the second dataset.
[0051] Optionally, a first preset identifier is added to the target location of the target dataset, wherein the first preset identifier is used to indicate that the target dataset contains typos.
[0052] Specifically, the target dataset includes multiple texts, and the target position can be the beginning of a sentence in the replaced text. When generating the target dataset, a first preset identifier is added to the beginning of the text containing the target word. The first identifier is essentially a special character used for identification; for example, the first identifier can be [C].
[0053] Optionally, training the multi-task word segmentation model based on preset word segmentation labels, prediction probabilities, a first prediction label, and a second prediction label includes: determining a first loss function based on the preset probabilities; determining a second loss function based on the preset word segmentation labels and the first preset label; determining a third loss function based on the preset word segmentation labels and the second preset label; obtaining the sum of the first loss function, the second loss function, and the third loss function to obtain a target loss function; and training the multi-task word segmentation model based on the target loss function.
[0054] Specifically, the first, second, and third loss functions mentioned above can all be cross-entropy loss functions. The first loss function is determined based on the word segmentation probability and a preset probability. The second loss function is determined by comparing the preset word segmentation labels with the first word segmentation labels. The third loss function is determined based on the preset word segmentation labels and the second preset label. Then, the sum of the first, second, and third loss functions is calculated to obtain the target loss function. The target loss function is then applied to the training of the multi-task word segmentation model.
[0055] Optionally, selecting a target word segmentation tag from the first word segmentation tag and the second word segmentation tag includes: determining the first word segmentation tag as the target word segmentation tag in response to a word segmentation probability less than or equal to a preset threshold; and determining the second word segmentation tag as the target word segmentation tag in response to a word segmentation probability greater than the preset threshold.
[0056] Specifically, the aforementioned preset threshold can be an error correction threshold. In one optional embodiment, the error correction threshold is 0.9. When the word segmentation probability is less than or equal to the preset threshold, the first word segmentation tag is determined as the target word segmentation tag. When the word segmentation probability is greater than the preset threshold, the second word segmentation tag is determined as the target word segmentation tag.
[0057] Optionally, the input text is segmented to obtain the initial segmentation result of the input text, which includes: adding a first preset identifier and a second preset identifier to the target position of the input text to obtain a first target text and a second target text, wherein the first preset identifier is used to indicate that the first target text contains typos and the second preset identifier is used to indicate that the second target text does not contain typos; and the first target text and the second target text are identified to obtain the initial segmentation result.
[0058] Specifically, in the Inference stage of the multi-task word segmentation model, Probability Fusion is performed. For the input text T, a first identifier and a second identifier are added to the front of the input text to obtain the first target text and the second target text. Then, the multi-task word segmentation model is used to identify the first target text and the second target text to obtain the initial word segmentation result.
[0059] In a preferred embodiment of the present invention, such as Figure 2 As shown, before training the multi-task recognition model, a training dataset is constructed. First, a first dataset D containing preset word segmentation labels is obtained. The text table in the first dataset is T, and each text T consists of several characters. Ti is used to represent the i-th character in text T, and the word segmentation label of Ti is L. First, according to a preset probability of 15%, Ti is replaced with its similar-looking characters. The replaced character is denoted as Ti', and a first preset identifier, namely the special character "[C]", is added before T. The text after replacement and the addition of the first preset identifier is denoted as T', and T' is added to dataset D'. It should be noted that if T' is the same as T, it is not added to dataset D'. Since the text in dataset D' all come from dataset D, the word segmentation labels of the text in dataset D' are the same as those of the text in dataset D.
[0060] In the training stage of the multi-task model, a pre-trained language model is used to extract features from the text in D and D', obtaining a first feature vector and a second feature vector. Then, the first feature vector is processed using Segmentation FFN and the normalization function Softmax to obtain a first preset label. The second feature vector is processed using Correction FFN and the normalization function Softmax to obtain a second preset label. The second feature vector is then processed using Correction FFN and the normalization function to obtain a prediction probability (Correction probability). The second feature vector is then processed using Correction segmentation FFN to obtain a second predicted label. Finally, in the inference stage, probability fusion is performed based on the normalized prediction probability, the Correction segmentation probability, and the Segmentation probability. It should be noted that the Correction segmentation probability represents the probability that the second predicted label is the same as the preset label. The probability is used to represent the probability that the first predicted tag is the same as the preset tag. In the process of probability fusion, the first segmentation tag and the second segmentation tag can be merged to determine the final target segmentation tag.
[0061] According to the embodiments of this application, an embodiment of a text processing device is also provided. The specific implementation methods in this embodiment are similar to or the same as those in the above embodiments, and will not be described again.
[0062] Figure 3 This is a schematic diagram of the structure of a text processing device according to an embodiment of this application, such as... Figure 3 As shown, the device includes:
[0063] The acquisition module 30 is used to acquire input text, wherein the input text includes multiple input characters.
[0064] The word segmentation module 32 is used to segment the input text to obtain the initial word segmentation result of the input text. The initial word segmentation result includes: the word segmentation probability of multiple input characters, the first word segmentation label and the second word segmentation label. The first word segmentation label is used to represent the word segmentation label of the input text, and the second word segmentation label is used to represent the word segmentation label obtained after correcting the typos contained in the input text.
[0065] The selection module 34 is used to select the target word segmentation tag from the first word segmentation tag and the second word segmentation tag based on the word segmentation probability.
[0066] Processing module 36 is used to input the target word segmentation result of the text through the target word segmentation label.
[0067] In this embodiment of the invention, input text is obtained, wherein the input text includes multiple input characters; the input text is segmented into words to obtain an initial segmentation result, wherein the initial segmentation result includes: segmentation probabilities of multiple input characters, a first segmentation label, and a second segmentation label. The first segmentation label is used to characterize the segmentation label of the input text, and the second segmentation label is used to characterize the segmentation label obtained after correcting typos contained in the input text; based on the segmentation probabilities, a target segmentation label is selected from the first and second segmentation labels; the target segmentation result of the input text is determined by the target segmentation label. It is noteworthy that this application combines the first and second segmentation methods to segment the input text, rather than directly segmenting the input text, thereby solving the technical problem of low accuracy in word segmentation model recognition caused by typos in the training samples, and achieving the technical effect of improving the accuracy of word segmentation.
[0068] Optionally, the word segmentation module includes: a feature extraction unit, used to extract features from the input text using the feature extraction module in the multi-task word segmentation model to obtain target feature vectors for multiple input characters; a first label unit, used to segment the target feature vectors using the first word segmentation model in the multi-task word segmentation model to obtain first word segmentation labels, wherein the first word segmentation model is trained using a first dataset that does not contain misspelled characters; a probability unit, used to identify the target feature vectors using the sub-task module in the multi-task word segmentation model to obtain the word segmentation probabilities for multiple input characters; and a second label unit, used to segment the target feature vectors using the second word segmentation model in the multi-task word segmentation model to obtain second word segmentation labels, wherein the second word segmentation model is trained using a second dataset that contains misspelled characters.
[0069] Optionally, the device further includes: a dataset acquisition module for acquiring a first dataset, wherein the first dataset contains: multiple first characters and multiple preset word segmentation labels for the first characters; a dataset generation module for generating a second dataset based on the first dataset, wherein the second dataset contains: multiple second characters and preset word segmentation labels, the multiple second characters being generated by replacing target characters among the multiple first characters; a feature extraction module for performing feature extraction on the first dataset and the second dataset respectively using the feature extraction module to obtain first feature vectors for the multiple first characters and second feature vectors for the multiple second characters; a first label module for performing word segmentation processing on the first feature vectors using a first word segmentation model to obtain first predicted labels for the multiple first characters; a first probability module for recognizing the second feature vectors using a subtask module to obtain predicted probabilities for the multiple second characters; a second label module for performing word segmentation processing on the second feature vectors using a second word segmentation model to obtain second predicted labels for the multiple second characters; and a training module for training a multi-task word segmentation model based on preset word segmentation labels, predicted probabilities, first predicted labels, and second predicted labels.
[0070] Optionally, the dataset generation module includes: a target character module for determining a target character according to a preset probability; a replacement module for replacing the target character to obtain a replaced character; a target generation unit for generating a target dataset based on other characters in the first dataset besides the target character and the replaced character; and a determination unit for determining the target dataset as a second dataset in response to the target dataset being different from the first dataset.
[0071] Optionally, the device is also used to add a first preset identifier to the target location of the target dataset, wherein the first preset identifier is used to indicate that the target dataset contains typos.
[0072] Optionally, the training module includes: a first function unit for determining a first loss function based on a preset probability; a second function unit for determining a second loss function based on preset word segmentation labels and a first preset label; a third function unit for determining a third loss function based on preset word segmentation labels and a second preset label; a target function unit for obtaining the sum of the first, second, and third loss functions to obtain a target loss function; and a model training unit for training a multi-task word segmentation model based on the target loss function.
[0073] Optionally, the selection module includes: a first response unit, used to determine the first segmentation tag as the target segmentation tag in response to a segmentation probability less than or equal to a preset threshold; and a second response unit, used to determine the second segmentation tag as the target segmentation tag in response to a segmentation probability greater than the preset threshold.
[0074] Optionally, the word segmentation module includes: an identifier unit, used to add a first preset identifier and a second preset identifier to the target position of the input text respectively to obtain a first target text and a second target text, wherein the first preset identifier is used to indicate that the first target text contains typos, and the second preset identifier is used to indicate that the second target text does not contain typos; and an identification unit, used to identify the first target text and the second target text to obtain an initial word segmentation result.
[0075] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0076] This application also provides a computer storage medium, which includes a stored program, wherein when the program is executed, it controls the device where the computer-readable storage medium is located to perform any one of the above-described text processing methods.
[0077] According to another aspect of the present invention, a processor is also provided for running a program, wherein the program executes any one of the above-described text processing methods during runtime.
[0078] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0079] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.
[0080] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0081] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0082] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0083] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A text processing method characterized by, include: Obtain input text, wherein the input text includes multiple input characters; The input text is segmented into words to obtain an initial segmentation result of the input text. The initial segmentation result includes: the segmentation probability of the multiple input characters, a first segmentation label, and a second segmentation label. The first segmentation label is used to characterize the segmentation label of the input text, and the second segmentation label is used to characterize the segmentation label obtained after correcting the typos contained in the input text. Based on the word segmentation probability, a target word segmentation tag is selected from the first word segmentation tag and the second word segmentation tag; The target word segmentation result of the input text is determined by the target word segmentation label; The method further includes: obtaining a first dataset, wherein the first dataset contains: a plurality of first characters and preset word segmentation tags for the plurality of first characters; generating a second dataset based on the first dataset, wherein the second dataset contains: a plurality of second characters and the preset word segmentation tags, wherein the plurality of second characters are generated by replacing target characters among the plurality of first characters; The initial word segmentation result of the input text includes: extracting features from the input text using the feature extraction module in the multi-task word segmentation model to obtain target feature vectors for the multiple input characters; segmenting the target feature vectors using the first word segmentation model in the multi-task word segmentation model to obtain the first word segmentation label, wherein the first word segmentation model is trained on the first dataset without typos; identifying the target feature vectors using the sub-task module in the multi-task word segmentation model to obtain the word segmentation probability of the multiple input characters; and segmenting the target feature vectors using the second word segmentation model in the multi-task word segmentation model to obtain the second word segmentation label, wherein the second word segmentation model is trained on the second dataset containing typos.
2. The method of claim 1, wherein, The method further includes: The feature extraction module is used to extract features from the first dataset and the second dataset respectively, to obtain the first feature vector of the plurality of first characters and the second feature vector of the plurality of second characters; The first feature vector is segmented using the first word segmentation model to obtain the first predicted label of the plurality of first characters; The subtask module is used to identify the second feature vector to obtain the predicted probabilities of the multiple second characters; The second feature vector is segmented using the second word segmentation model to obtain the second predicted labels of the multiple second characters; The multi-task word segmentation model is trained based on the preset word segmentation labels, the prediction probabilities, the first prediction labels, and the second prediction labels.
3. The method of claim 2, wherein, Generating the second dataset based on the first dataset includes: The target word is determined according to a preset probability; The target character is replaced to obtain the replaced character; A target dataset is generated based on the other characters in the first dataset besides the target character and the replaced character. In response to the fact that the target dataset is different from the first dataset, the target dataset is determined to be the second dataset.
4. The method of claim 3, wherein, A first preset identifier is added to the target location of the target dataset, wherein the first preset identifier is used to indicate that the target dataset contains typos.
5. The method of claim 2, wherein, Training the multi-task word segmentation model based on the preset word segmentation labels, the predicted probabilities, the first predicted label, and the second predicted label includes: The first loss function is determined based on a preset probability. Based on the preset word segmentation tags and the first preset tags, a second loss function is determined; Based on the preset word segmentation tags and the second preset tags, a third loss function is determined; The target loss function is obtained by summing the first loss function, the second loss function, and the third loss function. The multi-task word segmentation model is trained based on the target loss function.
6. The method of claim 1, wherein, Based on the word segmentation probability, selecting the target word segmentation tag from the first word segmentation tag and the second word segmentation tag includes: In response to the word segmentation probability being less than or equal to a preset threshold, the first word segmentation tag is determined to be the target word segmentation tag; In response to the word segmentation probability being greater than the preset threshold, the second word segmentation tag is determined to be the target word segmentation tag.
7. The method of claim 1, wherein, The initial word segmentation result of the input text is obtained by performing word segmentation on the input text, including: A first preset identifier and a second preset identifier are added to the target positions of the input text to obtain a first target text and a second target text, wherein the first preset identifier is used to indicate that the first target text contains typos, and the second preset identifier is used to indicate that the second target text does not contain typos; The first target text and the second target text are identified to obtain the initial word segmentation result.
8. A text processing apparatus characterized by comprising: include: An acquisition module is used to acquire input text, wherein the input text includes multiple input characters; The word segmentation module is used to segment the input text to obtain an initial word segmentation result of the input text. The initial word segmentation result includes: the word segmentation probability of the multiple input characters, a first word segmentation label and a second word segmentation label. The first word segmentation label is used to characterize the word segmentation label of the input text, and the second word segmentation label is used to characterize the word segmentation label obtained after correcting the typos contained in the input text. The selection module is used to select a target word segmentation tag from the first word segmentation tag and the second word segmentation tag based on the word segmentation probability; The processing module is used to obtain the target word segmentation result of the input text through the target word segmentation label; The device further includes: a dataset acquisition module, configured to acquire a first dataset, wherein the first dataset contains: a plurality of first characters and preset word segmentation tags for the plurality of first characters; and a dataset generation module, configured to generate a second dataset based on the first dataset, wherein the second dataset contains: a plurality of second characters and the preset word segmentation tags, wherein the plurality of second characters are generated by replacing target characters among the plurality of first characters; The word segmentation module includes: a feature extraction unit, used to extract features from the input text using the feature extraction module in the multi-task word segmentation model to obtain target feature vectors of the multiple input characters; a first label unit, used to segment the target feature vectors using the first word segmentation model in the multi-task word segmentation model to obtain the first word segmentation label, wherein the first word segmentation model is trained using the first dataset without misspellings; a probability unit, used to identify the target feature vectors using the sub-task module in the multi-task word segmentation model to obtain the word segmentation probability of the multiple input characters; and a second label unit, used to segment the target feature vectors using the second word segmentation model in the multi-task word segmentation model to obtain the second word segmentation label, wherein the second word segmentation model is trained using the second dataset containing misspellings.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the text processing method according to any one of claims 1 to 7.
10. A processor, comprising: The processor is used to run a program, wherein the program executes the text processing method according to any one of claims 1 to 7 when it runs.