Natural language processing method, device, text processing method, equipment and medium

Abstract

The embodiments of the present disclosure disclose a natural language processing method, an apparatus, a text processing method, a device and a medium. The natural language processing method includes: obtaining labeled source domain data and unlabeled target domain data; using the labeled source domain data to machine-label the unlabeled target domain data; inputting the labeled source domain data and the machine-labeled target domain data into an adversarial network to obtain optimized target domain data. Based on the labeled source domain data and the machine-labeled target domain data, the adversarial network can be used to optimize the target domain data to improve the accuracy of word segmentation of the target domain data.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a natural language processing method, apparatus, text processing method, equipment and medium. Background Art

[0002] In Chinese, there are no explicit separators between characters. Therefore, if you want the machine to obtain information about word units, you must re-normalize the character sequence to form a word sequence. Today, word segmentation has become an essential step in natural language processing tasks. For example, if you want to correctly identify named entities such as "AAA City" and "BBB District" in the address "AAA City BBB District CCC Road", correct word segmentation is a necessary preprocessing step.

[0003] At present, deep learning models can achieve good results in supervised word segmentation in specific fields. However, the inventors have found that the performance of these excellent algorithms will drop sharply in the word segmentation effect in cross-domain scenarios, exposing major problems. This is mainly because each field has specific words, and the manually annotated corpus is often very limited and it is impossible to cover every field. The lack of such annotated corpus will cause the model to be unable to recognize words that have not been logged in during the training process. For example, a model trained on news corpus will find it difficult to recognize the word "streptococcus" because the latter often only appears in medical corpus. The cross-domain setting will greatly weaken the model performance under the supervised corpus. Summary of the invention

[0004] In order to solve the problems in the related art, the embodiments of the present disclosure provide a natural language processing method, an apparatus, a text processing method, a device and a medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a natural language processing method, including:

[0006] Obtain labeled source domain data and unlabeled target domain data;

[0007] Using the labeled source domain data, machine-labeling the unlabeled target domain data;

[0008] The labeled source domain data and the machine-labeled target domain data are input into the adversarial network to obtain optimized target domain data.

[0009] In combination with the first aspect, in a first implementation of the first aspect of the present disclosure, the step of using the labeled source domain data to machine-label the unlabeled target domain data includes:

[0010] Matching the annotated source domain data with the unannotated target domain data to obtain source domain specific words, shared words, and target domain data to be annotated;

[0011] Obtaining a shared word vocabulary using the shared words;

[0012] Acquire a target domain-specific dictionary using the target domain data to be annotated;

[0013] According to the shared word vocabulary and the target domain-specific dictionary, machine-annotated target domain data is obtained.

[0014] In combination with the first implementation of the first aspect, in a second implementation of the first aspect of the present disclosure, the step of obtaining a target domain-specific dictionary by using the target domain data to be annotated includes:

[0015] Identify words in the target domain data to be annotated by calculating one or more of the following indicators:

[0016] cohesion score, freedom score, word frequency, and word frequency-inverse text frequency index.

[0017] In combination with the second implementation of the first aspect, in a third implementation of the first aspect of the present disclosure, the coagulation score is calculated in the following manner, including:

[0018] Acquire a first text segment from the target domain data to be annotated;

[0019] Obtaining a first probability that the first text segment as a whole appears in the target domain data to be annotated;

[0020] Segment the first text segment into two sub-segments, and determine a first segmentation method that maximizes the product of probabilities of the two sub-segments appearing in the target domain data to be annotated; and

[0021] The target domain-specific dictionary is determined based on a ratio of the first probability and a product corresponding to the first segmentation method.

[0022] In combination with the second implementation of the first aspect, in a fourth implementation of the first aspect of the present disclosure, the free score is calculated in the following manner, including:

[0023] Acquire a second text segment from the target domain data to be annotated;

[0024] Determine the left neighbor character set entropy and the right neighbor character set entropy of the second text segment;

[0025] The target domain-specific dictionary is determined based on the smaller of the left neighbor word set entropy and the right neighbor word set entropy.

[0026] In combination with the first implementation of the first aspect, in a fifth implementation of the first aspect of the present disclosure, the step of obtaining a target domain-specific dictionary by using the target domain data to be annotated includes:

[0027] The target domain data to be annotated is processed by a predetermined word segmentation model to obtain a target domain-specific dictionary.

[0028] In combination with the first aspect and any one of the first to fifth implementations of the first aspect, in a sixth implementation of the first aspect of the present disclosure, the adversarial network includes a shared encoder, a discriminator, and a target domain word segmenter, and the inputting the annotated source domain data and the machine-annotated target domain data into the adversarial network to obtain the optimized target domain data includes:

[0029] Encoding a third text segment in the labeled source domain data to obtain a first latent vector;

[0030] Encoding a fourth text segment in the machine-annotated target domain data to obtain a second latent vector;

[0031] The shared encoder randomly selects the first latent vector or the second latent vector as the first code and inputs it into the discriminator, so that the discriminator determines that the first code is the first latent vector or the second latent vector;

[0032] The target domain word segmenter obtains optimized target domain data according to the result of the discriminator.

[0033] In combination with the sixth implementation of the first aspect, in a seventh implementation of the first aspect of the present disclosure, the target domain word segmenter includes a target domain encoder and a first word segmenter, and the step of obtaining optimized target domain data according to the result of the discriminator includes:

[0034] Obtaining a second code through the target domain encoder, wherein the second code is obtained according to the second latent vector;

[0035] Based on the first encoding and the second encoding, an optimized word segmentation result of the fourth text segment is obtained through a first word segmenter.

[0036] In combination with the sixth implementation manner of the first aspect, in an eighth implementation manner of the first aspect of the present disclosure, the adversarial network further includes a source domain encoder and a second word segmenter, and the method further includes:

[0037] The source domain encoder obtains a third code, wherein the third code is obtained according to the first latent vector;

[0038] Based on the first code and the third code, an optimized word segmentation result of the third text segment is obtained through a second word segmenter.

[0039] In a second aspect, an embodiment of the present disclosure provides a natural language processing device, including:

[0040] An acquisition module, configured to acquire labeled source domain data and unlabeled target domain data;

[0041] a labeling module configured to perform machine labeling on the unlabeled target domain data using the labeled source domain data;

[0042] The input module is configured to input the labeled source domain data and the machine-labeled target domain data into the adversarial network to obtain optimized target domain data.

[0043] In combination with the second aspect, in a first implementation of the second aspect of the present disclosure, the annotation module includes:

[0044] A matching submodule is configured to match the annotated source domain data with the unannotated target domain data to obtain source domain specific words, shared words, and target domain data to be annotated;

[0045] A shared word vocabulary acquisition submodule, configured to acquire a shared word vocabulary using the shared word;

[0046] A target domain specific dictionary word acquisition submodule is configured to acquire a target domain specific dictionary using the target domain data to be annotated;

[0047] The target domain data acquisition submodule is configured to acquire machine-annotated target domain data according to the shared word vocabulary and the target domain-specific dictionary.

[0048] In combination with the first implementation of the second aspect, in a second implementation of the second aspect of the present disclosure, the specific dictionary word list acquisition submodule is further configured as follows:

[0049] Identify words in the target domain data to be annotated by calculating one or more of the following indicators:

[0050] cohesion score, freedom score, word frequency, and word frequency-inverse text frequency index.

[0051] In combination with the second implementation of the second aspect, in a third implementation of the second aspect of the present disclosure, the coagulation score is calculated in the following manner, including:

[0052] Acquire a first text segment from the target domain data to be annotated;

[0053] Obtaining a first probability that the first text segment as a whole appears in the target domain data to be annotated;

[0054] Segment the first text segment into two sub-segments, and determine a first segmentation method that maximizes the product of probabilities of the two sub-segments appearing in the target domain data to be annotated; and

[0055] The target domain-specific dictionary is determined based on a ratio of the first probability and a product corresponding to the first segmentation method.

[0056] In combination with the second implementation of the second aspect, in a fourth implementation of the second aspect of the present disclosure, the free score is calculated in the following manner, including:

[0057] Acquire a second text segment from the target domain data to be annotated;

[0058] Determine the left neighbor character set entropy and the right neighbor character set entropy of the second text segment;

[0059] The target domain-specific dictionary is determined based on the smaller of the left neighbor word set entropy and the right neighbor word set entropy.

[0060] In combination with the first implementation of the second aspect, in a fifth implementation of the second aspect of the present disclosure, the target domain-specific dictionary word acquisition submodule is further configured as follows:

[0061] The target domain data to be annotated is processed by a predetermined word segmentation model to obtain a target domain-specific dictionary.

[0062] In combination with the second aspect and any one of the first to fifth implementations of the second aspect, in a sixth implementation of the second aspect of the present disclosure, the adversarial network includes a shared encoder, a discriminator, and a target domain word segmenter, and the input module includes:

[0063] a first encoding submodule, configured to encode a third text segment in the annotated source domain data to obtain a first latent vector;

[0064] The second encoding submodule is configured to encode a fourth text segment in the target domain data annotated by the machine to obtain a second latent vector,

[0065] The shared encoder randomly selects the first latent vector or the second latent vector as the first code and inputs it into the discriminator, so that the discriminator determines whether the first code is the first latent vector or the second latent vector;

[0066] The target domain word segmenter obtains optimized target domain data according to the result of the discriminator.

[0067] In combination with the sixth implementation manner of the second aspect, in a seventh implementation manner of the second aspect of the present disclosure, the target domain word segmenter includes a target domain encoder and a first word segmenter, and the target domain word segmenter is configured as follows:

[0068] Obtaining a second code through the target domain encoder, wherein the second code is obtained according to the second latent vector;

[0069] Based on the first encoding and the second encoding, an optimized word segmentation result of the fourth text segment is obtained through a first word segmenter.

[0070] In combination with the sixth implementation manner of the second aspect, in an eighth implementation manner of the second aspect of the present disclosure, the adversarial network further includes a source domain encoder and a second word segmenter, wherein:

[0071] The source domain encoder obtains a third code, wherein the third code is obtained according to the first latent vector;

[0072] Based on the first encoding and the third encoding, the second word segmenter obtains an optimized word segmentation result of the third text segment.

[0073] In a third aspect, an embodiment of the present disclosure provides a text processing method, including:

[0074] Acquire annotated first text data and unannotated second text data;

[0075] Using the annotated first text data, machine annotating the unannotated second text data;

[0076] Inputting the annotated first text data and the machine-annotated second text data into an adversarial network to obtain optimized second text data;

[0077] Based on the optimized second text data, an optimized word segmentation result of the second text data is output.

[0078] In a fourth aspect, an electronic device is provided in an embodiment of the present disclosure, including a memory and a processor; wherein:

[0079] The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method as described in any one of the first aspect, the first to eighth implementations of the first aspect, or the third aspect.

[0080] In a fifth aspect, a readable storage medium is provided in an embodiment of the present disclosure, on which computer instructions are stored. When the computer instructions are executed by a processor, a method as described in the first aspect, the first to eighth implementations of the first aspect, or any one of the methods described in the third aspect is implemented.

[0081] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0082] According to the technical solution provided by the embodiments of the present disclosure, by obtaining labeled source domain data and unlabeled target domain data; using the labeled source domain data to machine-label the unlabeled target domain data; inputting the labeled source domain data and the machine-labeled target domain data into an adversarial network to obtain optimized target domain data, the adversarial network can be used to optimize the target domain data based on the labeled source domain data and the machine-labeled target domain data, thereby improving the accuracy of word segmentation of the target domain data.

[0083] According to the technical solution provided by the embodiment of the present disclosure, the unlabeled target domain data is machine-labeled by utilizing the labeled source domain data, including: matching the labeled source domain data with the unlabeled target domain data to obtain source domain-specific words, shared words, and target domain data to be labeled; utilizing the shared words to obtain a shared word vocabulary; utilizing the target domain data to be labeled to obtain a target domain-specific dictionary; and obtaining machine-labeled target domain data based on the shared word vocabulary and the target domain-specific dictionary. Machine-labeled target domain data can be obtained for target domain data that has not been manually labeled, thereby improving the accuracy of word segmentation of target domain data.

[0084] According to the technical solution provided by the embodiment of the present disclosure, a target domain-specific dictionary is obtained by utilizing the target domain data to be annotated, including: identifying words in the target domain data to be annotated by calculating one or more of the following indicators: cohesion score, free score, word frequency, and word frequency-inverse text frequency index, so that machine-annotated target domain data without manual annotating can be obtained, thereby improving the accuracy of word segmentation of target domain data.

[0085] According to the technical solution provided by the embodiment of the present disclosure, the cohesion score is calculated in the following manner, including: obtaining a first text segment from the target domain data to be annotated; obtaining a first probability that the first text segment appears as a whole in the target domain data to be annotated; dividing the first text segment into two sub-segments, and determining a first segmentation method that maximizes the product of the probabilities of the two sub-segments appearing in the target domain data to be annotated; and determining the target domain-specific dictionary based on the ratio of the first probability to the product corresponding to the first segmentation method. This can obtain machine-annotated target domain data for target domain data that is not manually annotated, thereby improving the accuracy of word segmentation of the target domain data.

[0086] According to the technical solution provided by the embodiment of the present disclosure, the free score is calculated in the following manner, including: obtaining a second text segment from the target domain data to be annotated; determining the left neighbor character set entropy and the right neighbor character set entropy of the second text segment; determining the target domain-specific dictionary based on the smaller of the left neighbor character set entropy and the right neighbor character set entropy. Based on the annotated source domain data and the machine-annotated target domain data, an adversarial network can be used to optimize the target domain data to improve the accuracy of word segmentation of the target domain data.

[0087] According to the technical solution provided by the embodiment of the present disclosure, a target domain-specific dictionary is obtained by utilizing the target domain data to be annotated, including: processing the target domain data to be annotated by a predetermined word segmentation model to obtain a target domain-specific dictionary, and based on the annotated source domain data and the machine-annotated target domain data, an adversarial network is used to optimize the target domain data to improve the accuracy of word segmentation of the target domain data.

[0088] According to the technical solution provided by the embodiment of the present disclosure, the adversarial network includes a shared encoder, a discriminator and a target domain word segmenter, and the annotated source domain data and the machine-annotated target domain data are input into the adversarial network to obtain the optimized target domain data, including: encoding the third text segment in the annotated source domain data to obtain a first latent vector; encoding the fourth text segment in the machine-annotated target domain data to obtain a second latent vector; the shared encoder randomly selects the first latent vector or the second latent vector as the first code and inputs it into the discriminator, so that the discriminator determines that the first code is the first latent vector or the second latent vector; the target domain word segmenter obtains the optimized target domain data according to the result of the discriminator, and can optimize the target domain data based on the annotated source domain data and the machine-annotated target domain data using the adversarial network to improve the accuracy of the target domain data segmentation.

[0089] According to the technical solution provided by the embodiment of the present disclosure, the target domain word segmenter includes a target domain encoder and a first word segmenter, and the optimized target domain data is obtained according to the result of the discriminator, including: obtaining a second code through the target domain encoder, and the second code is obtained according to the second implicit vector; based on the first code and the second code, the optimized word segmentation result of the fourth text fragment is obtained through the first word segmenter. Based on the labeled source domain data and the machine-labeled target domain data, the adversarial network can be used to optimize the target domain data to improve the accuracy of the target domain data segmentation.

[0090] According to the technical solution provided by the embodiment of the present disclosure, the adversarial network also includes a source domain encoder and a second word segmenter, and the method also includes: the source domain encoder obtains a third code, and the third code is obtained according to the first implicit vector; based on the first code and the third code, the optimized word segmentation result of the third text fragment is obtained by the second word segmenter. Based on the annotated source domain data and the machine-annotated target domain data, the adversarial network can be used to optimize the target domain data to improve the accuracy of the source domain data segmentation.

[0091] According to the technical solution provided by the embodiment of the present disclosure, an acquisition module is configured to acquire labeled source domain data and unlabeled target domain data; a labeling module is configured to use the labeled source domain data to machine label the unlabeled target domain data; an input module is configured to input the labeled source domain data and the machine-labeled target domain data into an adversarial network to acquire optimized target domain data. Based on the labeled source domain data and the machine-labeled target domain data, the adversarial network can be used to optimize the target domain data, thereby improving the accuracy of word segmentation of the target domain data.

[0092] According to the technical solution provided by the embodiment of the present disclosure, the annotation module includes: a matching submodule, configured to match the annotated source domain data with the unannotated target domain data, and obtain source domain specific words, shared words, and target domain data to be annotated; a shared word vocabulary acquisition submodule, configured to use the shared words to obtain a shared word vocabulary; a target domain-specific dictionary word acquisition submodule, configured to use the target domain data to be annotated to obtain a target domain-specific dictionary; a target domain data acquisition submodule, configured to obtain machine-annotated target domain data based on the shared word vocabulary and the target domain-specific dictionary, which can obtain machine-annotated target domain data for target domain data without manual annotation, thereby improving the accuracy of word segmentation of target domain data.

[0093] According to the technical solution provided by the embodiment of the present disclosure, the submodule for acquiring the unique dictionary word list is also configured to: identify the words in the target domain data to be annotated by calculating one or more of the following indicators: cohesion score, free score, word frequency and word frequency-inverse text frequency index, so as to obtain machine-annotated target domain data without manual annotating, thereby improving the accuracy of word segmentation of the target domain data.

[0094] According to the technical solution provided by the embodiment of the present disclosure, the cohesion score is calculated in the following manner, including: obtaining a first text segment from the target domain data to be annotated; obtaining a first probability that the first text segment appears as a whole in the target domain data to be annotated; dividing the first text segment into two sub-segments, and determining a first segmentation method that maximizes the product of the probabilities of the two sub-segments appearing in the target domain data to be annotated; and determining the target domain-specific dictionary based on the ratio of the first probability to the product corresponding to the first segmentation method. This can obtain machine-annotated target domain data for target domain data that is not manually annotated, thereby improving the accuracy of word segmentation of the target domain data.

[0095] According to the technical solution provided by the embodiment of the present disclosure, the free score is calculated in the following manner, including: obtaining a second text segment from the target domain data to be annotated; determining the left neighbor character set entropy and the right neighbor character set entropy of the second text segment; determining the target domain-specific dictionary based on the smaller of the left neighbor character set entropy and the right neighbor character set entropy. Based on the annotated source domain data and the machine-annotated target domain data, an adversarial network can be used to optimize the target domain data to improve the accuracy of word segmentation of the target domain data.

[0096] According to the technical solution provided by the embodiment of the present disclosure, the target domain-specific dictionary word acquisition submodule is also configured to: process the target domain data to be annotated through a predetermined word segmentation model to obtain a target domain-specific dictionary, and can optimize the target domain data based on the annotated source domain data and the machine-annotated target domain data using an adversarial network to improve the accuracy of the target domain data segmentation.

[0097] According to the technical solution provided by the embodiment of the present disclosure, the adversarial network includes a shared encoder, a discriminator and a target domain word segmenter, and the input module includes: a first encoding submodule, configured to encode the third text segment in the labeled source domain data to obtain a first latent vector; a second encoding submodule, configured to encode the fourth text segment in the machine-labeled target domain data to obtain a second latent vector, wherein the shared encoder randomly selects the first latent vector or the second latent vector as the first code input to the discriminator, so that the discriminator determines that the first code is the first latent vector or the second latent vector; wherein the target domain word segmenter obtains the optimized target domain data according to the result of the discriminator, and can optimize the target domain data based on the labeled source domain data and the machine-labeled target domain data by using the adversarial network to improve the accuracy of the target domain data segmentation.

[0098] According to the technical solution provided by the embodiment of the present disclosure, the target domain word segmenter includes a target domain encoder and a first word segmenter, and the target domain word segmenter is configured to: obtain a second code through the target domain encoder, and the second code is obtained according to the second implicit vector; based on the first code and the second code, obtain the optimized word segmentation result of the fourth text fragment through the first word segmenter, and can optimize the target domain data based on the annotated source domain data and the machine-annotated target domain data by using an adversarial network to improve the accuracy of the target domain data segmentation.

[0099] According to the technical solution provided by the embodiment of the present disclosure, the adversarial network also includes a source domain encoder and a second word segmenter, wherein: the source domain encoder obtains a third code, and the third code is obtained according to the first implicit vector; based on the first code and the third code, the second word segmenter obtains the optimized word segmentation result of the third text fragment, and based on the annotated source domain data and the machine-annotated target domain data, the adversarial network can be used to optimize the target domain data to improve the accuracy of the source domain data segmentation.

[0100] According to the technical solution provided by the embodiment of the present disclosure, by obtaining annotated first text data and unannotated second text data; using the annotated first text data, machine annotating the unannotated second text data; inputting the annotated first text data and the machine-annotated second text data into an adversarial network to obtain optimized second text data; based on the optimized second text data, outputting the optimized word segmentation result of the second text data, the adversarial network can be used to optimize the word segmentation result of the second text data based on the annotated first text data and the machine-annotated second text data, thereby improving the accuracy of the word segmentation of the second text data.

[0101] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0103] Figure 1 A flowchart showing a natural language processing method according to an embodiment of the present disclosure is shown;

[0104] Figure 2 A schematic diagram showing a data processing flow in a natural language processing method according to an embodiment of the present disclosure;

[0105] Figure 3A schematic diagram showing a method of obtaining machine-annotated target domain data in a natural language processing method according to an embodiment of the present disclosure;

[0106] Figure 4 A schematic diagram showing an adversarial network according to an embodiment of the present disclosure;

[0107] Figure 5 A structural block diagram of a natural language processing device according to an embodiment of the present disclosure is shown;

[0108] Figure 6 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0109] Figure 7 It is a structural diagram of a computer system suitable for implementing the natural language processing method or text processing method according to the embodiments of the present disclosure. DETAILED DESCRIPTION

[0110] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0111] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of labels, numbers, steps, behaviors, components, parts, or combinations thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other labels, numbers, steps, behaviors, components, parts, or combinations thereof exist or are added.

[0112] It should also be noted that, in the absence of conflict, the embodiments and labels in the embodiments of the present disclosure can be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0113] According to the technical solution provided by the embodiment of the present disclosure, by obtaining labeled source domain data and unlabeled target domain data; using the labeled source domain data, machine-labeling the unlabeled target domain data; inputting the labeled source domain data and machine-labeled target domain data into an adversarial network to obtain optimized target domain data, the adversarial network can be used to optimize the target domain data based on the labeled source domain data and machine-labeled target domain data to improve the accuracy of target domain data segmentation. No supervisory information on the target domain (such as domain dictionaries, labeling information, etc.) is required, and the adversarial network is used to reduce the noise of the machine-automatically labeled data, which effectively utilizes the semantic information of the source and target domains, reduces the impact of insufficient dictionaries, and greatly improves the scalability of the model in new domains.

[0114] Figure 1FIG. 1 is a flow chart of a natural language processing method according to an embodiment of the present disclosure. Figure 1 As shown, the natural language processing method includes the following steps S110, S120 and S130:

[0115] In step S110 , the labeled source domain data and the unlabeled target domain data are obtained.

[0116] In step S120, the unlabeled target domain data is machine labeled using the labeled source domain data.

[0117] In step S130, the labeled source domain data and the machine-labeled target domain data are input into the adversarial network to obtain optimized target domain data.

[0118] In one embodiment of the present disclosure, the source domain data and the target domain data belong to natural language and can be data in text form. In one embodiment of the present disclosure, the annotated source domain data includes annotation information for the source domain data. In one embodiment of the present disclosure, the optimized target domain data can include annotation information for the target domain data. For example, annotation can refer to labeling each character, and the label indicates its specific position in the word. For example, B, M, and E can be used to represent the beginning, middle, and end of a word respectively, and S can be used to represent that the single character is an independent word.

[0119] In one embodiment of the present disclosure, the source domain and the target domain are two different domains. There may be some domain-independent shared words between the two, such as "is", "research", "discovery", etc. There are also some domain-specific words, words that are common in the source domain are very rare or even non-existent in the target domain, and words that are common in the target domain are very rare or even non-existent in the source domain. For example, the source domain may be the news domain, and the target domain may be the medical domain, and the common words in the two domains are very different. Therefore, the model trained for the source domain text data in the prior art is difficult to apply to the target domain text data.

[0120] In one embodiment of the present disclosure, the annotated source domain text data is text data with annotations. For example, the annotation information corresponding to the annotated text "the worst economic recession since the Great Depression" may be "BMEBEBEBEBE", and the annotation information is the word segmentation result of the annotated source domain text data. The unannotated target domain data is text data to be segmented, without such a mark.

[0121] Figure 2 A schematic diagram showing the data processing flow in a natural language processing method according to an embodiment of the present disclosure.

[0122] like Figure 2As shown, in one embodiment of the present disclosure, the natural language processing method uses two data sets, one data set includes annotated (with artificial labels) source domain text data, and the annotated source domain text data includes source domain text data and word segmentation results of the source domain text data. The other data set includes unannotated (without labels) target domain text data. The goal of model training is to perform word segmentation on the target domain data. First, the source domain text data and the target domain text data are configured to execute step S120 described with reference to the context through a processing module, and construct the word segmentation results of the target domain text sequence automatically annotated by the machine using the source domain knowledge and statistical information. Then, the annotated source domain text data and the machine-annotated (with machine labels) target domain text data will be jointly passed through an adversarial network for noise reduction and information reuse to obtain the optimized target domain text data, that is, the target domain text data optimizes the word segmentation results. The obtained optimized word segmentation results can be output in various ways, for example, the word segmentation labels corresponding to the characters in the target domain text data can be directly output, or delimiters can be added to the target domain text data according to the obtained optimized word segmentation results, and the target domain text data with delimiters can be output. In this whole process, no manually set dictionary or annotation information of the target domain is required.

[0123] Figure 3 A schematic diagram showing a method for obtaining machine-annotated target domain data in a natural language processing method according to an embodiment of the present disclosure.

[0124] like Figure 3 As shown, the labeled source domain data and the unlabeled target domain data are matched to obtain source domain specific words, shared words, and target domain data to be labeled. Figure 3 What is shown in the figure are target domain specific words, but this is to illustrate that the unannotated target domain data in the matching results other than the shared words are target domain specific words. However, the target domain specific words are not actually annotated at this time, so this target domain specific word refers to the target domain data to be annotated other than the shared words. Mining shared words can obtain a shared word vocabulary. Mining new words on the target domain data to be annotated can obtain a target domain-specific dictionary, which is automatically acquired. According to the shared word vocabulary and the target domain-specific dictionary, machine-annotated target domain data can be obtained.

[0125] In one embodiment of the present disclosure, source domain specific words refer to words exclusive to the source domain, such as the specific words in the aforementioned news field, for example, "reporter", "report", etc. Shared words include the aforementioned "is", "research", "discovery", etc. The target domain data to be annotated can be, for example, text in the aforementioned medical field. By matching the annotated source domain data with the unannotated target domain data, since the source domain specific words and shared words have been annotated, the source domain specific words, shared words, and the target domain data to be annotated can be identified from the matching results.

[0126] In one embodiment of the present disclosure, step S120 may include: matching the annotated source domain data with the unannotated target domain data to obtain source domain specific words, shared words, and target domain data to be annotated; using the shared words to obtain a shared word vocabulary; using the target domain data to be annotated to obtain a target domain-specific dictionary; and obtaining machine-annotated target domain data based on the shared word vocabulary and the target domain-specific dictionary.

[0127] According to the technical solution provided by the embodiment of the present disclosure, the unlabeled target domain data is machine-labeled by utilizing the labeled source domain data, including: matching the labeled source domain data with the unlabeled target domain data to obtain source domain-specific words, shared words, and target domain data to be labeled; utilizing the shared words to obtain a shared word vocabulary; utilizing the target domain data to be labeled to obtain a target domain-specific dictionary; and obtaining machine-labeled target domain data based on the shared word vocabulary and the target domain-specific dictionary. Machine-labeled target domain data can be obtained for target domain data that has not been manually labeled, thereby improving the accuracy of word segmentation of target domain data.

[0128] In one embodiment of the present disclosure, the target domain text data can be processed in a statistical manner to obtain a target domain-specific dictionary, that is, to complete the tagging of target domain-specific words. In one embodiment of the present disclosure, the shared words in the target domain text data can be marked by processing the shared words through a predetermined word segmentation model.

[0129] In one embodiment of the present disclosure, the predetermined word segmentation model may include, for example, a GCNN-CRF model (a convolutional neural network with a gating mechanism-conditional random field classification layer model). This is a supervised Chinese word segmentation model. For an input sentence s={c 1 ,c 2 ,…,c n}, the model aims to give each word c iAdd a special tag, which is one of the set {B, M, E, S}. Among them, B means that this character is the beginning of a word, M means that this character is in the middle of a word, E means that this character is the end of a word, and S means that this character can be an independent word. For each character c i , first convert it into a word vector e i , so the sentence is represented by e s ={e 1 ,e 2 ,…,e n}. Then e s Through a GCNN, it is encoded into a sentence-level latent vector. Finally, the latent vector is sent to the CRF classification layer, the label score is calculated for each word, the probability distribution of this character in the label set is obtained, and the label with the highest probability is selected as the label of this character. Applying the GCNN-CRF model trained based on any one or more training data from other fields to the target domain text data can effectively identify domain-independent words in the target domain text data. These words can also be called shared words, such as "discovery", "already", "research", "is", "of", etc.

[0130] In one embodiment of the present disclosure, the target domain text data is processed based on statistics to obtain specific words of the target domain text data. One or more statistical indicators may be used to determine whether a text segment constitutes a word.

[0131] In one embodiment of the present disclosure, the step of obtaining a target domain-specific dictionary using the target domain data to be annotated includes: identifying words in the target domain data to be annotated by calculating one or more of the following indicators: cohesion score, free score, word frequency, and word frequency-inverse text frequency index.

[0132] In one embodiment of the present disclosure, the cohesion score can be used to measure the cohesion degree of a text segment. In one embodiment of the present disclosure, the free score can be used as a characteristic value to measure whether the text segment can be flexibly used. In one embodiment of the present disclosure, the word frequency or the word frequency-inverse text frequency index can be used as an indicator to measure whether the text segment constitutes a word. In one embodiment of the present disclosure, at least one of the cohesion score, the free score, the word frequency and the word frequency-inverse text frequency index can be used as the aforementioned statistical indicator.

[0133] According to the technical solution provided by the embodiment of the present disclosure, a target domain-specific dictionary is obtained by utilizing the target domain data to be annotated, including: identifying words in the target domain data to be annotated by calculating one or more of the following indicators: cohesion score, free score, word frequency, and word frequency-inverse text frequency index, so that machine-annotated target domain data without manual annotating can be obtained, thereby improving the accuracy of word segmentation of target domain data.

[0134] In one embodiment of the present disclosure, the cohesion score is calculated in the following manner, including: obtaining a first text segment from the target domain data to be annotated; obtaining a first probability that the first text segment appears as a whole in the target domain data to be annotated; dividing the first text segment into two sub-segments, and determining a first segmentation method that maximizes the product of the probabilities of the two sub-segments appearing in the target domain data to be annotated; and determining the target domain-specific dictionary based on the ratio of the first probability to the product corresponding to the first segmentation method.

[0135] In one embodiment of the present disclosure, the probability that a first text segment in the to-be-annotated data of the target domain appears as a whole in the target domain text can be obtained, which can be referred to as the first probability. In addition, the first text segment can be segmented into two sub-segments by various segmentation methods. In particular, all segmentation methods can be traversed. For each segmentation method, the probabilities that the two sub-segments appear in the target domain text data are determined, and the product of the two is used as a feature value. Since there may be multiple segmentation methods, the embodiment of the present disclosure selects the segmentation method with the largest product, that is, the first segmentation method. The ratio of the first probability to the product corresponding to the first segmentation method is used as the cohesion score. If the frequency of occurrence of a fixed text segment is higher (the overall appearance probability is greater), it indicates that the possibility of it being a word is greater. At the same time, among various segmentation methods, the first segmentation method has the greatest possibility of occurrence. Therefore, using the ratio of the first probability to the product corresponding to the first segmentation method as the cohesion score can be used to measure the cohesion degree of a text segment. For example, for a text segment "lysozyme", it can be segmented into "lyso-" - "zyme", or it can be segmented into "lys-" - "ozyme". For the first segmentation method, the product of the probabilities of "lyso-" and "zyme" appearing is calculated respectively. For the second segmentation method, the product of the probabilities of "lys-" and "ozyme" appearing is calculated respectively. The larger the product, the greater the probability that this segmentation method appears in the target domain text, and thus the text segment is more likely to be two independent words according to this segmentation method. Since the probability that the same text segment appears as a whole in the same target domain text is fixed, therefore, the larger the product corresponding to the first segmentation method, the smaller the cohesion score, and the lower the possibility that the text segment is an independent word. In the embodiment of the present disclosure, it can be set whether the cohesion score is greater than or less than a certain threshold to determine the high or low cohesion degree of the text segment.

[0136] According to the technical solution provided by the embodiment of the present disclosure, the cohesion score is calculated in the following manner, including: obtaining a first text segment from the to-be-annotated data of the target domain; obtaining the first probability that the first text segment appears as a whole in the to-be-annotated data of the target domain; segmenting the first text segment into two sub-segments, and determining the first segmentation method that maximizes the product of the probabilities that the two sub-segments appear in the to-be-annotated data of the target domain; and determining the target domain-specific dictionary based on the ratio of the first probability to the product corresponding to the first segmentation method, so as to obtain machine-annotated target domain data for the target domain data without manual annotation, and improve the accuracy of word segmentation of the target domain data.

[0137] In one embodiment of the present disclosure, the free fraction is calculated in the following manner, including: obtaining a second text segment from the data to be labeled in the target domain; determining the left adjacent word set entropy and the right adjacent word set entropy of the second text segment; and determining the specific dictionary of the target domain based on the smaller of the left adjacent word set entropy and the right adjacent word set entropy.

[0138] In one embodiment of the present disclosure, for the second text segment in the data to be labeled in the target domain obtained, the left adjacent word set entropy and the right adjacent word set entropy of the text segment in the text data of the target domain can be determined. The smaller of them is used as the free fraction. Since a text segment with a high condensation fraction is not necessarily a word and may also be a fixed collocation containing more than two words, the characteristic of a word should be that it can be flexibly used in context. This free fraction can be used to measure a characteristic value of whether the text segment can be flexibly used. Whether the text segment is an independent word that can be flexibly used can be measured by comparing it with a threshold.

[0139] The entropy ES can be determined by the following formula:

[0140]

[0141] where k is the number of element types in the word set, and j is a certain element in the word set. Take a simple example. For the text data "When eating grapes, don't spit out the grape skins. When not eating grapes, spit out the grape skins", the word "grape" appears four times. The left adjacent word sets are {eat, spit, eat, spit} respectively, and the right adjacent word sets are {don't, skins, don't, skins} respectively. According to the formula, the information entropy of the left adjacent words of the word "grape" is –(1 / 2)·log(1 / 2)–(1 / 2)·log(1 / 2)≈0.693, and the information entropy of its right adjacent words is –(1 / 2)·log(1 / 2)–(1 / 4)·log(1 / 4)–(1 / 4)·log(1 / 4)≈1.04. The smaller one, that is, the left adjacent word set entropy 0.693, can be taken as the free fraction of "grape". If this free fraction is large enough, it means that the text segment can be flexibly used, that is, connected with a variety of different words. If "taopi" is taken as a text segment during division, its left adjacent word set is {pu, pu}, and its left adjacent information entropy is -1·log(1) = 0, indicating that this text segment is not free and can only be connected with fixed words and cannot become an independent word.

[0142] According to the technical solution provided by the embodiment of the present disclosure, the free score is calculated in the following manner, including: obtaining a second text segment from the target domain data to be annotated; determining the left neighbor character set entropy and the right neighbor character set entropy of the second text segment; determining the target domain-specific dictionary based on the smaller of the left neighbor character set entropy and the right neighbor character set entropy. Based on the annotated source domain data and the machine-annotated target domain data, an adversarial network can be used to optimize the target domain data to improve the accuracy of word segmentation of the target domain data.

[0143] In one embodiment of the present disclosure, the step of acquiring a target domain-specific dictionary using the target domain data to be annotated includes: processing the target domain data to be annotated by using a predetermined word segmentation model to acquire a target domain-specific dictionary.

[0144] In one embodiment of the present disclosure, a predetermined word segmentation model may be used to mine new words on the target domain data to be annotated to obtain a target domain-specific dictionary.

[0145] According to the technical solution provided by the embodiment of the present disclosure, a target domain-specific dictionary is obtained by utilizing the target domain data to be annotated, including: processing the target domain data to be annotated by a predetermined word segmentation model to obtain a target domain-specific dictionary, and based on the annotated source domain data and the machine-annotated target domain data, an adversarial network is used to optimize the target domain data to improve the accuracy of word segmentation of the target domain data.

[0146] In one embodiment of the present disclosure, for target domain data containing sentences such as "scientific research on lysozyme", the specific word "lysozyme" can be first mined out in a statistical manner and annotated as "BME", and then the trained predetermined word segmentation model is used for word segmentation, and "scientific research on lysozyme" is annotated as "SBEBE". In this way, machine automatic annotation can be implemented in unannotated target domain text data.

[0147] In the embodiments of the present disclosure, a combination of a statistical method and a method based on a trained predetermined word segmentation model is adopted, and the target domain vocabulary containing domain-specific words and shared words can be more accurately mined without any domain dictionary or annotation information, which effectively solves the problem in the prior art that the model cannot accurately recognize words in testing due to the presence of "unregistered words" in the training data.

[0148] In one embodiment of the present disclosure, a text segment can be obtained from the target domain data to be annotated, the word frequency or word frequency-inverse text frequency index of the text segment can be determined as feature information of the text segment, and the word segmentation result of the target domain data can be determined based on the feature information.

[0149] In one embodiment of the present disclosure, for a text segment in the target domain data to be annotated, a term frequency or a term frequency-inverse text frequency index can be used as an indicator to measure whether the text segment constitutes a word. Among them, term frequency (TF) refers to the number of times the text segment appears in the target domain text data; the inverse document frequency (IDF) can be obtained by dividing the total number of files by the number of files containing the text segment, and then taking the logarithm of the obtained quotient. The two can be used independently or in combination. For example, term frequency can be used alone. If a text segment is a word, it should appear frequently in the corpus. A threshold can be used to measure whether the term frequency reaches a certain number, as a reference indicator to determine whether the text segment becomes a word. For example, the importance of the text segment can be determined by the term frequency-inverse text frequency index ("TF-IDF", i.e. TF*IDF) indicator to determine whether it is a specific word in the target domain, and a threshold can also be set for comparison.

[0150] In the embodiments of the present disclosure, part or all of the cohesion score, free score, word frequency, and word frequency-inverse text frequency index discussed above can be used as a basis for processing the target domain text data to obtain the word segmentation result as needed. Those skilled in the art can understand that according to the teachings of the present disclosure, various thresholds can be set for the cohesion score, free score, word frequency, and word frequency-inverse text frequency index to obtain the word segmentation result.

[0151] Figure 4 A schematic diagram of an adversarial network according to an embodiment of the present disclosure is shown. Figure 4 As shown, in one embodiment of the present disclosure, the adversarial network includes a shared encoder, a discriminator and a target domain word segmenter, and step S130 may include the following steps: encoding the third text segment in the annotated source domain data to obtain a first latent vector; encoding the fourth text segment in the machine-annotated target domain data to obtain a second latent vector; the shared encoder randomly selects the first latent vector or the second latent vector as the first code and inputs it into the discriminator, so that the discriminator determines that the first code is the first latent vector or the second latent vector; the target domain word segmenter obtains the optimized target domain data according to the result of the discriminator.

[0152] According to the technical solution provided by the embodiment of the present disclosure, the adversarial network includes a shared encoder, a discriminator and a target domain word segmenter, and the annotated source domain data and the machine-annotated target domain data are input into the adversarial network to obtain the optimized target domain data, including: encoding the third text segment in the annotated source domain data to obtain a first latent vector; encoding the fourth text segment in the machine-annotated target domain data to obtain a second latent vector; the shared encoder randomly selects the first latent vector or the second latent vector as the first code and inputs it into the discriminator, so that the discriminator determines that the first code is the first latent vector or the second latent vector; the target domain word segmenter obtains the optimized target domain data according to the result of the discriminator, and can optimize the target domain data based on the annotated source domain data and the machine-annotated target domain data using the adversarial network to improve the accuracy of the target domain data segmentation.

[0153] In one embodiment of the present disclosure, the input of the adversarial network comes from source domain data and target domain data, wherein the source domain data originally carries annotation information, i.e., the word segmentation result of the source domain text data, and the target domain text data is automatically annotated by the machine through the operation S120 described above to obtain the word segmentation result of the target domain text data. The third text segment and the fourth text segment input to the adversarial network are respectively intercepted from the source domain data and the target domain data, for example, they can be intercepted according to the granularity of the whole paragraph, the whole sentence or the sentence.

[0154] In one embodiment of the present disclosure, a shared encoder and a discriminator are trained as a pair of adversarial roles. The shared encoder can be, for example, a GCNN model, or other neural network models, such as RNN (recurrent neural network), Transformer language model, etc. The discriminator is a binary classifier. The shared encoder randomly obtains a text segment s* (a third text segment or a fourth text segment) and a word segmentation result from the source domain text data and the target domain text data, encodes it to obtain an implicit vector H* (a first implicit vector or a second implicit vector) as a first code; the discriminator discriminates that the first code is a first implicit vector or a second implicit vector according to the first code, thereby determining that the text segment corresponding to the first code is a third text segment from the source domain data or a fourth text segment from the target domain data. For example, the discriminator can generate the probability that the input text segment comes from the source domain or the target domain. If the judgment is correct, the parameters of the shared encoder are improved, hoping that the discriminator cannot make a correct judgment; if the judgment is wrong, the parameters of the discriminator are improved, hoping that the source of the text segment can be correctly judged. In this way, the shared encoder can be trained to focus on domain-independent features, which helps to correctly segment the target domain text data. In the adversarial training process of the shared encoder and the discriminator, the source domain text data and the target domain text data can be reused continuously to obtain optimized target domain data and improve the word segmentation effect of the model.

[0155] In one embodiment of the present disclosure, before the text segment s* enters the shared encoder, s* needs to be converted into a coding sequence X* (eg, a word vector), and the shared encoder further encodes the coding sequence X* to obtain H*.

[0156] In one embodiment of the present disclosure, for a text segment s* or s from target domain text data, tgt , the trained shared encoder outputs the generated first encoding H* to the target domain segmenter, which works together with the text segment and the segmentation result of the text segment to obtain the optimized segmentation result.

[0157] In one embodiment of the present disclosure, the target domain word segmenter includes a target domain encoder and a first word segmenter. Obtaining optimized target domain data according to the result of the discriminator may include the following steps: obtaining a second encoding through the target domain encoder, the second encoding being obtained according to the second implicit vector; and obtaining an optimized word segmentation result of the fourth text fragment through the first word segmenter based on the first encoding and the second encoding.

[0158] According to the technical solution provided by the embodiment of the present disclosure, the target domain word segmenter includes a target domain encoder and a first word segmenter, and the optimized target domain data is obtained according to the result of the discriminator, including: obtaining a second code through the target domain encoder, and the second code is obtained according to the second implicit vector; based on the first code and the second code, the optimized word segmentation result of the fourth text fragment is obtained through the first word segmenter. Based on the labeled source domain data and the machine-labeled target domain data, the adversarial network can be used to optimize the target domain data to improve the accuracy of the target domain data segmentation.

[0159] In one embodiment of the present disclosure, the target domain encoder may be a GCNN model or other neural network models, such as RNN, Transformer language model, etc. The first word segmenter may be a CRF (conditional random field) model. The text segment s from the target domain text data tgt is converted into the coding sequence X t The target domain encoder can convert X t Further encoded into the second code H t The text fragment tgt It can also be input as s* to the shared encoder to generate the first encoding H*. The first word segmenter can convert the second encoding H t and the first code H* to form a vector H(x t ) is input into the conditional random field model to obtain the optimized target domain data, that is, the updated word segmentation result y t .

[0160] In one embodiment of the present disclosure, since the parameters of the shared encoder change during the training process, the first encoding H* generated by the same text segment is different, and thus the word segmentation results output by the first word segmenter may also be different. The results output by the first word segmenter during the training process can be used to update the word segmentation results in the target domain text data, and the updated word segmentation results can be used for training again. After the training is completed, the word segmentation results output by the first word segmenter are the final optimized word segmentation results.

[0161] In one embodiment of the present disclosure, the adversarial network also includes a source domain encoder and a second word segmenter, and the natural language processing method may further include the following steps: the source domain encoder obtains a third encoding, and the third encoding is obtained according to the first implicit vector; based on the first encoding and the third encoding, the optimized word segmentation result of the third text segment is obtained through the second word segmenter.

[0162] According to the technical solution provided by the embodiment of the present disclosure, the adversarial network also includes a source domain encoder and a second word segmenter, and the method also includes: the source domain encoder obtains a third code, and the third code is obtained according to the first implicit vector; based on the first code and the third code, the optimized word segmentation result of the third text fragment is obtained by the second word segmenter. Based on the annotated source domain data and the machine-annotated target domain data, the adversarial network can be used to optimize the target domain data to improve the accuracy of the source domain data segmentation.

[0163] In one embodiment of the present disclosure, the source domain encoder may be a GCNN model or other neural network models, such as RNN, Transformer language model, etc. The second tokenizer may be a CRF model. src is converted into the coding sequence X s The source domain encoder can convert X s Further encoded into the third code H s The text fragment src It is also input as s* to the shared encoder to generate the first encoding H*. The second word segmenter can convert the third encoding H s and the first code H* to form a vector H(x s ) is input into the conditional random field model to obtain the optimized word segmentation result y of the text fragment of the source domain text data s .

[0164] It should be noted that the embodiments of the present disclosure perform natural language word segmentation on target domain (text) data by adopting a series of means such as obtaining source domain (text) data, the word segmentation results of the source domain (text) data and utilizing the word segmentation results of the source domain, optimizing the target domain data through an adversarial network, etc., to achieve a complete set of solutions from target domain word segmentation to optimizing the word segmentation results, thereby improving the accuracy of word segmentation of the target domain data.

[0165] The present disclosure also provides a text processing method, including: obtaining annotated first text data and unannotated second text data; using the annotated first text data to machine-annotate the unannotated second text data; inputting the annotated first text data and the machine-annotated second text data into an adversarial network to obtain optimized second text data; and based on the optimized second text data, outputting an optimized word segmentation result of the second text data.

[0166] According to the technical solution provided by the embodiment of the present disclosure, by obtaining annotated first text data and unannotated second text data; using the annotated first text data, machine annotating the unannotated second text data; inputting the annotated first text data and the machine-annotated second text data into an adversarial network to obtain optimized second text data; based on the optimized second text data, outputting the optimized word segmentation result of the second text data, the adversarial network can be used to optimize the word segmentation result of the second text data based on the annotated first text data and the machine-annotated second text data, thereby improving the accuracy of the word segmentation of the second text data.

[0167] In one embodiment of the present disclosure, the annotation may be, for example, an annotation performed in the above-described {B, M, E, S} annotation manner. In one embodiment of the present disclosure, the first text data may refer to the annotated source domain text data in the above-described embodiment.

[0168] In one embodiment of the present disclosure, the second text data may refer to the unlabeled target domain text data in the aforementioned embodiment. In one embodiment of the present disclosure, the processing method for obtaining the label of the second text data may be, for example, obtaining the word segmentation result of the second text data by processing the second text data based on a statistical method; and / or, obtaining the word segmentation result of the second text data by processing the second text data through a predetermined word segmentation model. The predetermined word segmentation model may be, for example, the GCNN-CRF model described above.

[0169] In one embodiment of the present disclosure, the adversarial network can be implemented as a structure similar to Figure 4 The adversarial network, the results of the adversarial network and the word segmentation optimization process shown are similar to those in the aforementioned embodiment and will not be described in detail here.

[0170] In one embodiment of the present disclosure, the obtained optimized word segmentation results can be output in various ways, for example, the word segmentation labels corresponding to the characters in the second text data can be directly output, or delimiters can be added to the second text data according to the obtained optimized word segmentation results, and the second text data with delimiters can be output.

[0171] The following reference Figure 5 A natural language processing device according to an embodiment of the present disclosure is described.

[0172] Figure 5 FIG. 2 shows a structural block diagram of a natural language processing device according to an embodiment of the present disclosure. Figure 5 As shown, the natural language processing device 500 includes an acquisition module 510 , a labeling module 520 and an input module 530 .

[0173] The acquisition module 510 is configured to acquire labeled source domain data and unlabeled target domain data.

[0174] The labeling module 520 is configured to perform machine labeling on the unlabeled target domain data using the labeled source domain data.

[0175] The input module 530 is configured to input the labeled source domain data and the machine-labeled target domain data into the adversarial network to obtain optimized target domain data.

[0176] According to the technical solution provided by the embodiment of the present disclosure, an acquisition module is configured to acquire labeled source domain data and unlabeled target domain data; a labeling module is configured to use the labeled source domain data to machine label the unlabeled target domain data; an input module is configured to input the labeled source domain data and the machine-labeled target domain data into an adversarial network to acquire optimized target domain data. Based on the labeled source domain data and the machine-labeled target domain data, the adversarial network can be used to optimize the target domain data, thereby improving the accuracy of word segmentation of the target domain data.

[0177] In one embodiment of the present disclosure, the annotation module 520 includes:

[0178] A matching submodule is configured to match the annotated source domain data with the unannotated target domain data to obtain source domain specific words, shared words, and target domain data to be annotated;

[0179] A shared word vocabulary acquisition submodule, configured to acquire a shared word vocabulary using the shared word;

[0180] A target domain specific dictionary word acquisition submodule is configured to acquire a target domain specific dictionary using the target domain data to be annotated;

[0181] The target domain data acquisition submodule is configured to acquire machine-annotated target domain data according to the shared word vocabulary and the target domain-specific dictionary.

[0182] According to the technical solution provided by the embodiment of the present disclosure, the annotation module includes: a matching submodule, configured to match the annotated source domain data with the unannotated target domain data, and obtain source domain specific words, shared words, and target domain data to be annotated; a shared word vocabulary acquisition submodule, configured to use the shared words to obtain a shared word vocabulary; a target domain-specific dictionary word acquisition submodule, configured to use the target domain data to be annotated to obtain a target domain-specific dictionary; a target domain data acquisition submodule, configured to obtain machine-annotated target domain data based on the shared word vocabulary and the target domain-specific dictionary, which can obtain machine-annotated target domain data for target domain data without manual annotation, thereby improving the accuracy of word segmentation of target domain data.

[0183] In one embodiment of the present disclosure, the specific dictionary vocabulary acquisition submodule is further configured to: identify words in the target domain data to be annotated by calculating one or more of the following indicators: cohesion score, free score, word frequency, and word frequency-inverse text frequency index.

[0184] According to the technical solution provided by the embodiment of the present disclosure, the submodule for acquiring the unique dictionary word list is also configured to: identify the words in the target domain data to be annotated by calculating one or more of the following indicators: cohesion score, free score, word frequency and word frequency-inverse text frequency index, so as to obtain machine-annotated target domain data without manual annotating, thereby improving the accuracy of word segmentation of the target domain data.

[0185] In one embodiment of the present disclosure, the cohesion score is calculated in the following manner, including: obtaining a first text segment from the target domain data to be annotated; obtaining a first probability that the first text segment appears as a whole in the target domain data to be annotated; dividing the first text segment into two sub-segments, and determining a first segmentation method that maximizes the product of the probabilities of the two sub-segments appearing in the target domain data to be annotated; and determining the target domain-specific dictionary based on the ratio of the first probability to the product corresponding to the first segmentation method.

[0186] According to the technical solution provided by the embodiment of the present disclosure, the cohesion score is calculated in the following manner, including: obtaining a first text segment from the target domain data to be annotated; obtaining a first probability that the first text segment appears as a whole in the target domain data to be annotated; dividing the first text segment into two sub-segments, and determining a first segmentation method that maximizes the product of the probabilities of the two sub-segments appearing in the target domain data to be annotated; and determining the target domain-specific dictionary based on the ratio of the first probability to the product corresponding to the first segmentation method. This can obtain machine-annotated target domain data for target domain data that is not manually annotated, thereby improving the accuracy of word segmentation of the target domain data.

[0187] In one embodiment of the present disclosure, the free score is calculated in the following manner, including: obtaining a second text segment from the target domain data to be annotated; determining the left neighbor character set entropy and the right neighbor character set entropy of the second text segment; and determining the target domain-specific dictionary based on the smaller of the left neighbor character set entropy and the right neighbor character set entropy.

[0188] According to the technical solution provided by the embodiment of the present disclosure, the free score is calculated in the following manner, including: obtaining a second text segment from the target domain data to be annotated; determining the left neighbor character set entropy and the right neighbor character set entropy of the second text segment; determining the target domain-specific dictionary based on the smaller of the left neighbor character set entropy and the right neighbor character set entropy. Based on the annotated source domain data and the machine-annotated target domain data, an adversarial network can be used to optimize the target domain data to improve the accuracy of word segmentation of the target domain data.

[0189] In one embodiment of the present disclosure, the target domain specific dictionary word acquisition submodule is further configured to: process the target domain to-be-annotated data through a predetermined word segmentation model to acquire the target domain specific dictionary.

[0190] According to the technical solution provided by the embodiment of the present disclosure, the target domain-specific dictionary word acquisition submodule is also configured to: process the target domain data to be annotated through a predetermined word segmentation model to obtain a target domain-specific dictionary, and can optimize the target domain data based on the annotated source domain data and the machine-annotated target domain data using an adversarial network to improve the accuracy of the target domain data segmentation.

[0191] In one embodiment of the present disclosure, the adversarial network includes a shared encoder, a discriminator and a target domain word segmenter, and the input module includes: a first encoding submodule, configured to encode the third text segment in the annotated source domain data to obtain a first latent vector; a second encoding submodule, configured to encode the fourth text segment in the machine-annotated target domain data to obtain a second latent vector, wherein the shared encoder randomly selects the first latent vector or the second latent vector as the first code input to the discriminator, so that the discriminator determines that the first code is the first latent vector or the second latent vector; wherein the target domain word segmenter obtains the optimized target domain data according to the result of the discriminator.

[0192] According to the technical solution provided by the embodiment of the present disclosure, the adversarial network includes a shared encoder, a discriminator and a target domain word segmenter, and the input module includes: a first encoding submodule, configured to encode the third text segment in the labeled source domain data to obtain a first latent vector; a second encoding submodule, configured to encode the fourth text segment in the machine-labeled target domain data to obtain a second latent vector, wherein the shared encoder randomly selects the first latent vector or the second latent vector as the first code input to the discriminator, so that the discriminator determines that the first code is the first latent vector or the second latent vector; wherein the target domain word segmenter obtains the optimized target domain data according to the result of the discriminator, and can optimize the target domain data based on the labeled source domain data and the machine-labeled target domain data by using the adversarial network to improve the accuracy of the target domain data segmentation.

[0193] In one embodiment of the present disclosure, the target domain word segmenter includes a target domain encoder and a first word segmenter, and the target domain word segmenter is configured to: obtain a second code through the target domain encoder, and the second code is obtained according to the second implicit vector; based on the first code and the second code, obtain the optimized word segmentation result of the fourth text fragment through the first word segmenter.

[0194] According to the technical solution provided by the embodiment of the present disclosure, the target domain word segmenter includes a target domain encoder and a first word segmenter, and the target domain word segmenter is configured to: obtain a second code through the target domain encoder, and the second code is obtained according to the second implicit vector; based on the first code and the second code, obtain the optimized word segmentation result of the fourth text fragment through the first word segmenter, and can optimize the target domain data based on the annotated source domain data and the machine-annotated target domain data by using an adversarial network to improve the accuracy of the target domain data segmentation.

[0195] In one embodiment of the present disclosure, the adversarial network also includes a source domain encoder and a second word segmenter, wherein: the source domain encoder obtains a third encoding, and the third encoding is obtained according to the first implicit vector; based on the first encoding and the third encoding, the second word segmenter obtains an optimized word segmentation result of the third text segment.

[0196] According to the technical solution provided by the embodiment of the present disclosure, the adversarial network also includes a source domain encoder and a second word segmenter, wherein: the source domain encoder obtains a third code, and the third code is obtained according to the first implicit vector; based on the first code and the third code, the second word segmenter obtains the optimized word segmentation result of the third text fragment, and based on the annotated source domain data and the machine-annotated target domain data, the adversarial network can be used to optimize the target domain data to improve the accuracy of the source domain data segmentation.

[0197] Those skilled in the art will understand that referring to Figure 5 The technical solutions described can be compared with Figures 1 to 4 The described embodiments are combined to provide a reference Figures 1 to 4 The technical effects achieved by the described embodiments. Figures 1 to 4 The specific content of the description will not be repeated here.

[0198] The above embodiments describe the internal functions and structures of the natural language processing device. In a possible design, the structure of the natural language processing device can be implemented as an electronic device, such as Figure 6 As shown in , the electronic device 600 may include a processor 601 and a memory 602 .

[0199] The memory 602 is used to store a program that supports the electronic device to execute the natural language processing method or code generation method in any of the above embodiments, and the processor 601 is configured to execute the program stored in the memory 602.

[0200] In one embodiment of the present disclosure, the memory 602 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 601 to implement the following steps:

[0201] Obtain labeled source domain data and unlabeled target domain data;

[0202] Using the labeled source domain data, machine-labeling the unlabeled target domain data;

[0203] The labeled source domain data and the machine-labeled target domain data are input into the adversarial network to obtain optimized target domain data.

[0204] In one embodiment of the present disclosure, the step of using the annotated source domain data to machine annotate the unannotated target domain data includes:

[0205] Matching the annotated source domain data with the unannotated target domain data to obtain source domain specific words, shared words, and target domain data to be annotated;

[0206] Obtaining a shared word vocabulary using the shared words;

[0207] Acquire a target domain-specific dictionary using the target domain data to be annotated;

[0208] According to the shared word vocabulary and the target domain-specific dictionary, machine-annotated target domain data is obtained.

[0209] In one embodiment of the present disclosure, the step of obtaining a target domain-specific dictionary using the target domain data to be annotated includes:

[0210] Identify words in the target domain data to be annotated by calculating one or more of the following indicators:

[0211] cohesion score, freedom score, word frequency, and word frequency-inverse text frequency index.

[0212] In one embodiment of the present disclosure, the coagulation score is calculated in the following manner, including:

[0213] Acquire a first text segment from the target domain data to be annotated;

[0214] Obtaining a first probability that the first text segment as a whole appears in the target domain data to be annotated;

[0215] Segment the first text segment into two sub-segments, and determine a first segmentation method that maximizes the product of probabilities of the two sub-segments appearing in the target domain data to be annotated; and

[0216] The target domain-specific dictionary is determined based on a ratio of the first probability and a product corresponding to the first segmentation method.

[0217] In one embodiment of the present disclosure, the free fraction is calculated in the following manner, including:

[0218] Acquire a second text segment from the target domain data to be annotated;

[0219] Determine the left neighbor character set entropy and the right neighbor character set entropy of the second text segment;

[0220] The target domain-specific dictionary is determined based on the smaller of the left neighbor word set entropy and the right neighbor word set entropy.

[0221] In one embodiment of the present disclosure, the step of obtaining a target domain-specific dictionary using the target domain data to be annotated includes:

[0222] The target domain data to be annotated is processed by a predetermined word segmentation model to obtain a target domain-specific dictionary.

[0223] In one embodiment of the present disclosure, the adversarial network includes a shared encoder, a discriminator, and a target domain word segmenter, and the step of inputting the labeled source domain data and the machine-labeled target domain data into the adversarial network to obtain optimized target domain data includes:

[0224] Encoding a third text segment in the labeled source domain data to obtain a first latent vector;

[0225] Encoding a fourth text segment in the machine-annotated target domain data to obtain a second latent vector;

[0226] The shared encoder randomly selects the first latent vector or the second latent vector as the first code and inputs it into the discriminator, so that the discriminator determines that the first code is the first latent vector or the second latent vector;

[0227] The target domain word segmenter obtains optimized target domain data according to the result of the discriminator.

[0228] In one embodiment of the present disclosure, the target domain word segmenter includes a target domain encoder and a first word segmenter, and the step of obtaining optimized target domain data according to the result of the discriminator includes:

[0229] Obtaining a second code through the target domain encoder, wherein the second code is obtained according to the second latent vector;

[0230] Based on the first encoding and the second encoding, an optimized word segmentation result of the fourth text segment is obtained through a first word segmenter.

[0231] In one embodiment of the present disclosure, the adversarial network further includes a source domain encoder and a second word segmenter, and the method further includes:

[0232] The source domain encoder obtains a third code, wherein the third code is obtained according to the first latent vector;

[0233] Based on the first code and the third code, an optimized word segmentation result of the third text segment is obtained through a second word segmenter.

[0234] In one embodiment of the present disclosure, the memory 602 is used to store one or more computer instructions, wherein the one or more computer instructions are also executed by the processor 601 to implement the following steps:

[0235] Acquire annotated first text data and unannotated second text data;

[0236] Using the annotated first text data, machine annotating the unannotated second text data;

[0237] Inputting the annotated first text data and the machine-annotated second text data into an adversarial network to obtain optimized second text data;

[0238] Based on the optimized second text data, an optimized word segmentation result of the second text data is output.

[0239] The exemplary embodiments of the present disclosure further provide a computer storage medium for storing computer software instructions used by the positioning device, which includes a program for executing any of the above embodiments, thereby achieving the technical effects brought by the method.

[0240] Figure 7 It is a structural diagram of a computer system suitable for implementing a natural language processing method or a text processing method according to an embodiment of the present disclosure.

[0241] like Figure 7 As shown, the computer system 700 includes a processor (CPU, GPU, TPU, FPGA, etc.) 701, which can perform various processes in the embodiments shown in the above figures according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The processor 701, ROM 702 and RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0242] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0243] In particular, according to an embodiment of the present disclosure, the method described above with reference to the accompanying drawings can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a readable medium thereof, and the computer program includes program code for executing the method in the accompanying drawings. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 709, and / or installed from the removable medium 711.

[0244] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the road map or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0245] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0246] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure, thereby achieving the technical effects brought by the method.

[0247] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A natural language processing method, characterized in that: include: Obtain labeled source domain data and unlabeled target domain data; Matching the annotated source domain data with the unannotated target domain data to obtain source domain specific words, shared words, and target domain data to be annotated, wherein the source domain specific words are used to represent words that are exclusive to a domain that is different from the target domain, and the shared words are used to represent words that are shared by the source domain and the target domain; Acquire machine-annotated target domain data according to the source domain-specific words, the shared words, and the target domain data to be annotated; The labeled source domain data and the machine-labeled target domain data are input into the adversarial network to obtain optimized target domain data.

2. The method according to claim 1, characterized in that: The step of obtaining machine-annotated target domain data according to the source domain specific words, the shared words, and the target domain data to be annotated includes: Obtaining a shared word vocabulary using the shared words; Acquire a target domain-specific dictionary using the target domain data to be annotated; According to the shared word vocabulary and the target domain-specific dictionary, machine-annotated target domain data is obtained.

3. The method according to claim 2, characterized in that The step of obtaining a target domain-specific dictionary using the target domain data to be annotated includes: Identify words in the target domain data to be annotated by calculating one or more of the following indicators: Cohesion score, freedom score, word frequency and word frequency-inverse text frequency index, wherein the cohesion score is used to measure the degree of cohesion of the text segment, and the freedom score is used to measure the degree of freedom of the text segment to connect with a variety of different words.

4. The method according to claim 3, characterized in that The coagulation score is calculated by the following method, including: Acquire a first text segment from the target domain data to be annotated; Obtaining a first probability that the first text segment as a whole appears in the target domain data to be annotated; Segment the first text segment into two sub-segments, and determine a first segmentation method that maximizes the product of probabilities of the two sub-segments appearing in the target domain data to be annotated; and The target domain-specific dictionary is determined based on a ratio of the first probability and a product corresponding to the first segmentation method.

5. The method according to claim 3, characterized in that: The free fraction is calculated as follows, including: Acquire a second text segment from the target domain data to be annotated; Determine the left neighbor character set entropy and the right neighbor character set entropy of the second text segment; The target domain-specific dictionary is determined based on the smaller of the left neighbor word set entropy and the right neighbor word set entropy.

6. The method according to claim 2, characterized in that The step of obtaining a target domain-specific dictionary using the target domain data to be annotated includes: The target domain data to be annotated is processed by a predetermined word segmentation model to obtain a target domain-specific dictionary.

7. The method according to any one of claims 1 to 6, characterized in that: The adversarial network includes a shared encoder, a discriminator and a target domain word segmenter. The labeled source domain data and the machine-labeled target domain data are input into the adversarial network to obtain optimized target domain data, including: Encoding a third text segment in the labeled source domain data to obtain a first latent vector; Encoding a fourth text segment in the machine-annotated target domain data to obtain a second latent vector; The shared encoder randomly selects the first latent vector or the second latent vector as the first code and inputs it into the discriminator, so that the discriminator determines that the first code is the first latent vector or the second latent vector; The target domain word segmenter obtains optimized target domain data according to the result of the discriminator.

8. The method according to claim 7, characterized in that The target domain word segmenter includes a target domain encoder and a first word segmenter, and the step of obtaining optimized target domain data according to the result of the discriminator includes: Obtaining a second code through the target domain encoder, wherein the second code is obtained according to the second latent vector; Based on the first encoding and the second encoding, an optimized word segmentation result of the fourth text segment is obtained through a first word segmenter.

9. The method according to claim 7, characterized in that: The adversarial network also includes a source domain encoder and a second word segmenter, and the method also includes: The source domain encoder obtains a third code, wherein the third code is obtained according to the first latent vector; Based on the first code and the third code, an optimized word segmentation result of the third text segment is obtained through a second word segmenter.

10. A natural language processing device, characterized in that: include: An acquisition module, configured to acquire labeled source domain data and unlabeled target domain data; The annotation module is configured to match the annotated source domain data with the unannotated target domain data, obtain source domain specific words, shared words, and target domain data to be annotated, wherein the source domain specific words are used to represent words that are exclusive to a domain that is different from the target domain, and the shared words are used to represent words that are shared by the source domain and the target domain; according to the source domain specific words, the shared words, and the target domain data to be annotated, obtain the machine-annotated target domain data; The input module is configured to input the labeled source domain data and the machine-labeled target domain data into the adversarial network to obtain optimized target domain data.

11. The device according to claim 10, characterized in that The marking module comprises: A matching submodule is configured to match the annotated source domain data with the unannotated target domain data to obtain source domain specific words, shared words, and target domain data to be annotated; A shared word vocabulary acquisition submodule, configured to acquire a shared word vocabulary using the shared word; A target domain specific dictionary word acquisition submodule is configured to acquire a target domain specific dictionary using the target domain data to be annotated; The target domain data acquisition submodule is configured to acquire machine-annotated target domain data according to the shared word vocabulary and the target domain-specific dictionary.

12. The device according to claim 11, characterized in that The specific dictionary word list acquisition submodule is further configured as follows: Identify words in the target domain data to be annotated by calculating one or more of the following indicators: Cohesion score, freedom score, word frequency and word frequency-inverse text frequency index, wherein the cohesion score is used to measure the degree of cohesion of the text segment, and the freedom score is used to measure the degree of freedom of the text segment to connect with a variety of different words.

13. The device according to claim 12, characterized in that The coagulation score is calculated by the following method, including: Acquire a first text segment from the target domain data to be annotated; Obtaining a first probability that the first text segment as a whole appears in the target domain data to be annotated; Segment the first text segment into two sub-segments, and determine a first segmentation method that maximizes the product of probabilities of the two sub-segments appearing in the target domain data to be annotated; and The target domain-specific dictionary is determined based on a ratio of the first probability and a product corresponding to the first segmentation method.

14. The device according to claim 12, characterized in that The free fraction is calculated as follows, including: Acquire a second text segment from the target domain data to be annotated; Determine the left neighbor character set entropy and the right neighbor character set entropy of the second text segment; The target domain-specific dictionary is determined based on the smaller of the left neighbor word set entropy and the right neighbor word set entropy.

15. The device according to claim 11, characterized in that The target domain specific dictionary word acquisition submodule is further configured as follows: The target domain data to be annotated is processed by a predetermined word segmentation model to obtain a target domain-specific dictionary.

16. The device according to any one of claims 10 to 15, characterized in that The adversarial network includes a shared encoder, a discriminator and a target domain word segmenter, and the input module includes: a first encoding submodule, configured to encode a third text segment in the annotated source domain data to obtain a first latent vector; The second encoding submodule is configured to encode a fourth text segment in the target domain data annotated by the machine to obtain a second latent vector, The shared encoder randomly selects the first latent vector or the second latent vector as the first code and inputs it into the discriminator, so that the discriminator determines whether the first code is the first latent vector or the second latent vector; The target domain word segmenter obtains optimized target domain data according to the result of the discriminator.

17. The device according to claim 16, characterized in that The target domain word segmenter includes a target domain encoder and a first word segmenter, and the target domain word segmenter is configured as follows: Obtaining a second code through the target domain encoder, wherein the second code is obtained according to the second latent vector; Based on the first encoding and the second encoding, an optimized word segmentation result of the fourth text segment is obtained through a first word segmenter.

18. The device according to claim 16, characterized in that The adversarial network also includes a source domain encoder and a second word segmenter, wherein: The source domain encoder obtains a third code, wherein the third code is obtained according to the first latent vector; Based on the first encoding and the third encoding, the second word segmenter obtains an optimized word segmentation result of the third text segment.

19. A text processing method, characterized in that: include: Acquire annotated first text data and unannotated second text data; Matching the annotated first text data with the unannotated second text data to obtain first text specific words, shared words, and second text data to be annotated, wherein the first text specific words are used to represent words that are exclusive to a text that is different from the second text, and the shared words are used to represent words that are shared by the first text and the second text; Acquire machine-annotated second text data according to the first text specific word, the shared word, and the second text data to be annotated; Inputting the annotated first text data and the machine-annotated second text data into an adversarial network to obtain optimized second text data; Based on the optimized second text data, an optimized word segmentation result of the second text data is output.

20. An electronic device, characterized in that: comprising a memory and a processor; wherein, The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1 to 9 or claim 19.

21. A readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method as described in any one of claims 1-9 or claim 19 is implemented.

Images