A Tibetan word segmentation and part-of-speech tagging integrated method and system

Abstract

The present invention proposes a method and system for integrating Tibetan word segmentation and part-of-speech tagging, and relates to the field of electronic information. The method provided by the present invention obtains Tibetan text information input by a user, calls an integrated model and segments Tibetan syllables and non-Tibetan character blocks, performs CRF prediction, obtains the optimal label prediction, and sorts the writing form of each Tibetan syllable according to the result of the label prediction to obtain the corresponding tagging result. In this application, by proposing to establish a corresponding integrated model, Tibetan word segmentation and part-of-speech tagging are performed based on the corresponding integrated network. Compared with previous solutions, due to the integrity of the integrated model introduced in this application, the problem of cumulative part-of-speech tagging errors caused by word segmentation errors due to the two-stage execution of domain word segmentation and part-of-speech tagging tasks is avoided, and the corresponding Tibetan word segmentation and part-of-speech tagging can be processed more accurately, further increasing the practicality of the solution.

Description

Technical Field

[0001] The present invention relates to the field of electronic information, and in particular to a method and system for integrating Tibetan word segmentation and part-of-speech tagging. Background Art

[0002] Deep learning has been highly favored in the field of NLP in recent years. Using deep neural networks to train Tibetan word segmentation and part-of-speech tagging models will greatly improve the efficiency of Tibetan word segmentation and part-of-speech tagging. It has become the main research method in the field of Tibetan natural language processing. This not only helps to promote the development of information technology for ethnic minorities, but also points out the direction for informatization research on other ethnic minority languages ​​in my country.

[0003] There is no specific method for integrating word segmentation and part-of-speech tagging for specific Tibetan words in the prior art. Generally, traditional deep learning methods are used to use neural networks to perform Tibetan word segmentation and part-of-speech tagging. However, in traditional neural network models, RNN has the limitation of being difficult to perform parallel calculations efficiently, and CNN has the problem of being unable to capture long-distance features. In addition, errors will accumulate in the process of Tibetan part-of-speech tagging after Tibetan word segmentation. At the same time, the processing of Tibetan glued words cannot be solved by deep learning methods alone, and its practicality is low.

[0004] In view of the above technologies, finding a more practical method for integrating Tibetan word segmentation and part-of-speech tagging is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for integrating Tibetan word segmentation and part-of-speech tagging, which can solve the problem that in the traditional neural network model, RNN has the limitation of being difficult to perform parallel calculation efficiently, CNN has the problem of being unable to capture long-distance features, and error accumulation will occur in the process of Tibetan word segmentation and Tibetan part-of-speech tagging. At the same time, the problem of low practicality cannot be solved by deep learning methods alone in the processing of Tibetan glued words.

[0006] The present invention is achieved in that:

[0007] In a first aspect, the present application provides a method for integrating Tibetan word segmentation and part-of-speech tagging, the method comprising the following steps:

[0008] Obtaining Tibetan text information to be segmented input by a user, and obtaining each Tibetan syllable and corresponding label of the Tibetan text information;

[0009] Calling the integrated model and using the sound node and Unicode encoding to segment the Tibetan syllables and non-Tibetan character blocks and sorting them to obtain a Tibetan sequence, wherein the integrated model is a Tibetan word segmentation and part-of-speech tagging learning model pre-established based on Conformer;

[0010] The integrated model is retrieved to perform CRF prediction on the Tibetan sequence and the corresponding label to obtain the predicted label sequence corresponding to each Tibetan syllable in the Tibetan sequence;

[0011] The writing forms of the Tibetan syllables are sorted according to the predicted label sequence to obtain corresponding annotation results.

[0012] Based on the first aspect, the establishment of the integrated model includes the following steps:

[0013] Read the Tibetan training corpus input by the user and obtain each Tibetan syllable and its corresponding label;

[0014] Construct an integrated model framework for Tibetan word segmentation and part-of-speech tagging based on Conformer;

[0015] The relevant parameters of the integrated model of Tibetan word segmentation and part-of-speech tagging based on Conformer are obtained through training and saved, and the corresponding integrated model is generated based on the relevant parameters.

[0016] Based on the first aspect, the model framework for building an integrated Tibetan word segmentation and part-of-speech tagging based on Conformer includes:

[0017] An Embedding layer is built based on the encoding part of the Transformer model. The Embedding layer is used to obtain the input Tibetan sequence to form a corresponding embedding vector;

[0018] A Decoding layer is built based on the CRF model. The Decoding layer is used to count the probability P of each Tibetan syllable corresponding to a label in the text, and calculate the optimal label category according to the constraint relationship of the sequence label;

[0019] The Tibetan word segmentation and part-of-speech tagging integrated model framework is established based on the Embedding layer and the Decoding layer.

[0020] Based on the first aspect, after reading the Tibetan training corpus input by the user and obtaining each Tibetan syllable and the corresponding label, the method further includes:

[0021] Acquire a data set of the Tibetan training corpus, and perform feature extraction on each Tibetan syllable and its adjacent syllables in the data set to obtain a fused syllable vector representing a feature word;

[0022] A corresponding word vector matrix is ​​established based on the feature words, and the word vector matrix is ​​used to standardize the corresponding integrated model.

[0023] Based on the first aspect, calling the integrated model to perform CRF prediction on the Tibetan sequence and the corresponding label to obtain the predicted label sequence corresponding to each Tibetan syllable in the Tibetan sequence includes:

[0024] Based on the Tibetan syllables, a corresponding Tibetan sequence is obtained where x i In the Tibetan word segmentation task, it represents a syllable, and in the Tibetan part-of-speech tagging task, it represents a Tibetan word;

[0025] Input the Tibetan sequence into the Embedding layer to generate an embedding vector

[0026] Input the embedding vector into the CNN network to fully learn the local features of the syllable or word, convert the syllable or word into a local semantic vector, and concatenate the current syllable or word vector in the local semantic vector;

[0027] The concatenated vector is input into the CRF layer to perform classification prediction and data decoding operations on the label. And the state sequence corresponding to the predicted syllable Output the optimal label category, where y i Represents the label corresponding to each Tibetan syllable or Tibetan word.

[0028] Based on the first aspect, if the Tibetan text information includes a binder word, the method further includes:

[0029] Using the syllable node "·" as a separator, the input sentence is segmented to obtain corresponding segmented syllables, wherein the segmented syllables are the parts between every two syllable nodes;

[0030] Traversing each of the syncopated syllables from back to front to determine the last two characters of each of the syncopated syllables;

[0031] If the last two characters of the syllable are Then the segmented syllable is marked as a pseudo-bonded word, and the pseudo-bonded word is matched with a first dictionary. If the pseudo-bonded word is not in the first dictionary, it is determined that the segmented syllable is not a bonded word; if the pseudo-bonded word is in the first dictionary, it is determined that the segmented syllable is a bonded word;

[0032] If the last character of the syllable is The segmented syllable is marked as a pseudo-grafted word, the pseudo-grafted word is segmented from the function word part, the first half is matched with the second dictionary, and if the first half is supplemented in the second dictionary, The second half is converted. If it is not in the second dictionary, the first half remains unchanged and the second half is converted.

[0033] Match the simulated sticky word with the third dictionary. If the simulated sticky word is not in the third dictionary, determine that the simulated sticky word is not a sticky word; if the simulated sticky word is in the third dictionary, match the simulated sticky word with the fourth dictionary. If the simulated sticky word is in the fourth dictionary, return to the step of segmenting the simulated sticky word from the function word part and matching the first half with the second dictionary. If the simulated sticky word is in the fourth dictionary, combine the simulated sticky word with the previous syllable and match the fifth dictionary. If the combined simulated sticky word is in the fifth dictionary, return to the step of segmenting the simulated sticky word from the function word part and matching the first half with the second dictionary. If the combined simulated sticky word is not in the fifth dictionary, determine that the simulated sticky word is not a sticky word;

[0034] The first dictionary contains The dictionary of all the four function words with a total of 4420 words, the second dictionary contains 48 Tibetan structures, which is used to determine whether the word needs to be transformed, and the third dictionary contains A dictionary of 2201 words to be written in two function words, wherein the fourth dictionary contains A dictionary of four words for gluing two function words, wherein the fifth dictionary contains A dictionary of 9 non-aggregated words composed of two function words and the previous syllable.

[0035] In a second aspect, the present application also provides an integrated Tibetan word segmentation and part-of-speech tagging system, the system comprising:

[0036] An acquisition module is used to acquire the Tibetan text information to be segmented input by the user, and acquire each Tibetan syllable and the corresponding label of the Tibetan text information;

[0037] A calling module, used to call the integrated model and segment the Tibetan syllables and non-Tibetan character blocks using sound nodes and Unicode encoding and sort them to obtain Tibetan sequences, wherein the integrated model is a Tibetan word segmentation and part-of-speech tagging integrated model based on Conformer;

[0038] A prediction module, used to call the integrated model to perform CRF prediction on the Tibetan sequence and the corresponding label, and obtain the predicted label sequence corresponding to each Tibetan syllable in the Tibetan sequence;

[0039] The annotation module is used to sort the writing forms of each Tibetan syllable according to the predicted label sequence to obtain corresponding annotation results.

[0040] In a third aspect, the present application provides an electronic device, comprising a memory for storing one or more programs; a processor; when the one or more programs are executed by the processor, the method as described in any one of the first aspects is implemented.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the above-mentioned first aspects.

[0042] Compared with the prior art, the present invention has at least the following advantages or beneficial effects:

[0043] The present invention proposes an integrated method for Tibetan word segmentation and part-of-speech tagging, which obtains the Tibetan text information to be segmented input by the user, obtains each Tibetan syllable and the corresponding label of the Tibetan text information, calls the integrated model and uses the sound node and Unicode encoding to segment the Tibetan syllables and non-Tibetan character blocks, calls the integrated model for CRF prediction, obtains the corresponding label relationship between any Tibetan sequence of the Tibetan text, and performs global optimal label prediction on the Tibetan syllables, and sorts out the writing form of each Tibetan syllable according to the result of the label prediction to obtain the corresponding tagging result. In this application, by proposing to establish a corresponding integrated model, Tibetan word segmentation and part-of-speech tagging are performed based on the corresponding integrated network. Compared with previous solutions, due to the integrity of the integrated model introduced in this application, the problem of word segmentation errors leading to the accumulation of part-of-speech tagging errors is avoided due to the two-stage execution of the domain word segmentation and part-of-speech tagging tasks, and the corresponding Tibetan word segmentation and part-of-speech tagging can be processed more accurately, further increasing the practicality of the solution.

[0044] The present invention also provides an integrated Tibetan word segmentation and part-of-speech tagging system, an electronic device and a computer-readable storage medium. Since the above-mentioned integrated Tibetan word segmentation and part-of-speech tagging system, electronic device and computer-readable storage medium correspond to the above-mentioned integrated Tibetan word segmentation and part-of-speech tagging method, the beneficial effects are the same as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 A flowchart of a method for integrating Tibetan word segmentation and part-of-speech tagging provided in an embodiment of the present application;

[0047] Figure 2A display diagram of training data provided in an embodiment of the present application;

[0048] Figure 3 A structural diagram of an integrated model framework provided for an embodiment of the present application;

[0049] Figure 4 A flowchart of a Tibetan word segmentation and part-of-speech tagging task provided in an embodiment of the present application;

[0050] Figure 5 A flowchart of a method for processing adhesive words provided in an embodiment of the present application;

[0051] Figure 6 A structural diagram of a Tibetan word segmentation and part-of-speech tagging integrated system provided in an embodiment of the present application;

[0052] Figure 7 A structural block diagram of an electronic device provided by an embodiment of the present invention.

[0053] Icons: 1. Acquisition module; 2. Prediction module; 3. Calling module; 4. Labeling module; 101. Memory; 102. Processor; 103. Communication interface. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0055] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0056] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0057] Example

[0058] See also Figure 1 , Figure 1 A flowchart of a method for integrating Tibetan word segmentation and part-of-speech tagging provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method comprises the following steps:

[0059] S10: Acquire the Tibetan text information to be segmented input by the user, and acquire each Tibetan syllable and the corresponding label of the Tibetan text information;

[0060] It should be noted that in this embodiment, the specific Tibetan text information content is not limited. It can be understood that, for Tibetan, Tibetan part-of-speech tagging is performed after Tibetan word segmentation. In this embodiment, the matching method of Tibetan syllables and labels is not specifically limited. The Tibetan word segmentation label uses the four BMES labels, and the Tibetan part-of-speech tag label uses the tag code mentioned in the "Tibetan Speech Class Tag Set for Information Processing". The part before "-" indicates the position of the syllable in the word, and the part after "-" indicates the part of speech. For example, the sentence The labels for each syllable are B-nq, E-nq, S-ww.

[0061] S11: calling the integrated model and using the phonetic node and Unicode encoding to segment the Tibetan syllables and non-Tibetan character blocks and sorting them to obtain Tibetan sequences;

[0062] It can be understood that the integrated model is a Tibetan word segmentation and part-of-speech tagging learning model pre-established based on Conformer. In this embodiment, the specific establishment method of the integrated model is not limited, and the specific segmentation method of Tibetan syllables and non-Tibetan character blocks is not specifically limited.

[0063] S12: call the integrated model to perform CRF prediction on the Tibetan sequence and the corresponding labels to obtain the corresponding predicted label sequence between each Tibetan syllable in the Tibetan sequence;

[0064] The Tibetan sequence is a sequence sorted based on Tibetan syllables. In this embodiment, there is no specific limitation on CRF prediction and related optimal label prediction. It can be understood that the label prediction in this embodiment is the annotation of the content of specific Tibetan syllables, that is, the corresponding part of speech tagging is performed through labels.

[0065] S13: Arrange the writing forms of each Tibetan syllable according to the predicted label sequence to obtain corresponding annotation results.

[0066] Through the analysis of experimental results, Table 1 is a comparison table of the accuracy of the integrated Tibetan word segmentation results, and Table 2 is a comparison table of the accuracy of the integrated Tibetan part-of-speech tagging results. As shown in Tables 1 and 2, the word segmentation effect and part-of-speech tagging effect of the integrated model are better than those of independent word segmentation and part-of-speech tagging. This shows that the integrated model can more fully consider the dependency between word segmentation tags and part-of-speech tagging information, effectively combine Tibetan word segmentation and part-of-speech tagging, and thus improve the overall performance of Tibetan word segmentation and part-of-speech tagging.

[0067] Table 1: Comparison of the accuracy of the integrated Tibetan word segmentation results:

[0068]

[0069] Table 2: Comparison of the accuracy of integrated Tibetan part-of-speech tagging results:

[0070]

[0071]

[0072] This embodiment proposes an integrated method for Tibetan word segmentation and part-of-speech tagging, by obtaining the Tibetan text information to be segmented input by the user, and obtaining each Tibetan syllable and the corresponding label of the Tibetan text information, calling the integrated model and using the sound node and Unicode encoding to segment the Tibetan syllables and non-Tibetan character blocks, calling the integrated model for CRF prediction, obtaining the corresponding label relationship between any Tibetan sequence of the Tibetan text, and performing global optimal label prediction on the Tibetan syllables, the Tibetan sequence is a sequence sorted based on the Tibetan syllables, and the writing form of each Tibetan syllable is sorted according to the result of the label prediction to obtain the corresponding tagging result. In this application, by proposing to establish a corresponding integrated model, Tibetan word segmentation and part-of-speech tagging are performed based on the corresponding integrated network. Compared with previous solutions, due to the integrity of the integrated model introduced in this application, the problem of word segmentation errors leading to the accumulation of part-of-speech tagging errors is avoided due to the two-stage execution of the domain word segmentation and part-of-speech tagging tasks, and the corresponding Tibetan word segmentation and part-of-speech tagging can be processed more accurately, further increasing the practicality of the solution.

[0073] The above embodiments do not limit the establishment of the integrated model. By way of example, in some embodiments of the present application, the establishment of the integrated model includes the following steps:

[0074] Read the Tibetan training corpus input by the user and obtain each Tibetan syllable and its corresponding label;

[0075] Construct an integrated model framework for Tibetan word segmentation and part-of-speech tagging based on Conformer;

[0076] The relevant parameters of the Conformer-based Tibetan word segmentation and part-of-speech tagging integrated model are obtained through training and saved, and the corresponding integrated model is generated based on the relevant parameters.

[0077] In this embodiment, a specific method for establishing an integrated model is proposed, which is trained by pre-input training corpus and the built framework. Figure 2A display diagram of training data provided in an embodiment of the present application, through the above-mentioned data training, a corresponding integrated model is finally generated, thereby further increasing the completeness of the solution.

[0078] The above embodiments do not specifically describe how to build a corresponding integrated model framework. For example, in some embodiments of the present application, building an integrated model framework of Tibetan word segmentation and part-of-speech tagging based on Conformer includes:

[0079] The Embedding layer is built based on the encoding part of the Transformer model. The Embedding layer is used to obtain the input Tibetan sequence to form the corresponding embedding vector;

[0080] A decoding layer is built based on the CRF model. The decoding layer is used to count the probability P of each Tibetan syllable corresponding to a label in the text and calculate the optimal label category based on the constraint relationship of the sequence label.

[0081] An integrated model framework for Tibetan word segmentation and part-of-speech tagging is established based on the Embedding layer and Decoding layer.

[0082] For ease of understanding, Figure 3 A structural diagram of an integrated model framework provided in an embodiment of the present application, such as Figure 3 As mentioned above, the CNN+Transformer+CRF model framework is mainly used. It should be noted that the input Tibetan sequence Embedding layer forms the corresponding embedding vector, the Encoding layer uses the encoding part of the Transformer model, and the Decoding still uses the CRF model. For each input sequence, it is first encoded and input into the Transformer. The encoding layer of the Transformer includes a vector embedding layer and a position embedding layer. At the same time, the self-attention mechanism enables the model to pay attention to semantic information at different levels, which helps to establish the semantic connection of the context, and then inputs it into the conditional random field. The CRF layer counts the probability P of each syllable corresponding to the label in the text, and calculates the optimal label category according to the constraint relationship of the sequence label. Through the above scheme, the corresponding integrated model can be built. On the basis of using Transformer to learn the context information of the sentence, the CNN network structure is introduced so that the model can learn the local feature information in the text more fully. The present invention constructs a CNN+Transformer+CRF model for Tibetan word segmentation and Tibetan part-of-speech tagging.

[0083] Considering the subsequent specification of the model, illustratively, in some embodiments of the present application, after reading the Tibetan training corpus input by the user and obtaining each Tibetan syllable and the corresponding label, the following is further included:

[0084] Obtain a dataset of Tibetan training corpus, and perform feature extraction on each Tibetan syllable and its adjacent syllables in the dataset to obtain a fused syllable vector representing a feature word;

[0085] A corresponding word vector matrix is ​​established based on the feature words, and the word vector matrix is ​​used to standardize the corresponding integrated model.

[0086] In this embodiment, there is no specific limitation on the method of establishing the data set and the corresponding word vector matrix. It should be noted that through the above scheme, fused feature words can be extracted, thereby further optimizing the model and increasing the accuracy of the method.

[0087] In the above embodiments, the specific CRF prediction method for obtaining the corresponding labels is not limited. For example, in some embodiments of the present application, the integrated model is called to perform CRF prediction on the Tibetan sequence and the corresponding labels to obtain the corresponding predicted label sequence between the Tibetan syllables in the Tibetan sequence, including:

[0088] Get the corresponding Tibetan sequence based on Tibetan syllables where x i In the Tibetan word segmentation task, it represents a syllable, and in the Tibetan part-of-speech tagging task, it represents a Tibetan word;

[0089] Input the Tibetan sequence into the Embedding layer to generate an embedding vector

[0090] Input the embedding vector into the CNN network to fully learn the local features of the syllable or word, convert the syllable or word into a local semantic vector, and concatenate the current syllable or word vector in the local semantic vector;

[0091] The concatenated vector is input into the CRF layer to classify and predict the label and decode the data. And the state sequence corresponding to the predicted syllable Output the optimal label category, where y i Represents the label corresponding to each Tibetan syllable or Tibetan word.

[0092] For ease of understanding, Figure 4 A flowchart of a Tibetan word segmentation and part-of-speech tagging task provided in an embodiment of the present application, as described in the above embodiment and the specific Figure 4 The content in the above can be understood, and the embodiment of the present invention defines the optimal label output through the corresponding sequence, such as Figure 4The content shown in is an example. The model combines the Tibetan word segmentation and part-of-speech tagging labels. The label consists of the word segmentation label and the part-of-speech label. The integrated result only shows the part-of-speech label, and the part-of-speech label is connected with " / " after the syllable with the word segmentation label E or S. For example, for the sentence (official shelf)", its input sequence Its tag sequence Then its segmentation result is This further increases the specificity of the solution by performing CRF prediction and obtaining corresponding labels between any Tibetan sequences of Tibetan text, thereby increasing the completeness of the solution.

[0093] Since there are still binding words in Tibetan, Figure 5 A flowchart of a method for processing a bonding word provided in an embodiment of the present application. By way of example, in some embodiments of the present application, if the Tibetan text information includes a bonding word, the method further includes:

[0094] Using the syllable node "·" as a separator, the input sentence is segmented to obtain corresponding segmented syllables, where the segmented syllables are the parts between every two syllable nodes;

[0095] Traverse each syllable from back to front to determine the last two characters of each syllable;

[0096] If the last two characters of the syllable are Then the segmented syllable is marked as a pseudo-glued word, and the pseudo-glued word is matched with the first dictionary. If the pseudo-glued word is not in the first dictionary, it is determined that the segmented syllable is not a glued word; if the pseudo-glued word is in the first dictionary, it is determined that the segmented syllable is a glued word;

[0097] If the last character of the syllable is If one of the following is found, the segmented syllable is marked as a pseudo-grafted word, and the pseudo-grafted word is segmented from the function word part, and the first half is matched with the second dictionary. If the first half is found in the second dictionary, the The second half is converted. If it is not in the second dictionary, the first half remains unchanged and the second half is converted.

[0098] Match the proposed glue-writing word with the third dictionary. If the proposed glue-writing word is not in the third dictionary, determine that the proposed glue-writing word is not a glue-writing word; if the proposed glue-writing word is in the third dictionary, match the proposed glue-writing word with the fourth dictionary. If the proposed glue-writing word is in the fourth dictionary, return to the step of segmenting the proposed glue-writing word from the function word part and matching the first half with the second dictionary. If the proposed glue-writing word is in the fourth dictionary, combine the proposed glue-writing word with the previous syllable and match the fifth dictionary. If the combined proposed glue-writing word is in the fifth dictionary, return to the step of segmenting the proposed glue-writing word from the function word part and matching the first half with the second dictionary. If the combined proposed glue-writing word is not in the fifth dictionary, determine that the proposed glue-writing word is not a glue-writing word;

[0099] The first dictionary contains The dictionary contains 4420 words in total for all the four function words. The second dictionary contains 48 Tibetan structures, which is used to determine whether the word needs to be transformed. The third dictionary contains A dictionary of all pseudo-glued words for two function words, totaling 2201 words. The fourth dictionary contains The fifth dictionary contains 4 words for the two function words. A dictionary of 9 non-aggregated words composed of two function words and the previous syllable.

[0100] In this embodiment, a solution for processing corresponding adhesive words is proposed through corresponding dictionary matching, which further improves the processing width of the model and increases the completeness and practicality of the solution.

[0101] Based on the same inventive concept, the present application also provides a system, Figure 6 A structure diagram of a Tibetan word segmentation and part-of-speech tagging integrated system provided in an embodiment of the present application, such as Figure 6 As shown, the system includes:

[0102] Acquisition module 1 is used to acquire the Tibetan text information to be segmented input by the user, and acquire each Tibetan syllable and corresponding label of the Tibetan text information;

[0103] Calling module 2, for calling the integrated model and using phonetic nodes and Unicode encoding to segment Tibetan syllables and non-Tibetan character blocks and sort them to obtain Tibetan sequences. The integrated model is a Tibetan word segmentation and part-of-speech tagging integrated model based on Conformer;

[0104] Prediction module 3 is used to call the integrated model to perform CRF prediction on the Tibetan sequence and the corresponding label to obtain the corresponding predicted label sequence between each Tibetan syllable in the Tibetan sequence;

[0105] The annotation module 4 is used to sort out the writing forms of each Tibetan syllable according to the predicted label sequence to obtain the corresponding annotation results.

[0106] The specific embodiments of the above system and the corresponding beneficial effects are shown in the method section and will not be described in detail here.

[0107] See also Figure 7 , Figure 7 A structural block diagram of an electronic device provided in an embodiment of the present invention. The electronic device includes a memory 101, a processor 102 and a communication interface 103. The memory 101, the processor 102 and the communication interface 103 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, such as program instructions / modules corresponding to a Tibetan word segmentation and part-of-speech tagging integrated system provided in an embodiment of the present application. The processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used to communicate signaling or data with other node devices.

[0108] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0109] The processor 102 may be an integrated circuit chip with signal processing capability. The processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0110] Understandably, Figure 7The structure shown is for illustration only. The electronic device may also include Figure 7 More or fewer components as shown, or with Figure 7 Different configurations are shown. Figure 7 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0111] Finally, the present application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.

[0112] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0113] In summary, the embodiments of the present application provide an electronic device and a computer-readable storage medium. The specific embodiments and corresponding beneficial effects are described in the above method section and will not be elaborated here.

[0114] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for integrating Tibetan word segmentation and part-of-speech tagging, characterized in that: The following steps are involved: Obtaining Tibetan text information to be segmented input by a user, and obtaining each Tibetan syllable and corresponding label of the Tibetan text information; Calling the integrated model and using the sound node and Unicode encoding to segment the Tibetan syllables and non-Tibetan character blocks and sorting them to obtain a Tibetan sequence, wherein the integrated model is a Tibetan word segmentation and part-of-speech tagging learning model pre-established based on Conformer; The integrated model is called to perform CRF prediction on the Tibetan sequence and the corresponding label to obtain a corresponding predicted label sequence between each Tibetan syllable in the Tibetan sequence; Arranging the writing forms of each of the Tibetan syllables according to the predicted label sequence to obtain corresponding annotation results; If the Tibetan text information includes a binder word, the method further includes: Using the syllable "·" as a separator, the input sentence is segmented to obtain corresponding segmented syllables, wherein the segmented syllables are the parts between every two syllables; Traversing each of the syncopated syllables from back to front to determine the last two characters of each of the syncopated syllables; If the last two characters of the syllable are Then the segmented syllable is marked as a pseudo-bonded word, and the pseudo-bonded word is matched with a first dictionary. If the pseudo-bonded word is not in the first dictionary, it is determined that the segmented syllable is not a bonded word; if the pseudo-bonded word is in the first dictionary, it is determined that the segmented syllable is a bonded word; If the last character of the syllable is The segmented syllable is marked as a pseudo-grafted word, the pseudo-grafted word is segmented from the function word part, the first half is matched with the second dictionary, and if the first half is supplemented in the second dictionary, The second half is converted. If it is not in the second dictionary, the first half remains unchanged and the second half is converted. Match the simulated sticky word with the third dictionary. If the simulated sticky word is not in the third dictionary, determine that the simulated sticky word is not a sticky word; if the simulated sticky word is in the third dictionary, match the simulated sticky word with the fourth dictionary. If the simulated sticky word is in the fourth dictionary, return to the step of segmenting the simulated sticky word from the function word part and matching the first half with the second dictionary. If the simulated sticky word is in the fourth dictionary, combine the simulated sticky word with the previous syllable and match the fifth dictionary. If the combined simulated sticky word is in the fifth dictionary, return to the step of segmenting the simulated sticky word from the function word part and matching the first half with the second dictionary. If the combined simulated sticky word is not in the fifth dictionary, determine that the simulated sticky word is not a sticky word; The first dictionary contains The dictionary of all the four function words with a total of 4420 words, the second dictionary contains 48 Tibetan structures, which is used to determine whether the word needs to be transformed, and the third dictionary contains A dictionary of 2201 words to be written in two function words, wherein the fourth dictionary contains A dictionary of four words for gluing two function words, wherein the fifth dictionary contains A dictionary of 9 non-aggregated words composed of two function words and the previous syllable.

2. The method for integrating Tibetan word segmentation and part-of-speech tagging as claimed in claim 1, characterized in that: The establishment of the integrated model includes the following steps: Read the Tibetan training corpus input by the user and obtain each Tibetan syllable and its corresponding label; Construct an integrated model framework for Tibetan word segmentation and part-of-speech tagging based on Conformer; The relevant parameters of the integrated model of Tibetan word segmentation and part-of-speech tagging based on Conformer are obtained through training and saved, and the corresponding integrated model is generated based on the relevant parameters.

3. The method for integrating Tibetan word segmentation and part-of-speech tagging as claimed in claim 2, characterized in that: The Conformer-based model framework for building an integrated Tibetan word segmentation and part-of-speech tagging model includes: An Embedding layer is built based on the encoding part of the Transformer model. The Embedding layer is used to obtain the input Tibetan sequence to form a corresponding embedding vector; A Decoding layer is built based on the CRF model. The Decoding layer is used to count the probability P of each Tibetan syllable corresponding to a label in the text, and calculate the optimal label category according to the constraint relationship of the sequence label; The Tibetan word segmentation and part-of-speech tagging integrated model framework is established based on the Embedding layer and the Decoding layer.

4. The method for integrating Tibetan word segmentation and part-of-speech tagging as claimed in claim 3, characterized in that: After reading the Tibetan training corpus input by the user and obtaining each Tibetan syllable and the corresponding label, the method further includes: Acquire a data set of the Tibetan training corpus, and perform feature extraction on each Tibetan syllable and its adjacent syllables in the data set to obtain a fused syllable vector representing a feature word; A corresponding word vector matrix is ​​established based on the feature words, and the word vector matrix is ​​used to standardize the corresponding integrated model.

5. The method for integrating Tibetan word segmentation and part-of-speech tagging as claimed in claim 4, characterized in that: The step of calling the integrated model to perform CRF prediction on the Tibetan sequence and the corresponding labels to obtain the corresponding predicted label sequence between the Tibetan syllables in the Tibetan sequence includes: Based on the Tibetan syllables, a corresponding Tibetan sequence is obtained where x i In the Tibetan word segmentation task, it represents a syllable, and in the Tibetan part-of-speech tagging task, it represents a Tibetan word; Input the Tibetan sequence into the Embedding layer to generate an embedding vector Input the embedding vector into the CNN network to fully learn the local features of the syllable or word, convert the syllable or word into a local semantic vector, and concatenate the current syllable or word vector in the local semantic vector; The concatenated vector is input into the CRF layer to perform classification prediction and data decoding operations on the label. And the state sequence corresponding to the predicted syllable Output the optimal label category, where y i Represents the label corresponding to each Tibetan syllable or Tibetan word.

6. A Tibetan word segmentation and part-of-speech tagging integrated system, characterized in that: The system comprises: An acquisition module is used to acquire the Tibetan text information to be segmented input by the user, and acquire each Tibetan syllable and the corresponding label of the Tibetan text information; A calling module, used to call the integrated model and segment the Tibetan syllables and non-Tibetan character blocks using sound nodes and Unicode encoding and sort them to obtain Tibetan sequences, wherein the integrated model is a Tibetan word segmentation and part-of-speech tagging integrated model based on Conformer; A prediction module, used to call the integrated model to perform CRF prediction on the Tibetan sequence and the corresponding label, and obtain a corresponding predicted label sequence between each Tibetan syllable in the Tibetan sequence; A labeling module, used for arranging the writing forms of each Tibetan syllable according to the predicted label sequence to obtain corresponding labeling results; If the Tibetan text information includes a binding word; Using the syllable "·" as a separator, the input sentence is segmented to obtain corresponding segmented syllables, wherein the segmented syllables are the parts between every two syllables; Traversing each of the syncopated syllables from back to front to determine the last two characters of each of the syncopated syllables; If the last two characters of the syllable are Then the segmented syllable is marked as a pseudo-bonded word, and the pseudo-bonded word is matched with a first dictionary. If the pseudo-bonded word is not in the first dictionary, it is determined that the segmented syllable is not a bonded word; if the pseudo-bonded word is in the first dictionary, it is determined that the segmented syllable is a bonded word; If the last character of the syllable is The segmented syllable is marked as a pseudo-grafted word, the pseudo-grafted word is segmented from the function word part, the first half is matched with the second dictionary, and if the first half is supplemented in the second dictionary, The second half is converted. If it is not in the second dictionary, the first half remains unchanged and the second half is converted. Match the simulated sticky word with the third dictionary. If the simulated sticky word is not in the third dictionary, determine that the simulated sticky word is not a sticky word; if the simulated sticky word is in the third dictionary, match the simulated sticky word with the fourth dictionary. If the simulated sticky word is in the fourth dictionary, return to the step of segmenting the simulated sticky word from the function word part and matching the first half with the second dictionary. If the simulated sticky word is in the fourth dictionary, combine the simulated sticky word with the previous syllable and match the fifth dictionary. If the combined simulated sticky word is in the fifth dictionary, return to the step of segmenting the simulated sticky word from the function word part and matching the first half with the second dictionary. If the combined simulated sticky word is not in the fifth dictionary, determine that the simulated sticky word is not a sticky word; The first dictionary contains The dictionary of all the four function words with a total of 4420 words, the second dictionary contains 48 Tibetan structures, which is used to determine whether the word needs to be transformed, and the third dictionary contains A dictionary of 2201 words to be written in two function words, wherein the fourth dictionary contains A dictionary of four words for gluing two function words, wherein the fifth dictionary contains A dictionary of 9 non-aggregated words composed of two function words and the previous syllable.

7. An electronic device, characterized in that: include: A memory for storing one or more programs; processor; When the one or more programs are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images