A method, system, medium and product for evaluating the word segmentation effect of a low-resource language

Abstract

The application discloses a low-resource language segmentation effect evaluation method, system, medium and product, and relates to the technical field of data processing. In the method, firstly, unannotated monolingual corpus and bilingual parallel corpus of a target language are acquired, and a character co-occurrence affinity matrix, a forced segmentation boundary position set, a morphological pattern rule library and a cross-language translation probability reference table are constructed according to the corpus. Then, test corpus is input into a segmentation evaluator to be evaluated to obtain candidate segmentation results and boundary position sets. Through combination of the matrix, the set, the rule library and the reference table constructed in the foregoing, boundary compliance inspection and word compliance inspection are respectively performed on the candidate results to obtain a boundary compliance rate, a word compliance rate and corresponding error sets. Finally, a comprehensive evaluation score is calculated based on the compliance rates, and an error positioning report is generated in combination with the error sets. The technical scheme provided by the application can improve the accuracy of low-resource language segmentation effect evaluation.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a method, system, medium, and product for evaluating the effectiveness of low-resource language segmentation. Background Technology

[0002] With the deepening of globalization and informatization, natural language processing (NLP) technology is playing an increasingly important role in machine translation, information retrieval, and text analysis. Word segmentation, as a fundamental step in NLP, directly impacts the effectiveness of subsequent tasks. For languages ​​without clear delimiters, such as Chinese, Tibetan, Uyghur, and Mongolian, accurate word segmentation is particularly crucial. To meet the needs of different language characteristics, application scenarios, and performance requirements, word segmenter evaluation technologies are showing diversified development trends. Their accuracy and reliability in low-resource language environments are of great significance for promoting the informatization of related languages ​​and improving the efficiency of word segmenter development.

[0003] Currently, the main method for evaluating word segmentation performance is comparative evaluation based on labeled test sets. This method compares the output of the word segmenter under evaluation with a manually labeled gold standard to determine the accuracy of the segmentation. This label-based comparison method quantifies the performance of the word segmenter by calculating metrics such as precision and recall between the segmentation results and the standard answer. It can directly reflect the basic accuracy of the word segmenter and has been widely used in the evaluation of word segmenters for high-resource languages ​​such as Chinese and English.

[0004] However, due to the diversity of language resources and the limited scale of annotations, the evaluation scenarios for word segmenters in low-resource languages ​​are becoming increasingly complex and varied. In practical applications, evaluation methods based on annotated test sets struggle to suppress the interference of insufficient annotated data size and lack of annotation consistency on the evaluation results, and the accuracy of word segmentation performance evaluation for low-resource languages ​​is difficult to guarantee. Particularly for low-resource languages ​​such as Tibetan, Uyghur, and Mongolian, the limited size of available annotated test sets and the potential differences in the understanding of segmentation boundaries among different annotators make existing methods, which use small-scale or inconsistent annotated data for evaluation, prone to biased or even incorrect evaluation results, thus reducing the accuracy of word segmentation performance evaluation for low-resource languages. Summary of the Invention

[0005] This application provides a method, system, medium, and product for evaluating the effectiveness of low-resource language segmentation, which can improve the accuracy of evaluating the effectiveness of low-resource language segmentation.

[0006] The first aspect of this application provides a method for evaluating the performance of low-resource language segmentation, including: Obtain unlabeled monolingual and bilingual parallel corpora of the target language; Based on the unlabeled monolingual corpus, a character co-occurrence affinity matrix and a forced word segmentation boundary position set are constructed, and a morphological pattern rule base and a cross-language translation probability reference table are constructed based on the bilingual parallel corpus. Input the test corpus into the word segmenter to be evaluated to obtain the candidate word segmentation result sequence and the candidate boundary position set; The candidate boundary position set is checked for boundary compliance by combining the forced word segmentation boundary position set and the character co-occurrence affinity matrix to obtain the boundary compliance rate and the boundary error set; The candidate word segmentation result sequence is subjected to word compliance verification using the morphological pattern rule base and the cross-language translation probability reference table to obtain the word compliance rate and word error set; A comprehensive evaluation score is calculated based on the boundary compliance rate and the word compliance rate, and an error location report is generated by combining the comprehensive evaluation score, the boundary error set, and the word error set.

[0007] By adopting the above technical solution, an evaluation reference system is established that does not rely on a large-scale manually annotated test set by acquiring unlabeled monolingual corpora and bilingual parallel corpora of the target language. Based on the unlabeled monolingual corpora, a character co-occurrence affinity matrix and a forced segmentation boundary position set are constructed. Based on the bilingual parallel corpora, a morphological pattern rule base and a cross-language translation probability reference table are constructed. On this basis, the test corpus is input into the segmenter to be evaluated to obtain candidate segmentation result sequences and candidate boundary position sets. The boundary compliance of the candidate boundary position sets is then checked by combining the forced segmentation boundary position set and the character co-occurrence affinity matrix. This allows for the identification of errors that violate explicit boundary rules or do not conform to character co-occurrence patterns from the perspective of segmentation boundary positions, yielding the boundary compliance rate and boundary error set. Simultaneously, the word compliance of the candidate segmentation result sequences is checked by using the morphological pattern rule base and the cross-language translation probability reference table. This allows for the identification of errors that do not conform to word formation rules or lack cross-language correspondences from the perspective of word morphology, yielding the word compliance rate and word error set. By cross-validating the boundary and word dimensions, a comprehensive evaluation score is calculated based on the boundary compliance rate and word compliance rate. An error localization report is generated by combining the comprehensive evaluation score, boundary error set, and word error set. This approach enables multi-dimensional verification of word segmentation results using linguistic rules automatically extracted from the corpus, even with limited labeled data. It effectively suppresses the interference of insufficient labeled data and lack of label consistency on the evaluation results, thereby improving the accuracy of low-resource language word segmentation effect evaluation.

[0008] Optionally, the co-occurrence probability and single-character occurrence probability of adjacent character pairs in the unlabeled monolingual corpus are statistically analyzed; the co-occurrence probability is divided by the product of the occurrence probabilities of the two single characters constituting the adjacent character pair and the logarithm is taken to obtain the mutual information value of the adjacent character pair, and the mutual information values ​​of all adjacent character pairs are combined into the character co-occurrence affinity matrix; target adjacent character pairs with mutual information values ​​less than a preset breakpoint threshold are extracted from the character co-occurrence affinity matrix, and the interval positions of the target adjacent character pairs in the unlabeled monolingual corpus are obtained to form the forced word segmentation boundary position set.

[0009] Optionally, positions in the mandatory segmentation boundary position set that do not appear in the candidate boundary position set are recorded as mandatory boundary missing errors, included in the boundary error set, and the mandatory boundary compliance rate is calculated; for non-mandatory boundary positions in the candidate boundary position set after excluding mandatory boundaries, the mutual information value of adjacent characters in the character co-occurrence affinity matrix is ​​queried, and boundary positions with mutual information values ​​greater than a preset affinity threshold are recorded as affinity violation errors, included in the boundary error set, and the affinity compliance rate is calculated; the mandatory boundary compliance rate and the affinity compliance rate are used as the boundary compliance rate.

[0010] Optionally, the target affix combination patterns and target strings with a frequency greater than a preset frequency threshold are extracted from the target language side of the bilingual parallel corpus to construct the morphological pattern rule base; the morphological pattern rule base is used as a dictionary to perform word segmentation preprocessing on the bilingual parallel corpus to obtain target language words and reference language words; the co-occurrence frequency of the target language words and the reference language words in the same sentence pair in the bilingual parallel corpus, as well as the independent occurrence frequency of the target language words in the bilingual parallel corpus, are counted; the co-occurrence frequency is divided by the independent occurrence frequency to obtain the translation probability of the target language words and the reference language words; target language words with a translation probability greater than a preset semantic verification threshold are selected and combined with reference language words into word pairs and stored in a template reference table to obtain the cross-language translation probability reference table.

[0011] Optionally, verify whether each word in the candidate word segmentation result sequence conforms to the target affix combination pattern and target string in the morphological pattern rule base. Record non-conforming words as morphological violation errors, add them to the word error set, and calculate the morphological compliance rate. Query whether each word in the candidate word segmentation result sequence has a corresponding translation word that meets the preset semantic verification threshold in the cross-language translation probability reference table. Record non-conforming words as semantic fragmentation errors, add them to the word error set, and calculate the semantic integrity rate. Use the morphological compliance rate and the semantic integrity rate as the word compliance rate.

[0012] Optionally, obtain the weighting coefficients of the boundary compliance rate and the word compliance rate; perform a weighted calculation on the boundary compliance rate and the word compliance rate based on the weighting coefficients to obtain the comprehensive evaluation score; restore the error positions of the boundary error set and the word error set to the original text of the test corpus and label the corresponding error types; calculate the product of a preset benchmark threshold and the comprehensive evaluation score, use the product as the error analysis threshold, and count the proportion of each error type; mark the error types whose proportion exceeds the error analysis threshold as target defects, summarize the target defects, the error positions in the original text, and the comprehensive evaluation score to generate the error location report.

[0013] Optionally, the target defect and its corresponding target error location are extracted from the error location report; a context fragment is extracted based on the target error location, and a user verification task is generated in conjunction with the target defect; the user verification task is sent to the user terminal, and the corrected text returned by the user terminal is received; target segmentation terms are extracted based on the corrected text and the context fragment; the target segmentation terms are synchronized to the segmentation dictionary of the segmenter to be evaluated, so as to update the segmenter to be evaluated.

[0014] Secondly, embodiments of this application provide a low-resource language segmentation performance evaluation system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, which includes computer instructions, and the one or more processors call the computer instructions to cause the low-resource language segmentation performance evaluation system to perform the method described in the first aspect and any possible implementation thereof.

[0015] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a low-resource language segmentation effect evaluation system, cause the low-resource language segmentation effect evaluation system to perform the method described in the first aspect and any possible implementation thereof.

[0016] Fourthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a low-resource language segmentation effect evaluation system, cause the low-resource language segmentation effect evaluation system to perform the method described in the first aspect and any possible implementation thereof.

[0017] In summary, one or more technical solutions provided in this application have at least the following technical effects or advantages: By adopting the above technical solution, an evaluation reference system is established that does not rely on a large-scale manually annotated test set by acquiring unlabeled monolingual corpora and bilingual parallel corpora of the target language. Based on the unlabeled monolingual corpora, a character co-occurrence affinity matrix and a forced segmentation boundary position set are constructed. Based on the bilingual parallel corpora, a morphological pattern rule base and a cross-language translation probability reference table are constructed. On this basis, the test corpus is input into the segmenter to be evaluated to obtain candidate segmentation result sequences and candidate boundary position sets. The boundary compliance of the candidate boundary position sets is then checked by combining the forced segmentation boundary position set and the character co-occurrence affinity matrix. This allows for the identification of errors that violate explicit boundary rules or do not conform to character co-occurrence patterns from the perspective of segmentation boundary positions, yielding the boundary compliance rate and boundary error set. Simultaneously, the word compliance of the candidate segmentation result sequences is checked by using the morphological pattern rule base and the cross-language translation probability reference table. This allows for the identification of errors that do not conform to word formation rules or lack cross-language correspondences from the perspective of word morphology, yielding the word compliance rate and word error set. By cross-validating the boundary and word dimensions, a comprehensive evaluation score is calculated based on the boundary compliance rate and word compliance rate. An error localization report is generated by combining the comprehensive evaluation score, boundary error set, and word error set. This approach enables multi-dimensional verification of word segmentation results using linguistic rules automatically extracted from the corpus, even with limited labeled data. It effectively suppresses the interference of insufficient labeled data and lack of label consistency on the evaluation results, thereby improving the accuracy of low-resource language word segmentation effect evaluation. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the evaluation process for low-resource language segmentation disclosed in the embodiments of this application; Figure 2 This is another flowchart illustrating the evaluation of low-resource language segmentation performance disclosed in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of a system provided in an embodiment of this application.

[0019] Explanation of reference numerals in the attached drawings: 301, Central Processing Unit; 302, Read-Only Memory; 303, Random Access Memory; 304, Bus; 305, Input / Output Interface; 306, Input Section; 307, Output Section; 308, Storage Section; 309, Communication Section; 310, Driver; 311, Removable Media. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0021] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0022] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0023] This application provides a method for evaluating the performance of low-resource language segmentation, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a method for evaluating the effectiveness of low-resource language segmentation according to an embodiment of this application. The method is applied to a system, which refers to a hardware and software integrated platform capable of executing an evaluation program for low-resource language segmentation. The system can execute an evaluation program for low-resource language segmentation, and the method includes steps 101 to 106, as follows: Step 101: Obtain unlabeled monolingual corpora and bilingual parallel corpora of the target language.

[0024] In this application embodiment, the target language refers to the language that requires word segmentation effect evaluation, such as low-resource languages ​​like Tibetan, Uyghur, and Mongolian. Unlabeled monolingual corpora refer to raw text sets containing only target language text without any word segmentation annotation, such as news articles or online text. Bilingual parallel corpora refer to corresponding translation text pairs consisting of the target language and a resource-rich reference language (such as Chinese or English), where each target language sentence has a corresponding reference language translation.

[0025] Specifically, firstly, unlabeled text data in the target language is selected from a language resource database to ensure that the text covers multiple domains and styles, thereby enhancing the representativeness of the corpus. Then, text content in the target language is obtained through automated web crawling or public corpora, and the text is cleaned to remove special symbols, duplicate content, and noisy data. At the same time, parallel texts containing both the target language and reference language are collected from bilingual corpora or multilingual websites, and these texts are processed using sentence alignment techniques to ensure a one-to-one correspondence between sentences in the target language and sentences in the reference language. Finally, the processed unlabeled monolingual corpus and bilingual parallel corpus are stored in standard formats for subsequent processing.

[0026] Step 102: Construct a character co-occurrence affinity matrix and a forced word segmentation boundary position set based on unlabeled monolingual corpora, and construct a morphological pattern rule base and a cross-language translation probability reference table based on bilingual parallel corpora.

[0027] In this embodiment, the character co-occurrence affinity matrix represents a two-dimensional data structure that quantifies the tendency of adjacent characters to combine into words, reflecting the degree of cohesion between characters through mutual information values. The forced segmentation boundary position set refers to the set of character positions that must be segmented, where the probability of adjacent characters combining into words is extremely low. The morphological pattern rule base is a knowledge base containing target language-specific affix combination patterns and high-frequency strings, used to identify effective word structures. The cross-language translation probability reference table is a data table recording the correspondence and probabilities between target language words and reference language words, used to determine the semantic completeness of the segmentation results.

[0028] Specifically, firstly, all text in the unlabeled monolingual corpus is traversed, and the co-occurrence frequency of each pair of adjacent characters and the occurrence frequency of each individual character are counted. Then, the mutual information value of each pair of adjacent characters is calculated, which is the logarithm of the product of the occurrence probabilities of the two individual characters divided by the co-occurrence probability, forming a character co-occurrence affinity matrix. Next, a breakpoint threshold is set, and the positions of adjacent character pairs with mutual information values ​​less than the threshold are extracted in the corpus, forming a forced word segmentation boundary position set. At the same time, common affix combination patterns and high-frequency strings of the target language are extracted from the bilingual parallel corpus to construct a morphological pattern rule base. Subsequently, the rule base is used to perform word segmentation preprocessing on the bilingual parallel corpus, and the co-occurrence frequency of target language words and reference language words and the independent occurrence frequency of target language words are counted. Finally, the translation probability (co-occurrence frequency divided by independent occurrence frequency) is calculated, and word pairs with translation probabilities higher than the preset threshold are selected to establish a cross-language translation probability reference table.

[0029] In one possible implementation, a character co-occurrence affinity matrix and a forced word segmentation boundary position set are constructed based on an unlabeled monolingual corpus, specifically including steps 1021-1023, as follows: Step 1021: Calculate the co-occurrence probability of adjacent character pairs and the occurrence probability of a single character in the unlabeled monolingual corpus.

[0030] In this embodiment, adjacent character pairs represent two character combinations that appear consecutively in an unlabeled monolingual corpus. Co-occurrence probability refers to the ratio of the number of times a specific adjacent character pair appears in the corpus to the total number of all adjacent character pairs in the corpus, reflecting the frequency of use of that character pair in the language. Single character occurrence probability represents the ratio of the number of times a single character appears in the corpus to the total number of all characters in the corpus, reflecting the frequency of use of that character in the language.

[0031] Specifically, the unlabeled monolingual corpus is first preprocessed, including text normalization, removal of non-target language characters and special symbols, etc.; then, all text in the processed unlabeled monolingual corpus is traversed, scanning character by character; next, a character dictionary and an adjacent character pair dictionary are constructed, recording each occurrence of a character and its adjacent character pairs, respectively; then, the total number of occurrences of each adjacent character pair in the corpus is counted, and the co-occurrence probability of adjacent character pairs is calculated, i.e., the number of occurrences of the character pair divided by the total number of all adjacent character pairs in the corpus; simultaneously, the total number of occurrences of each single character in the corpus is counted, and the occurrence probability of the single character is calculated, i.e., the number of occurrences of the character divided by the total number of all characters in the corpus; finally, the statistical results are saved as a probability mapping table to prepare for subsequent mutual information value calculation.

[0032] Step 1022: Divide the co-occurrence probability by the product of the occurrence probabilities of the two single characters that constitute an adjacent character pair and take the logarithm to obtain the mutual information value of the adjacent character pair. Combine the mutual information values ​​of all adjacent character pairs into a character co-occurrence affinity matrix.

[0033] In this embodiment, mutual information value represents a statistical measure of the tendency of two characters to combine into a word. A positive value indicates that the two characters tend to combine together, while a negative value indicates that the two characters tend to separate. The character co-occurrence affinity matrix is ​​a two-dimensional data structure that records the mutual information values ​​of all possible character pairs. The rows and columns correspond to all characters in the language, and the intersections store the mutual information values ​​of the corresponding character pairs.

[0034] Specifically, first, all adjacent character pairs are traversed and counted. For each adjacent character pair, its co-occurrence probability and the occurrence probability of the two individual characters constituting the pair are extracted. Then, for each adjacent character pair, its mutual information value is calculated, which is the co-occurrence probability of the pair divided by the product of the occurrence probabilities of the two individual characters constituting the pair, and the natural logarithm of the result is taken. Next, a two-dimensional data structure is created, with the number of rows and columns equal to the number of different characters appearing in the corpus. The row index and column index correspond to the characters in the character set, respectively. Then, the calculated mutual information value of each adjacent character pair is filled into the corresponding matrix position, that is, the intersection of the row corresponding to the first character and the column corresponding to the second character. Finally, for character pairs that do not appear in the corpus, their mutual information value is set to the default value (usually negative infinity or zero), thus forming a complete character co-occurrence affinity matrix.

[0035] Step 1023: Extract target adjacent character pairs whose mutual information values ​​are less than the preset breakpoint threshold in the character co-occurrence affinity matrix, obtain the interval positions of the target adjacent character pairs in the unlabeled monolingual corpus, and form a forced word segmentation boundary position set.

[0036] In this embodiment, the preset breakpoint threshold represents the critical point of mutual information value used to determine whether adjacent characters should be forcibly separated, and is usually a negative number. A target adjacent character pair refers to a combination of characters whose mutual information value is lower than the preset breakpoint threshold, indicating that these two characters rarely combine to form a word, and should be segmented between them. The forced segmentation boundary position set refers to the set of positions between characters that must be segmented, representing explicit word boundaries in the language.

[0037] Specifically, firstly, a preset breakpoint threshold is set, which is usually obtained through analysis or statistics of the target language characteristics; then, all elements in the character co-occurrence affinity matrix are traversed, and character pairs with mutual information values ​​less than the preset breakpoint threshold are extracted and marked as target adjacent character pairs; next, the unlabeled monolingual corpus is rescanned to identify the position of each target adjacent character pair in the text; then, the position between the first and second characters in these target adjacent character pairs is recorded, that is, the interval position between the two characters; finally, all these interval positions are deduplicated to form a set of mandatory word segmentation boundary positions. This set contains all the position information that must be segmented, and each position consists of a document identifier and a character index, which is used for boundary compliance verification in subsequent word segmentation evaluation.

[0038] In one possible implementation, a morphological pattern rule base and a cross-language translation probability reference table are constructed based on a bilingual parallel corpus, specifically including steps 1024-1027, as follows: Step 1024: Extract the target affix combination patterns and target strings with a frequency greater than a preset frequency threshold from the target language side of the bilingual parallel corpus, and construct a morphological pattern rule base.

[0039] In this embodiment, the target affix combination pattern represents the combination rules of common word-forming units such as prefixes, suffixes, and roots in the target language. The preset frequency threshold refers to the critical value of occurrence frequency for determining whether a string is a valid word structure, usually a positive integer. The target string represents a sequence of characters that frequently appears in the target language side of the bilingual parallel corpus; these sequences are likely valid words or word-forming units in the target language. The morphological pattern rule base refers to a knowledge base containing target language-specific affix combination rules and high-frequency valid words, used to identify morphologically compliant word structures.

[0040] Specifically, the bilingual parallel corpus is first preprocessed to separate the target language text content. Then, linguistic knowledge is used to identify common affix types in the target language, including prefixes, suffixes, declension marks, and other word-forming units. Next, the combination patterns of these affixes in the actual text are extracted from the preprocessed target language text, and the collocation relationships and order between various affixes are recorded. Subsequently, n-gram analysis (where n is usually from 2 to 5) is performed on the target language text to count the frequency of occurrence of character sequences of various lengths in the corpus. Then, character sequences with a frequency greater than a preset frequency threshold are selected as target strings. Next, these target strings are deduplicated and merged to eliminate duplicate content and inclusion relationships. Finally, the extracted target affix combination patterns and the selected target strings are integrated into the same data structure. Each entry contains string content, type markers (affix patterns or high-frequency words), and frequency information, thereby constructing a morphological pattern rule base.

[0041] Step 1025: Use the morphological pattern rule base as a dictionary to perform word segmentation preprocessing on the bilingual parallel corpus to obtain target language words and reference language words.

[0042] In this embodiment, the dictionary represents a reference table used to perform word segmentation tasks, containing a predefined list of valid words. Word segmentation preprocessing refers to the process of dividing continuous text into independent word units before translation probability statistics. Target language words refer to independent word units identified from the target language text. Reference language words refer to independent word units identified from the reference language text, typically from resource-rich languages ​​such as Chinese or English, for which mature word segmentation tools already exist.

[0043] Specifically, the process begins by converting the morphological pattern rule base into a dictionary format suitable for the maximum matching algorithm, including affix combination patterns and high-frequency strings. Then, for the target language text in the bilingual parallel corpus, a maximum forward matching algorithm based on this dictionary is applied for preliminary word segmentation, i.e., starting from the beginning of the text, the longest matching string in the dictionary is found. Next, heuristic rules are applied to optimize the preliminary word segmentation results, handling segments that fail to match the dictionary and eliminating ambiguity. Subsequently, for the reference language text in the bilingual parallel corpus, existing mature word segmentation tools or methods are used for word segmentation, such as Jieba segmentation for Chinese and space segmentation with punctuation processing for English. Then, the segmented target language text and reference language text are standardized, including case unification, symbol processing, and whitespace normalization. Next, the processed target language words and reference language words are stored according to their correspondence in the original sentence pairs, preserving sentence pair boundary information. Finally, the processing results are saved as structured data, including sentence pair IDs, target language word sequences, and reference language word sequences, preparing for subsequent statistical translation probability analysis.

[0044] Step 1026: Calculate the co-occurrence frequency of target language words and reference language words in the same sentence pair in the bilingual parallel corpus, and the independent occurrence frequency of target language words in the bilingual parallel corpus.

[0045] In this embodiment, "same sentence pair" refers to a pair of corresponding target language sentences and reference language sentences in a bilingual parallel corpus. They are semantically equivalent and are translations of each other. Co-occurrence frequency refers to the cumulative number of times target language words and reference language words appear simultaneously in the same sentence pair, reflecting the strength of the association between the two words in the translation relationship. Independent occurrence frequency represents the total number of times the target language word appears in the target language side of the entire bilingual parallel corpus, reflecting the frequency of use of the word in the target language.

[0046] Specifically, the process begins by traversing the acquired bilingual parallel corpus after word segmentation, organizing target language words and reference language words according to sentence pair IDs. Then, two counters are created to record the co-occurrence frequency of word pairs and the independent occurrence frequency of target language words, respectively. Next, for each sentence pair, all target language words and reference language words are extracted. Then, for each target language word, its count in the independent occurrence frequency counter is incremented. Next, for each possible combination of target language words and reference language words, it is checked whether they appear in the same sentence pair; if so, the count of that word pair in the co-occurrence frequency counter is incremented. For larger corpora, batch processing and parallel computing methods are used to optimize the statistical process, reducing memory usage and improving processing speed. Finally, the statistical results are saved as two mapping tables: one recording the relationship between word pairs and their co-occurrence frequencies, and the other recording the relationship between target language words and their independent occurrence frequencies, preparing for the calculation of translation probabilities.

[0047] Step 1027: Divide the co-occurrence frequency by the independent occurrence frequency to obtain the translation probability of the target language word and the reference language word; filter the target language words and reference language words with a translation probability greater than the preset semantic verification threshold to form word pairs and store them in the template reference table to obtain the cross-language translation probability reference table.

[0048] In this embodiment, the translation probability represents the probability estimate of a target language word corresponding to a specific reference language word, calculated by dividing the co-occurrence frequency of the two words by the independent occurrence frequency of the target language word. The preset semantic verification threshold is a probability threshold for determining whether a translation relationship is valid, typically set between 0 and 1. A word pair represents a pairing of a target language word and a reference language word with a high translation probability relationship. The cross-language translation probability reference table is a data structure that records the translation probabilities between target language words and reference language words, used to evaluate the semantic integrity of the word segmentation results.

[0049] Specifically, the process begins by iterating through all target language and reference language word combinations. For each combination, its co-occurrence frequency and the independent occurrence frequency of the target language word are extracted. Next, the translation probability is calculated by dividing the co-occurrence frequency of the combination by the independent occurrence frequency of the target language word, resulting in a value between 0 and 1. The calculated translation probability is then compared to a preset semantic verification threshold, filtering out word combinations with a translation probability greater than the threshold. These high-translation-probability combinations are then constructed as word pairs, each containing a target language word, a reference language word, and the corresponding translation probability. A template reference table data structure is then created, indexing the selected word pairs according to the target language word for easy retrieval. Finally, the template reference table containing all high-translation-probability word pairs is named the Cross-Language Translation Probability Reference Table, serving as a reference for subsequent word compliance checks.

[0050] Step 103: Input the test corpus into the word segmenter to be evaluated to obtain the candidate word segmentation result sequence and the candidate boundary position set.

[0051] In this embodiment, the candidate word segmentation result sequence represents a list of word segmentation results output by the word segmenter to be evaluated after processing the test corpus, where each element is a string identified as an independent word. The candidate boundary position set refers to the set of boundary positions between each word in the candidate word segmentation results, marking the character positions that the word segmenter deems should be broken.

[0052] Specifically, first, a set of unsegmented test corpora is prepared, containing target language texts from various genres and domains. Then, the test corpora are input into the segmenter to be evaluated for processing, using the segmenter's standard interface or command-line tools to perform segmentation. Next, the processing results output by the segmenter are obtained, and the segmentation sequence information contained therein is extracted and saved as a candidate segmentation result sequence. Simultaneously, based on the spacing between adjacent words in the candidate segmentation result sequence, all segmentation boundary points are marked, constructing a candidate boundary position set. Finally, the candidate segmentation result sequence and the candidate boundary position set are saved as a standard data structure for use in subsequent evaluation steps.

[0053] Step 104: Combine the forced word segmentation boundary position set and the character co-occurrence affinity matrix to perform boundary compliance checks on the candidate boundary position set, and obtain the boundary compliance rate and boundary error set.

[0054] In this embodiment, boundary compliance testing refers to the process of evaluating the reasonableness of word segmentation boundary positions through linguistic statistical characteristics, including two parts: mandatory boundary compliance testing and character affinity testing. Boundary compliance rate refers to the degree to which word segmentation boundaries conform to linguistic statistical rules, typically including two indicators: mandatory boundary compliance rate and affinity compliance rate. Boundary error set refers to the set of positions that do not meet boundary compliance requirements, including mandatory boundary missing errors and affinity violation errors.

[0055] Specifically, firstly, it checks whether each position in the mandatory segmentation boundary position set appears in the candidate boundary position set. Positions that do not appear are recorded as mandatory boundary missing errors and added to the boundary error set. Then, the mandatory boundary compliance rate is calculated, which is the number of mandatory boundaries in the candidate boundary position set divided by the total number of mandatory segmentation boundary positions. Next, the non-mandatory boundary positions in the candidate boundary position set after excluding mandatory boundaries are examined, and the mutual information value of the adjacent characters on both sides of each boundary position in the character co-occurrence affinity matrix is ​​queried. Subsequently, boundary positions with mutual information values ​​greater than a preset affinity threshold are marked as affinity violation errors and added to the boundary error set. Finally, the affinity compliance rate is calculated, which is the number of non-affinity violation errors divided by the total number of non-mandatory boundary positions. The mandatory boundary compliance rate and the affinity compliance rate are used together as the evaluation index of the boundary compliance rate.

[0056] In one possible implementation, the candidate boundary position set is checked for compliance by combining the forced word segmentation boundary position set and the character co-occurrence affinity matrix to obtain the boundary compliance rate and the boundary error set. Specifically, this includes steps 1041-1042, as follows: Step 1041: Record the positions in the forced segmentation boundary position set that do not appear in the candidate boundary position set as forced boundary missing errors, add them to the boundary error set, and calculate the forced boundary compliance rate.

[0057] In this embodiment, a forced boundary missing error refers to a situation where the word segmenter fails to segment at the positions specified in the forced segmentation boundary position set, indicating that the word segmenter incorrectly combines characters that should be separated. The boundary error set represents the set of all segmentation positions that do not conform to the boundary rules, including two types of errors: forced boundary missing and affinity violation. The forced boundary compliance rate refers to the proportion of forced segmentation boundaries that the word segmenter correctly identifies, reflecting the accuracy of the word segmenter in identifying obvious word boundaries. It is calculated by dividing the number of correctly identified forced boundaries by the total number of forced boundaries.

[0058] Specifically, firstly, the forced segmentation boundary position set is compared with the candidate boundary position set to check whether each position in the forced segmentation boundary position set appears in the candidate boundary position set. Then, for each position in the forced segmentation boundary position set that does not appear in the candidate boundary position set, it is marked as a forced boundary missing error. Next, a boundary error set data structure is created, and all marked forced boundary missing error position information is added to the boundary error set. Each error record contains an error type identifier, text position information, and related character information. Then, the number of correctly identified forced boundaries is calculated, that is, the number of positions in the forced segmentation boundary position set that appear in the candidate boundary position set. Finally, the forced boundary compliance rate is calculated, that is, the number of correctly identified forced boundaries divided by the total number of positions in the forced segmentation boundary position set. The resulting ratio is the forced boundary compliance rate.

[0059] Step 1042: For the non-forced boundary positions after excluding forced boundaries in the candidate boundary position set, query the mutual information value of adjacent characters in the character co-occurrence affinity matrix, record the boundary positions with mutual information values ​​greater than the preset affinity threshold as affinity violation errors, classify them into the boundary error set, and calculate the affinity compliance rate; use the forced boundary compliance rate and the affinity compliance rate as the boundary compliance rate.

[0060] In this embodiment, non-mandatory boundary positions refer to boundary positions in the candidate boundary position set that do not belong to the mandatory word segmentation boundary position set. The word segmentation correctness of these positions needs to be further judged by character affinity. The preset affinity threshold represents the critical point of mutual information value for determining whether character pairs should be connected, and is usually a positive number. Affinity violation error refers to the word segmenter segmenting between character pairs with high mutual information values, incorrectly separating characters that should be combined. Affinity compliance rate refers to the proportion of non-mandatory boundary positions that conform to the character affinity rule, reflecting the accuracy of the word segmenter in recognizing general word boundaries. Boundary compliance rate is a boundary correctness evaluation index that comprehensively considers both mandatory boundary and affinity dimensions.

[0061] Specifically, firstly, all positions contained in the mandatory segmentation boundary position set are removed from the candidate boundary position set to obtain the non-mandatory boundary position set. Then, for each position in the non-mandatory boundary position set, the adjacent characters on both sides of the position are extracted to form a character pair. Next, the character co-occurrence affinity matrix is ​​queried to obtain the mutual information value of the character pair. Then, the mutual information value is compared with a preset affinity threshold. If the mutual information value is greater than the preset affinity threshold, it indicates that the two characters should be combined rather than separated, and the boundary position is marked as an affinity violation error. Then, the position information of all affinity violation errors is added to the boundary error set. Each error record contains an error type identifier, text position information, related character information, and mutual information value. Next, the number of non-mandatory boundary positions that are not marked as affinity violation errors is calculated, that is, the number of affinity compliant boundaries. Then, the affinity compliance rate is calculated, that is, the number of affinity compliant boundaries divided by the total number of non-mandatory boundary positions. Finally, the mandatory boundary compliance rate and the affinity compliance rate are used as two components of the boundary compliance rate for subsequent comprehensive evaluation.

[0062] Step 105: Perform word compliance checks on the candidate word segmentation result sequences using the morphological pattern rule base and cross-language translation probability reference table to obtain the word compliance rate and word error set.

[0063] In this embodiment, word compliance verification refers to the process of evaluating the rationality of each word in the segmentation result from two dimensions: morphological structure and semantic integrity. Word compliance rate refers to the degree to which words in the segmentation result conform to language rules and semantic integrity, including two indicators: morphological compliance rate and semantic integrity rate. The word error set refers to the set of words that do not meet the word compliance requirements, including morphological violations and semantic fragmentation errors.

[0064] Specifically, firstly, each word in the candidate word segmentation result sequence is traversed to check whether it conforms to the affix combination pattern or high-frequency string recorded in the morphological pattern rule base; then, words that do not conform to the morphological rules are marked as morphological violations and added to the word error set; next, the morphological compliance rate is calculated, which is the number of words that conform to the morphological rules divided by the total number of words in the candidate word segmentation result sequence; then, it is checked whether each word in the candidate word segmentation result sequence has a corresponding translation word that meets the preset semantic verification threshold in the cross-language translation probability reference table; then, words that do not have a corresponding high-probability translation are marked as semantic fragmentation errors and added to the word error set; finally, the semantic integrity rate is calculated, which is the number of words with a high probability translation divided by the total number of words, and the morphological compliance rate and semantic integrity rate are used together as the evaluation index of the word compliance rate.

[0065] In one possible implementation, the candidate word segmentation result sequence is checked for word compliance using a morphological pattern rule base and a cross-language translation probability reference table to obtain the word compliance rate and word error set. Specifically, this includes steps 1051-1052, as follows: Step 1051: Verify whether each word in the candidate word segmentation result sequence conforms to the target affix combination pattern and target string in the morphological pattern rule library. Record the words that do not conform as morphological violations, add them to the word error set, and calculate the morphological compliance rate.

[0066] In this embodiment, morphological violations represent words in the candidate word segmentation results that do not conform to the morphological structure rules of the target language. These words are generally not valid words in the target language. Morphological compliance rate refers to the ratio of the number of words in the word segmentation results whose morphological structure conforms to the rules to the total number of words, reflecting the segmenter's ability to recognize word morphological features. The word error set represents the collection of all word segmentation results that do not conform to the word rules, including morphological violations and semantic fragmentation errors. Target affix combination pattern refers to the common word-forming unit combination rules in the target language, such as the combination of specific prefixes and suffixes. The target string represents valid words or word-forming units that frequently appear in the corpus.

[0067] Specifically, firstly, each word in the candidate segmentation result sequence is traversed to construct a list of words to be verified. Then, for each word to be verified, it is checked whether it directly matches the target string in the morphological pattern rule base. If it matches, it is marked as morphologically compliant. Next, for words that do not match the target string, it is further checked whether they conform to the target affix combination pattern in the morphological pattern rule base. The method is to analyze whether the prefix, suffix, and root of the word conform to the rules. Subsequently, for words that are still not confirmed as morphologically compliant, additional morphological rules are applied for verification, such as checking word length, character composition, and special markers. Then, all words that fail any of the above verifications are marked as morphological violations, and the word content, position information, and violation type are recorded. Next, these morphologically violating words are added to the word error set. Finally, the morphological compliance rate is calculated, which is the ratio of the number of morphologically compliant words to the total number of words in the candidate segmentation result sequence.

[0068] Step 1052: Query whether there are corresponding translation words that meet the preset semantic verification threshold in the cross-language translation probability reference table for each word in the candidate word segmentation result sequence. Record the non-existent words as semantic fragmentation errors, classify them into the word error set, and calculate the semantic integrity rate. Use the morphological compliance rate and semantic integrity rate as the word compliance rate.

[0069] In this embodiment, semantic fragmentation errors represent words in the candidate word segmentation results that lack complete semantics; these words cannot find a high-probability corresponding translation word in cross-language translation. Semantic completeness rate refers to the ratio of the number of words with complete semantics in the word segmentation results to the total number of words, reflecting the segmenter's ability to identify the semantic boundaries of words. The preset semantic verification threshold represents the probability critical point for determining the validity of translation relationships, typically a value between 0 and 1. The cross-language translation probability reference table is a data structure that records the translation probabilities between target language words and reference language words, used to evaluate the semantic completeness of the word segmentation results. Word compliance rate is a word correctness evaluation index that comprehensively considers both morphological rules and semantic completeness.

[0070] Specifically, firstly, each word in the candidate word segmentation result sequence is traversed to construct a list of words to be checked; then, for each word to be checked, a cross-language translation probability reference table is queried to see if there is an entry indexed by that word; next, for words with corresponding entries, all possible corresponding translation words and their corresponding translation probabilities are extracted; then, it is checked whether at least one corresponding translation word has a translation probability greater than a preset semantic verification threshold, and if so, the word is marked as semantically complete; then, all words that cannot be marked as semantically complete are recorded as semantically fragmented errors, including word content, position information, and error type; then, these semantically fragmented error words are added to the word error set; then, the semantic completeness rate is calculated, which is the ratio of the number of semantically complete words to the total number of words in the candidate word segmentation result sequence; finally, the morphological compliance rate and the semantic completeness rate are used as two components of the word compliance rate for subsequent comprehensive evaluation of word segmentation quality.

[0071] Step 106: Calculate the comprehensive evaluation score based on the boundary compliance rate and word compliance rate, and generate an error location report by combining the comprehensive evaluation score, boundary error set, and word error set.

[0072] In this embodiment, the comprehensive evaluation score represents a quantitative assessment of the overall performance of the word segmenter, and is a weighted sum of boundary compliance rate and word compliance rate. The error location report is an analytical document that records in detail the types, locations, and quantities of word segmentation errors, used to guide improvements and optimizations to the word segmenter.

[0073] Specifically, firstly, weight coefficients are set for boundary compliance rate and word compliance rate, with the sum of the weights being 1. Then, the comprehensive evaluation score is calculated according to the formula, which is the boundary compliance rate multiplied by its weight plus the word compliance rate multiplied by its weight. Next, the error location information recorded in the boundary error set and word error set is mapped back to the original text of the test corpus, and the error type corresponding to each error location is labeled (forced boundary missing, affinity violation, morphological violation, or semantic fragmentation). Then, the product of the preset benchmark threshold and the comprehensive evaluation score is calculated, and the result is used as the error analysis threshold. Subsequently, the proportion of each type of error to the total number of errors is counted, and error types with a proportion exceeding the error analysis threshold are marked as target defects. Finally, the target defect information, error locations in the original text, various evaluation indicators, and comprehensive evaluation scores are summarized to generate an error localization report containing quantitative scores, error type distribution, and typical error instances, providing detailed guidance for word segmenter optimization.

[0074] In the above embodiments, basic word segmentation quality assessment functions are achieved through the independent calculation of boundary compliance rate and word compliance rate. To further improve the practicality and guiding value of the assessment results and reduce the impact of the abstractness of assessment indicators and the difficulty of interpreting results on the optimization efficiency of the word segmenter, this application also provides an assessment method for low-resource language word segmentation effects. This method integrates assessment indicators of different dimensions through weight coefficients, constructs a differentiated error type identification mechanism to adapt to the analysis needs of different error scenarios, and dynamically adjusts the word segmentation dictionary in conjunction with user verification feedback, enabling the system to handle word segmentation quality assessment and targeted optimization needs in complex language environments more efficiently and accurately. The following section combines... Figure 2 The evaluation method for low-resource language segmentation in the embodiments of this application is described below: Please see Figure 2 This is a flowchart illustrating an evaluation method for low-resource language segmentation in an embodiment of this application.

[0075] Step 201: Obtain the weighting coefficients of the boundary compliance rate and the word compliance rate; calculate the weighted average of the boundary compliance rate and the word compliance rate based on the weighting coefficients to obtain the comprehensive evaluation score.

[0076] In this embodiment, the weighting coefficient represents the relative importance assigned to the boundary compliance rate and word compliance rate when calculating the comprehensive evaluation score. It is typically a real number between 0 and 1, and its sum is 1. The boundary compliance rate refers to the proportion of correctly positioned word boundaries in the segmentation results, reflecting the accuracy of the segmentation system in recognizing word boundaries. The word compliance rate represents the proportion of correctly structured and semantically correct words in the segmentation results, including both morphological compliance rate and semantic integrity rate. The comprehensive evaluation score is a quantitative indicator of the overall performance of the segmentation system obtained through weighted calculation, comprehensively considering both boundary correctness and word correctness.

[0077] Specifically, first, the weight coefficients wb for the boundary compliance rate and ww for the word compliance rate are obtained from the configuration file or preset parameters; then, it is verified whether the sum of the obtained weight coefficients is equal to 1. If it is not equal to 1, the weight coefficients are normalized so that wb + ww = 1; next, the boundary compliance rate BR and word compliance rate WR calculated in the previous steps are extracted; then, the comprehensive evaluation score S is calculated using a weighted summation method, i.e., S = wb × BR + ww × WR; then, according to the evaluation needs, the calculated comprehensive evaluation score can be standardized to a specific range (such as a percentage system from 0 to 100); finally, the calculation result is stored as a comprehensive evaluation score for subsequent error analysis and report generation.

[0078] Step 202: Restore the error locations in the boundary error set and word error set to the original text of the test corpus, and label the corresponding error types.

[0079] In this embodiment, the boundary error set represents a collection recording all word boundary segmentation errors, including two types of error information: missing boundaries and redundant boundaries. The word error set refers to a collection recording all internal structural or semantic errors of words, including two types of error information: morphological violations and semantic fragmentation. The error location indicates the specific character index or range where the error occurs in the original text. The test corpus refers to the text data set used to evaluate the performance of the word segmentation system. The original text refers to a continuous sequence of characters before any word segmentation processing, representing the initial state before segmentation. The error type represents the classification label of the error, used to distinguish different types of word segmentation errors, such as missing boundaries, redundant boundaries, morphological violations, and semantic fragmentation.

[0080] Specifically, firstly, the error items in the boundary error set and the word error set are merged into a unified error list, with each error item containing the error type, location information, and related description. Then, the original text content of the test corpus is obtained, and error locations are marked within it. Next, for each error item, based on its recorded location information (such as word index, character offset, etc.), the relative position is converted to an absolute character position in the original text. Subsequently, special markers or annotations are inserted at the corresponding positions in the original text to indicate the start and end positions of the error and the error type. Then, for boundary errors, the specific locations of missing or redundant boundaries are marked, along with suggestions for correct boundary positions. Next, for word errors, the range of words with morphological violations or semantic fragmentation is marked, along with possible correct word forms. Finally, the marked original text is saved as a visual document with error locations and error types for subsequent analysis and improvement.

[0081] Step 203: Calculate the product of the preset benchmark threshold and the comprehensive evaluation score, use the product as the error analysis threshold, and count the proportion of each error type.

[0082] In this embodiment, the preset benchmark threshold represents the basic judgment standard set by the evaluation system, used to determine whether an error type requires priority attention. The comprehensive evaluation score refers to the quantitative indicator of the overall performance of the word segmentation system obtained through weighted calculation. The error analysis threshold represents the critical value used to judge the severity of error types; error types exceeding this value are considered problems requiring priority resolution. Error type refers to the classification label of word segmentation errors, including boundary missing, boundary redundancy, morphological violations, and semantic fragmentation. The quantity percentage represents the proportion of errors of a specific error type among all errors, reflecting the relative frequency of that type of error.

[0083] Specifically, firstly, a preset baseline threshold B is obtained from the configuration file or preset parameters. This threshold is usually set according to the expected performance level of the word segmentation system. Then, the calculated comprehensive evaluation score S is extracted. Next, the product of the preset baseline threshold and the comprehensive evaluation score, T = B × S, is calculated to obtain the error analysis threshold. Subsequently, the number of errors of various error types in the boundary error set and word error set is counted, including boundary missing, boundary redundancy, morphological violation, and semantic fragmentation. Then, the ratio of the number of errors of each error type to the total number of all errors is calculated to obtain the proportion of each error type, Pi. Next, the proportions of each error type are sorted to identify the main sources of errors. Finally, the error analysis threshold and the proportions of each error type are stored as intermediate results for subsequent target defect labeling.

[0084] Step 204: Mark the error types whose quantity exceeds the error analysis threshold as target defects, summarize the target defects, the error locations in the original text, and the comprehensive evaluation score, and generate an error location report.

[0085] In this embodiment, the target defect refers to the word segmentation system problem that needs to be addressed first; it is the error type whose quantity exceeds the error analysis threshold. The error location report is a document recording the word segmentation system evaluation results and error analysis, including system score, main defects, and error locations. The quantity percentage represents the proportion of errors of a specific error type among all errors. The error analysis threshold is a critical value used to determine the severity of the error type. The original text refers to a continuous sequence of characters before word segmentation. The error location represents the specific character index or range where the error occurs in the original text. The comprehensive evaluation score is a quantitative indicator of the overall performance of the word segmentation system.

[0086] Specifically, the process begins by comparing the proportion of each error type (Pi) with the calculated error analysis threshold (T). Then, for each error type, if its proportion (Pi) is greater than the threshold (T), it is marked as a target defect. Next, all error types marked as target defects are summarized to form a target defect list. Subsequently, error location information is extracted from the original text, including error type, location, and possible corrective suggestions. Then, the calculated comprehensive evaluation score is extracted. Next, the basic structure of the error location report is created, including an evaluation overview, score details, target defects, and error distribution. Then, the comprehensive evaluation score and the scores for each sub-item (boundary compliance rate and word compliance rate) are entered into the score details section. Next, the target defect list and its corresponding proportion are entered into the target defect section. Then, the error location information from the original text is organized into an error distribution section, organized according to error type and location. Finally, all information is integrated to generate a complete error location report to support subsequent optimization and improvement of the word segmentation system.

[0087] In one possible implementation, after generating the error location report, steps 2041-2043 are further included, as follows: Step 2041: Extract the target defects and corresponding target error locations from the error location report.

[0088] In this embodiment, the target defect refers to the error type identified as requiring priority resolution during the word segmentation system evaluation process. These are typically error types whose quantity exceeds the error analysis threshold, such as boundary missing, boundary redundancy, morphological violations, or semantic fragmentation. The error localization report is a document recording the word segmentation system evaluation results, including system score, main defects, and error distribution. The target error location indicates the specific location information of the error related to the target defect in the original text, typically including the start index, end index, and the sentence in which it occurs.

[0089] Specifically, the process begins by opening and parsing the error location report file to identify the list of error types marked as target defects. Then, iterates through the error distribution section of the report, filtering out error items related to the target defects. Next, it extracts the location information of each relevant error item, including the character index range or row and column numbers in the original text. Subsequently, it establishes a mapping relationship between each target defect and all its corresponding error locations, forming a target defect-error location mapping table. Finally, it stores the extracted list of target defects and their corresponding target error location information as structured data, preparing for subsequent context extraction and user verification task generation.

[0090] Step 2042: Extract context fragments based on the target error location and generate a user verification task in combination with the target defect; send the user verification task to the user terminal and receive the correction text returned by the user terminal.

[0091] In this embodiment, the context fragment refers to a section of text surrounding the error location, used to provide contextual information about the error's occurrence, typically containing several characters or sentences before and after the error location. The user verification task is an interactive task that displays the user a context fragment containing the error and requests the user to provide the correct word segmentation result, including elements such as task description, error type prompts, the context fragment, and an interactive interface. The user terminal refers to the device on which the user performs the verification operation, which can be a computer, a mobile device, or a dedicated terminal device. The corrected text refers to the text content submitted by the user after correcting the error based on their own language knowledge, containing correct word segmentation boundaries or word forms.

[0092] Specifically, the process begins by obtaining the target defect and its corresponding error location. Then, for each error location, a context fragment containing that error location is extracted from the original text, typically N characters before and after the error location (N can be adjusted based on the error type and text characteristics). Next, for each context fragment, a user verification task is generated based on its corresponding target defect type. This task includes an error type description, a context fragment display, and a correction input area. The generated user verification task is then packaged according to a predefined communication protocol and sent to the user terminal via a network interface. The verification task is then displayed on the user terminal, awaiting the user's correction of the errors in the context fragment. Next, the correction result returned by the user terminal is received, and the user-submitted correction text is parsed. Finally, the received correction text is associated with and stored along with the original context fragment and error information, preparing for subsequent term extraction.

[0093] Step 2043: Extract the target word segmentation term based on the corrected text and context fragments; synchronize the target word segmentation term to the word segmentation dictionary of the word segmenter to be evaluated, so as to update the word segmenter to be evaluated.

[0094] In this embodiment, the target segmentation term refers to the correctly segmented unit extracted from the user-corrected text, used to update the segmenter's dictionary. It typically includes the word itself and its attribute information (such as part-of-speech, word frequency, etc.). The segmenter to be evaluated refers to the segmentation processing system undergoing performance evaluation and optimization, responsible for dividing continuous text into discrete word units. The segmentation dictionary represents a database storing all words and their attributes identified by the segmenter; it is the knowledge base for the segmenter to perform correct segmentation and typically includes a core dictionary and a user-defined dictionary.

[0095] Specifically, the process begins by acquiring the user-submitted corrected text and its corresponding context fragment. Then, the corrected text is compared with the original context fragment to identify the parts modified by the user. Next, standardized word segmentation units are extracted from the corrected parts, including the words themselves, word boundary positions, and possible part-of-speech tags. Subsequently, it is checked whether the extracted word segmentation units already exist in the word segmentation dictionary of the word segmenter to be evaluated. If they do not exist, they are marked as new entries; if they exist but have attribute differences, they are marked as updated entries. Then, a target word segmentation entry conforming to the word segmenter's dictionary format requirements is constructed, containing necessary word information and attribute data. Next, a connection is established with the word segmenter to be evaluated to obtain access to the word segmentation dictionary. Then, the constructed target word segmentation entry is written into the word segmentation dictionary according to the word segmenter's dictionary update interface specification. Finally, a dictionary update request is submitted, triggering the word segmenter to be evaluated to reload its dictionary resources, completing the word segmenter update process.

[0096] The following describes an evaluation system for low-resource language segmentation performance from the perspective of hardware processing. Please refer to [link to relevant documentation]. Figure 3 This is a schematic diagram of the structure of an evaluation system for low-resource language segmentation in an embodiment of this application.

[0097] It should be noted that, Figure 3 The structure of the low-resource language segmentation performance evaluation system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0098] like Figure 3 As shown, a low-resource language segmentation performance evaluation system includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes based on a program stored in Read-Only Memory (ROM) 302 or a program loaded from storage portion 308 into Random Access Memory (RAM) 303, such as performing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0099] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0100] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the various functions defined in the present invention.

[0101] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0103] Specifically, the low-resource language segmentation effect evaluation system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the low-resource language segmentation effect evaluation method provided in the above embodiment.

[0104] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the low-resource language segmentation effect evaluation system described in the above embodiments; or it may exist independently and not assembled into the low-resource language segmentation effect evaluation system. The storage medium carries one or more computer programs, which, when executed by a processor of the low-resource language segmentation effect evaluation system, enable the low-resource language segmentation effect evaluation system to implement the low-resource language segmentation effect evaluation method based on IoT data encryption transmission provided in the above embodiments.

Claims

1. A method for evaluating the effectiveness of low-resource language segmentation, characterized in that, The method includes: Obtain unlabeled monolingual and bilingual parallel corpora of the target language; Based on the unlabeled monolingual corpus, a character co-occurrence affinity matrix and a forced word segmentation boundary position set are constructed, and a morphological pattern rule base and a cross-language translation probability reference table are constructed based on the bilingual parallel corpus. Input the test corpus into the word segmenter to be evaluated to obtain the candidate word segmentation result sequence and the candidate boundary position set; The candidate boundary position set is checked for boundary compliance by combining the forced word segmentation boundary position set and the character co-occurrence affinity matrix to obtain the boundary compliance rate and the boundary error set; The candidate word segmentation result sequence is subjected to word compliance verification using the morphological pattern rule base and the cross-language translation probability reference table to obtain the word compliance rate and word error set; A comprehensive evaluation score is calculated based on the boundary compliance rate and the word compliance rate, and an error location report is generated by combining the comprehensive evaluation score, the boundary error set, and the word error set.

2. The method according to claim 1, characterized in that, The construction of the character co-occurrence affinity matrix and the forced word segmentation boundary position set based on the unlabeled monolingual corpus includes: The co-occurrence probability of adjacent character pairs and the occurrence probability of a single character in the unlabeled monolingual corpus were statistically analyzed. The co-occurrence probability is divided by the product of the occurrence probabilities of the two single characters that constitute the adjacent character pair and the logarithm is taken to obtain the mutual information value of the adjacent character pair. The mutual information values ​​of all adjacent character pairs are combined to form the character co-occurrence affinity matrix. Extract target adjacent character pairs whose mutual information values ​​are less than a preset breakpoint threshold from the character co-occurrence affinity matrix, obtain the interval positions of the target adjacent character pairs in the unlabeled monolingual corpus, and construct the forced word segmentation boundary position set.

3. The method according to claim 1, characterized in that, The process of combining the forced word segmentation boundary position set and the character co-occurrence affinity matrix to perform boundary compliance checks on the candidate boundary position set yields a boundary compliance rate and a boundary error set, including: Positions in the forced word segmentation boundary position set that do not appear in the candidate boundary position set are recorded as forced boundary missing errors, included in the boundary error set, and the forced boundary compliance rate is calculated. For the non-forced boundary positions after excluding forced boundaries in the candidate boundary position set, query the mutual information value of adjacent characters in the character co-occurrence affinity matrix, and record the boundary positions with mutual information values ​​greater than a preset affinity threshold as affinity violation errors, classify them into the boundary error set, and calculate the affinity compliance rate. The mandatory boundary compliance rate and the affinity compliance rate are used as the boundary compliance rate.

4. The method according to claim 1, characterized in that, The construction of the morphological pattern rule base and cross-language translation probability reference table based on the bilingual parallel corpus includes: Extract the target affix combination patterns and target strings with a frequency greater than a preset frequency threshold from the target language side of the bilingual parallel corpus, and construct the morphological pattern rule base. The morphological pattern rule base is used as a dictionary to perform word segmentation preprocessing on the bilingual parallel corpus to obtain target language words and reference language words; The co-occurrence frequency of the target language words and the reference language words in the same sentence pair of the bilingual parallel corpus is calculated, as well as the independent occurrence frequency of the target language words in the bilingual parallel corpus. Dividing the co-occurrence frequency by the independent occurrence frequency yields the translation probability of the target language word and the reference language word. The target language words with a translation probability greater than a preset semantic verification threshold are selected and combined with reference language words to form word pairs and stored in a template reference table to obtain the cross-language translation probability reference table.

5. The method according to claim 4, characterized in that, The process involves performing word compliance checks on the candidate word segmentation result sequence using the morphological pattern rule base and the cross-language translation probability reference table to obtain a word compliance rate and a word error set, including: Verify whether each word in the candidate word segmentation result sequence conforms to the target affix combination pattern and target string in the morphological pattern rule base. Record non-conforming words as morphological violation errors, classify them into the word error set, and calculate the morphological compliance rate. Query whether each word in the candidate word segmentation result sequence has a corresponding translation word that meets the preset semantic verification threshold in the cross-language translation probability reference table. Record the non-existent words as semantic fragmentation errors, add them to the word error set, and calculate the semantic integrity rate. The morphological compliance rate and the semantic integrity rate are used as the word compliance rate.

6. The method according to claim 1, characterized in that, The process of calculating a comprehensive evaluation score based on the boundary compliance rate and the word compliance rate, and generating an error location report by combining the comprehensive evaluation score, the boundary error set, and the word error set, includes: Obtain the weighting coefficients of the boundary compliance rate and the word compliance rate; The boundary compliance rate and the word compliance rate are weighted and calculated based on the weighting coefficients to obtain the comprehensive evaluation score. The error locations in the boundary error set and the word error set are restored to the original text of the test corpus, and the corresponding error types are labeled. Calculate the product of the preset benchmark threshold and the comprehensive evaluation score, use the product as the error analysis threshold, and count the proportion of each error type. Error types whose proportion exceeds the error analysis threshold are marked as target defects. The target defects, the error locations in the original text, and the comprehensive evaluation score are summarized to generate the error location report.

7. The method according to claim 6, characterized in that, After generating the error location report, the process further includes: Extract the target defects and corresponding target error locations from the error location report; Context fragments are extracted based on the target error location, and user verification tasks are generated in conjunction with the target defect; The user verification task is sent to the user terminal, and the correction text returned by the user terminal is received. Extract target word segments based on the corrected text and the context fragment; The target word segmentation entry is synchronized to the word segmentation dictionary of the word segmenter to be evaluated, so as to update the word segmenter to be evaluated.

8. A system for evaluating the effectiveness of low-resource language segmentation, characterized in that, The low-resource language segmentation performance evaluation system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the low-resource language segmentation performance evaluation system to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instruction is run on a low-resource language segmentation performance evaluation system, the low-resource language segmentation performance evaluation system performs the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on a low-resource language segmentation performance evaluation system, the low-resource language segmentation performance evaluation system performs the method as described in any one of claims 1-7.

Images