Chinese analysis method and system based on artificial intelligence

Abstract

The invention belongs to the technical field of natural language processing, and discloses a Chinese analysis method and system based on artificial intelligence, and the method comprises the steps: firstly obtaining text data to be subjected to word segmentation, inputting the text data to be subjected to word segmentation into a pre-training word segmentation model, and obtaining label probability distribution of each character position, including probability values of a word start, a word middle, a word end and a single word; then, based on label probability distribution, adopting a diversity sampling strategy, and generating a plurality of candidate word segmentation results by obtaining a non-zero probability label and constructing a path conforming to a grammatical rule; and finally, the candidate results are evaluated through a self-evaluation mechanism, and the mechanism combines three dimensions of internal consistency score, word consensus score and probability rationality score to select and output an optimal word segmentation result. By keeping the inherent uncertainty of word segmentation, the model is prevented from being converged to a single standard, the advantages of multiple word segmentation perspectives are effectively fused, and the problem that a traditional word segmentation system is limited in performance in cross-domain application is solved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and more specifically, to a Chinese language parsing method and system based on artificial intelligence. Background Art

[0002] Chinese word segmentation is a fundamental task in natural language processing, dividing a continuous sequence of Chinese characters into meaningful word sequences. Unlike Western languages ​​like English, Chinese text lacks distinct word separators, resulting in multiple possible word segmentation methods for the same character sequence. Accurate Chinese word segmentation directly impacts subsequent tasks such as part-of-speech tagging, syntactic analysis, and semantic understanding. It is a critical prerequisite for numerous applications, including search engines, machine translation, conversational systems, and content recommendations.

[0003] With the development of deep learning technology, Chinese word segmentation methods have evolved from early rule- and dictionary-based approaches to end-to-end learning methods based on neural networks. The training data for neural network word segmentation models typically comes from a variety of dictionaries, including general dictionaries, medical dictionaries, legal dictionaries, and other specialized dictionaries. These diverse dictionaries reflect the diverse requirements for word segmentation granularity and standards in various fields. However, despite using data from multiple dictionaries during training, the inherent learning mechanism of neural networks tends to search for dominant patterns in the data during training and converge to a single word segmentation standard. This convergence behavior makes it impossible for the model to simultaneously preserve the diversity of multiple word segmentation standards, and instead naturally favors the word segmentation method that dominates the dataset.

[0004] This inability to preserve the diversity of word segments from different dictionaries severely limits the performance of word segmentation systems in cross-domain applications. In contrast, while traditional dictionary-based approaches have lower accuracy, they can flexibly adapt to the needs of different domains by switching dictionaries. Neural network models lack this flexibility. Therefore, maintaining the high accuracy of neural networks while preserving the diversity of word segments from different dictionaries has become a major challenge facing Chinese word segmentation technology. Summary of the Invention

[0005] In order to overcome the above problems of the prior art, the present invention proposes a Chinese parsing method and system based on artificial intelligence to solve the above problems.

[0006] The present invention provides the following technical solutions:

[0007] A Chinese parsing method based on artificial intelligence, comprising:

[0008] Get the text data to be segmented;

[0009] The text data is input into a pre-trained word segmentation model, and the word segmentation model outputs a label probability distribution for each character position in the text data; the label probability distribution includes a probability value of each character position being the beginning of a word, the middle of a word, the end of a word, and a single word;

[0010] Based on the label probability distribution, a diversity sampling strategy is used to generate multiple candidate word segmentation results;

[0011] The multiple candidate word segmentation results are evaluated through a self-evaluation mechanism, and one candidate word segmentation result is selected from the multiple candidate word segmentation results as the final word segmentation result and output.

[0012] Preferably, the training method of the pre-trained word segmentation model includes:

[0013] Acquire a plurality of training data to form a training data set, wherein the training data includes text samples and corresponding label probability distributions;

[0014] Constructing a neural network model, inputting text samples in the training data set into the neural network model to obtain a predicted label probability distribution for each character position;

[0015] Define a loss function that measures the distance between the predicted label probability distribution and the label probability distribution in the training dataset;

[0016] Performing batch training using the training data set, and iteratively adjusting the parameters of the neural network model through a parameter optimization method to minimize the loss function value;

[0017] The trained neural network model is used as the pre-trained word segmentation model to output the label probability distribution of each character position.

[0018] Preferably, the step of generating multiple candidate word segmentation results by adopting a diversity sampling strategy based on the label probability distribution includes:

[0019] According to the label probability distribution, obtain all labels with non-zero probabilities at each character position;

[0020] Based on the grammatical rules, starting from the beginning of the text, retrieve all label paths that meet the grammatical rules, and record the number M of retrieved label paths, where the label path consists of the word start, word middle, word end or single word label at each character position;

[0021] Set an upper limit N on the number of candidate paths. If M is greater than N, use the diversity priority strategy to select N candidate paths from the M label paths. Otherwise, use the M label paths as candidate paths.

[0022] Each candidate path is converted into a corresponding word segmentation result to obtain multiple candidate word segmentation results.

[0023] Preferably, the diversity priority strategy includes:

[0024] Calculate the word segmentation feature vector of each candidate word segmentation result, wherein the word segmentation feature vector includes the number of word segments, average word length, longest word length, single-word ratio and word length variance;

[0025] Calculating the difference between candidate word segmentation results based on the word segmentation feature vector;

[0026] Set the upper limit of the number of clusters to the upper limit of the number of candidate paths N;

[0027] Use a clustering algorithm to cluster the candidate word segmentation results into K categories, where K is less than or equal to N, and each category represents a different word segmentation pattern;

[0028] When K is equal to N, one candidate word segmentation result is selected from each category to form the final N candidate word segmentation results;

[0029] When K is less than N, the number of samples to be provided for each category is calculated according to the ratio of the number of candidate word segmentation results in each category, ensuring that at least one sample is provided for each category and the total number of samples provided by all categories is N;

[0030] Within each category, randomly select a specified number of candidate word segmentation results;

[0031] The samples selected from all categories are merged to form the final N candidate word segmentation results.

[0032] Preferably, the self-assessment mechanism includes:

[0033] Calculate the internal consistency score of each candidate word segmentation result based on the label probability distribution;

[0034] Calculate the word consensus score of each candidate word segmentation result based on the co-occurrence frequency of the words in the candidate word segmentation results;

[0035] Calculate the probability rationality score of each candidate word segmentation result based on the label probability distribution;

[0036] The internal consistency score, word consensus score and probability rationality score are weighted and summed to obtain the comprehensive score of each candidate word segmentation result;

[0037] The candidate with the highest comprehensive score is used as the final word segmentation result.

[0038] Preferably, the step of calculating the internal consistency score comprises:

[0039] Get the label path corresponding to each candidate word segmentation result. For each character position in the candidate word segmentation result, calculate the ratio of the probability of the label corresponding to the character position to the probability of the highest probability label at that character position as the consistency score of that character position.

[0040] The consistency scores for all character positions were averaged to obtain an internal consistency score.

[0041] Preferably, the step of calculating the word consensus score includes:

[0042] Sequentially number each word in all candidate word segmentation results to determine the sequence number of each word in the word segmentation result to which it belongs;

[0043] For each word in each candidate word segmentation result, count the number of times the same word appears in the same sequence number of all candidate word segmentation results;

[0044] Divide the above statistical results by the total number of candidate word segmentation results to obtain the position consensus of the word;

[0045] For each candidate word segmentation result, the average position consensus of all its words is calculated to obtain the word consensus score of the candidate word segmentation result.

[0046] Preferably, the steps of calculating the probability rationality score include:

[0047] Get the label path corresponding to each candidate word segmentation result. For each character position, get the probability value of the corresponding label in the label probability distribution.

[0048] Calculate the average of the label probability values ​​corresponding to all character positions to obtain the probability rationality score.

[0049] The present invention also provides an artificial intelligence-based Chinese parsing system for implementing an artificial intelligence-based Chinese parsing method, comprising:

[0050] A data acquisition module is used to obtain text data to be segmented;

[0051] A label probability prediction module is used to input the text data into a pre-trained word segmentation model and output a label probability distribution for each character position in the text data through the word segmentation model; the label probability distribution includes the probability value of each character position as the beginning, middle, end and single word of a word;

[0052] Diversity word segmentation generation module, which is used to generate multiple candidate word segmentation results based on the label probability distribution and adopt a diversity sampling strategy;

[0053] The evaluation output module is used to evaluate the multiple candidate word segmentation results through a self-evaluation mechanism, select one candidate word segmentation result from the multiple candidate word segmentation results as the final word segmentation result, and output it.

[0054] The present invention provides a Chinese parsing method and system based on artificial intelligence, which has the following beneficial effects:

[0055] By designing the pre-trained word segmentation model to output a label probability distribution rather than a single label result, this method successfully preserves the inherent uncertainty of word segmentation. This design prevents the model from converging to a single word segmentation standard, but instead preserves multiple possible word segmentation schemes and their probability information, providing a foundation for subsequent diverse word segmentation. By adopting a diverse sampling strategy based on the label probability distribution, the system can efficiently generate multiple representative candidate word segmentation results. Unlike traditional methods that only output one word segmentation result with the highest probability, by obtaining non-zero probability labels and constructing paths that conform to grammatical rules, it ensures that the generated candidate results both conform to language rules and maintain diversity.

[0056] The candidate word segmentation results are comprehensively evaluated across three dimensions: internal consistency scoring, word consensus scoring, and probabilistic rationality scoring. This multi-dimensional evaluation mechanism avoids the one-sidedness that can result from a single criterion, ensuring both consensus and probabilistic rationality in word segmentation while ensuring model prediction accuracy. The word consensus scoring mechanism, in particular, effectively integrates the advantages of multiple word segmentation perspectives by analyzing the positional consistency of words in different candidate results. It can flexibly adapt to word segmentation standards in different fields, addressing the limited performance of word segmentation systems in cross-domain applications and making the final selected word segmentation results more accurate and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Schematic diagram of a flow chart of a Chinese parsing method based on artificial intelligence of the present invention;

[0058] Figure 2 This is a module diagram of an artificial intelligence-based Chinese parsing system of the present invention. DETAILED DESCRIPTION

[0059] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0060] Example 1

[0061] See also Figure 1, in this embodiment, a Chinese parsing method based on artificial intelligence includes:

[0062] S1. Obtain the text data to be segmented;

[0063] In this embodiment, the text data to be segmented can be directly input text, or text data obtained through methods such as speech-to-text conversion and picture text extraction.

[0064] S2. Input the text data into the pre-trained segmentation model, and output the label probability distribution of each character position in the text data through the segmentation model; the label probability distribution includes the probability values of each character position as the start of a word, in the middle of a word, at the end of a word, and a single character word.

[0065] The training method of the pre-trained segmentation model includes:

[0066] Obtain a number of training data to form a training dataset, and the training data includes text samples and corresponding label probability distributions;

[0067] Build a neural network model, input the text samples in the training dataset into the neural network model, and obtain the predicted label probability distribution of each character position;

[0068] Define a loss function, and the loss function is a distance metric between the predicted label probability distribution and the label probability distribution in the training dataset;

[0069] Use the training dataset for batch training, and iteratively adjust the parameters of the neural network model through a parameter optimization method to minimize the loss function value;

[0070] Take the trained neural network model as the pre-trained segmentation model for outputting the label probability distribution of each character position.

[0071] In this embodiment, inputting the text data into the pre-trained segmentation model means encoding the text to be segmented in character order to form an input sequence. For example, for the text Learning Chinese is very interesting, input it into the pre-trained segmentation model.

[0072] The segmentation model is built based on a deep learning algorithm, and its core is to output a complete label probability distribution for each character position. Specifically, for each character, the model outputs a four-dimensional probability vector, respectively representing the probability values of the character as the start of a word (B), in the middle of a word (I), at the end of a word (E), or a single character word (S). For example, the model may output a probability distribution [B: 0.85, I: 0.05, E: 0.02, S: 0.08] for the Chinese character, indicating that the model believes that Han is most likely to be the start of a word, but other possibilities are not excluded.

[0073] In this embodiment, the training method of the pre-trained word segmentation model is specifically implemented as follows:

[0074] First, obtain a large-scale Chinese text corpus, and use a variety of existing word segmentation tools (such as jieba, THULAC, NLPIR, etc.) to perform word segmentation on the same text. By statistically analyzing the word segmentation results of different tools, a label probability distribution is generated for each character position. For example, for the word "学习" in learning Chinese, if 80% of the word segmentation tools regard "学习" as a single word, and 20% regard "学" as a single character word, then the label probability distribution of "学" may be [B:0.8, I:0, E:0, S:0.2], that is, the probability of the start of a word is 80%, the probability of a single character word is 20%, and the probabilities of the middle and end of a word are 0. This method based on multi-tool statistics can capture the inherent uncertainty of word segmentation and form more realistic training data.

[0075] Next, construct a neural network model suitable for outputting probability distributions. In this embodiment, the model can adopt a model architecture that combines a bidirectional long short-term memory network and an attention mechanism, including a character embedding layer, multiple layers of bidirectional LSTM layers, a self-attention mechanism, and an output layer. Different from traditional word segmentation models that directly predict the most likely label, this model outputs a complete four-dimensional probability distribution through a softmax function, directly reflecting the uncertainty of word segmentation. The loss function uses cross-entropy loss to calculate the difference between the predicted label probability distribution and the true label distribution in the training dataset.

[0076] The training process can adopt multiple small batch methods, with a batch size of 128, using the Adam optimizer and an initial learning rate of 0.001. The training is iterated until the loss function value on the validation set converges. After training, evaluate the model performance on the test set, including word segmentation accuracy, recall rate, and F1 score. After the performance meets the requirements, save the model parameters for use as a pre-trained word segmentation model. In practical applications, this model can generate a label probability distribution for each character position of any input Chinese text, providing a basis for subsequent diverse word segmentation.

[0077] S3. Generate multiple candidate word segmentation results by adopting a diversity sampling strategy based on the label probability distribution;

[0078] The step of generating multiple candidate word segmentation results by adopting a diversity sampling strategy based on the label probability distribution includes:

[0079] According to the label probability distribution, obtain all labels with non-zero probabilities for each character position;

[0080] Based on the grammar rules, starting from the beginning position of the text, retrieve all label paths that satisfy the grammar rules, and record the number M of the retrieved label paths. The label path is composed of the start, middle, end of a word or single character word label for each character position;

[0081] Set an upper limit N on the number of candidate paths. If M is greater than N, use the diversity priority strategy to select N candidate paths from the M label paths. Otherwise, use the M label paths as candidate paths.

[0082] Each candidate path is converted into a corresponding word segmentation result to obtain multiple candidate word segmentation results.

[0083] The diversity-first strategy includes:

[0084] Calculate the word segmentation feature vector of each candidate word segmentation result, wherein the word segmentation feature vector includes the number of word segments, average word length, longest word length, single-word ratio and word length variance;

[0085] Calculating the difference between candidate word segmentation results based on the word segmentation feature vector;

[0086] Set the upper limit of the number of clusters to the upper limit of the number of candidate paths N;

[0087] Use a clustering algorithm to cluster the candidate word segmentation results into K categories, where K is less than or equal to N, and each category represents a different word segmentation pattern;

[0088] When K is equal to N, one candidate word segmentation result is selected from each category to form the final N candidate word segmentation results;

[0089] When K is less than N, the number of samples to be provided for each category is calculated according to the ratio of the number of candidate word segmentation results in each category, ensuring that at least one sample is provided for each category and the total number of samples provided by all categories is N;

[0090] Within each category, randomly select a specified number of candidate word segmentation results;

[0091] The samples selected from all categories are merged to form the final N candidate word segmentation results.

[0092] In this embodiment, a diversity sampling strategy is adopted based on the label probability distribution to generate multiple candidate word segmentation results, which means using the probability information output by the pre-trained word segmentation model to construct multiple possible word segmentation schemes, rather than just selecting a single scheme with the highest probability.

[0093] First, for each character position, get all labels with non-zero probability. Taking "Learning Chinese is fun" as an example, assume that the word segmentation model outputs the following label probability distribution:

[0094] Learning: [B:0.81,I:0.03,E:0.01,S:0.15],

[0095] Xi: [B:0.09,I:0.03,E:0.86,S:0.02],

[0096] Chinese: [B:0.92,I:0.02,E:0.01,S:0.05],

[0097] Language: [B:0.04,I:0.01,E:0.93,S:0.02],

[0098] Very: [B:0.02,I:0.01,E:0.02,S:0.95],

[0099] There are: [B:0.89,I:0.01,E:0.05,S:0.05],

[0100] Interesting: [B:0.02, I:0.01, E:0.94, S:0.03]. Keep the labels with non-zero probability at each position, even if the probability is small, as possible options.

[0101] Next, a label path is constructed from the beginning of the text based on the grammatical rules of word segmentation. Word segmentation must follow certain grammatical rules to ensure that the generated word segmentation results conform to language logic. For example, a single word can be followed by the beginning of a new word or another single word, which conforms to language logic; however, a single word followed by the beginning of a word does not conform to language logic.

[0102] By searching, starting from the first character and moving along the path that conforms to the grammatical rules until the last character, all possible complete label paths are recorded. For the example text, the path that conforms to the grammatical rules may be:

[0103] Path 1: BEBESBE (word segmentation result: learning / Chinese / very / interesting);

[0104] Path 2: SBESSBE (word segmentation result: learning Chinese is very interesting);

[0105] Path 3: SSBESSS (word segmentation result: learning Chinese is very interesting);

[0106] Considering the limitation of computing resources, in practical applications, an upper limit N is usually set on the number of candidate paths (for example, N = 10). Since 30>10, a diversity-first strategy is needed to select 10 paths from the 30 paths.

[0107] When implementing the diversity-first strategy, the feature vector of each candidate word segmentation result (composed of the number of segmented words, average word length, longest word length, single-word ratio, and word length variance) is first calculated. After calculating the feature vectors of all candidate word segmentation results, the Euclidean distance is used to measure the differences between the different word segmentation results, and a difference matrix is ​​constructed. Next, the 30 candidate word segmentation results are clustered using the K-means clustering algorithm. The upper limit of the number of clusters is set to N = 10, and the actual clustering result may result in K = 5 categories, each representing a different word segmentation pattern. Since K = 5 is less than N, the number of samples to be provided for each category is calculated based on the ratio of the number of candidate results in each category, ensuring that each category provides at least one sample, and the total number of samples is 10. Within each category, a specified number of candidate word segmentation results are randomly selected. Finally, the samples selected from all categories are merged to form 10 diverse candidate word segmentation results.

[0108] The advantage of this diversity sampling strategy is that it not only retains high-probability word segmentation results, but also ensures that different word segmentation patterns are represented, which can more comprehensively cover possible word segmentation schemes and provide diverse options for subsequent self-evaluation mechanisms.

[0109] S4. Evaluate the multiple candidate word segmentation results through a self-evaluation mechanism, select one candidate word segmentation result from the multiple candidate word segmentation results as the final word segmentation result, and output it.

[0110] The self-assessment mechanism includes:

[0111] Calculate the internal consistency score of each candidate word segmentation result based on the label probability distribution;

[0112] Calculate the word consensus score of each candidate word segmentation result based on the co-occurrence frequency of the words in the candidate word segmentation results;

[0113] Calculate the probability rationality score of each candidate word segmentation result based on the label probability distribution;

[0114] The internal consistency score, word consensus score and probability rationality score are weighted and summed to obtain the comprehensive score of each candidate word segmentation result;

[0115] The candidate with the highest comprehensive score is used as the final word segmentation result.

[0116] The steps for calculating the internal consistency score include:

[0117] Get the label path corresponding to each candidate word segmentation result. For each character position in the candidate word segmentation result, calculate the ratio of the probability of the label corresponding to the character position to the probability of the highest probability label at that character position as the consistency score of that character position.

[0118] The consistency scores for all character positions were averaged to obtain an internal consistency score.

[0119] The steps for calculating the word consensus score include:

[0120] Sequentially number each word in all candidate word segmentation results to determine the sequence number of each word in the word segmentation result to which it belongs;

[0121] For each word in each candidate word segmentation result, count the number of times the same word appears in the same sequence number of all candidate word segmentation results;

[0122] Divide the above statistical results by the total number of candidate word segmentation results to obtain the position consensus of the word;

[0123] For each candidate word segmentation result, the average position consensus of all its words is calculated to obtain the word consensus score of the candidate word segmentation result.

[0124] The calculation steps of the probability reasonableness score include:

[0125] Get the label path corresponding to each candidate word segmentation result. For each character position, get the probability value of the corresponding label in the label probability distribution.

[0126] Calculate the average of the label probability values ​​corresponding to all character positions to obtain the probability rationality score.

[0127] In this embodiment, the self-evaluation mechanism is used to evaluate multiple candidate word segmentation results, which means using multiple scoring criteria to comprehensively evaluate the candidate word segmentation results generated in the previous step and select the optimal word segmentation solution. Take candidate 1: BEBESBE (word segmentation result: learning / Chinese / very / interesting) as an example,

[0128] First, calculate the internal consistency score of each candidate word segmentation result. The calculation process is as follows:

[0129] The learning label is B, with a probability of 0.81. The highest probability label at this position is also B, with a ratio of 0.81 / 0.81=1.

[0130] The calculation for the remaining character positions is similar. The internal consistency score for candidate 1 is the average of the ratios across all positions: (1+1+1+1+1+1+1) / 7 = 1. The internal consistency score reflects the degree of consistency between the candidate's segmentation result and the model's optimal prediction. A higher score indicates greater consistency between the candidate's segmentation result and the model's preferred prediction.

[0131] Next, we calculate the word consensus score. Assume that among the 10 candidate word segmentation results:

[0132] In the first position, 7 candidate results will be learned as a word, and 3 will be learned as single-word pairs. Calculate the word consensus score for candidate 1 (learning / Chinese / very / interesting):

[0133] When "learning" is the first word, its position consensus is 7 / 10 = 0.7. The same calculation is done for the words in the remaining positions. The word consensus score for candidate 1 is (0.7 + 0.8 + 0.6 + 0.5) / 4 = 0.65. The word consensus score reflects the degree of recognition of the words in the candidate word segmentation results among all candidate solutions. The higher the score, the more support the word segmentation method gets from other candidate solutions.

[0134] Finally, calculate the probability rationality score: The probability rationality score for candidate 1 is (0.81 + 0.86 + 0.92 + 0.93 + 0.95 + 0.89 + 0.94) / 7 = 0.90. The probability rationality score directly reflects the rationality of the candidate word segmentation result under the model probability distribution. The higher the score, the more likely the word segmentation result is correct statistically.

[0135] Perform a weighted sum of the three scores. Assuming the weights are 0.3, 0.4, and 0.3 respectively, then the comprehensive score for candidate 1 is 0.83.

[0136] Assume that finally candidate 1 has the highest score. That is, after comprehensively considering the model's preferred judgment, the support of other candidate solutions, and the rationality of the model probability distribution, candidate 1 is the most suitable. Therefore, select "learning / Chinese / very / interesting" as the final word segmentation result and output it. This method of having the model output the probability distribution and then combining the probability distribution output and multi-dimensional self-evaluation can give full play to the advantages of multiple word segmentation criteria more fully. <000028​​​​​​​​​​​​​​​​​The evaluation output module is used to evaluate the multiple candidate word segmentation results through a self-evaluation mechanism, select one candidate word segmentation result from the multiple candidate word segmentation results as the final word segmentation result, and output it.

[0143] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only one type. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0144] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention.

[0145] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A Chinese parsing method based on artificial intelligence, characterized in that: include: Get the text data to be segmented; The text data is input into a pre-trained word segmentation model, and the word segmentation model outputs a label probability distribution for each character position in the text data; the label probability distribution includes a probability value of each character position being the beginning of a word, the middle of a word, the end of a word, and a single word; Based on the label probability distribution, a diversity sampling strategy is used to generate multiple candidate word segmentation results; The multiple candidate word segmentation results are evaluated through a self-evaluation mechanism, and one candidate word segmentation result is selected from the multiple candidate word segmentation results as the final word segmentation result and output.

2. The Chinese parsing method based on artificial intelligence according to claim 1, characterized in that: The training method of the pre-trained word segmentation model includes: Acquire a plurality of training data to form a training data set, wherein the training data includes text samples and corresponding label probability distributions; Constructing a neural network model, inputting text samples in the training data set into the neural network model to obtain a predicted label probability distribution for each character position; Define a loss function that measures the distance between the predicted label probability distribution and the label probability distribution in the training dataset; Performing batch training using the training data set, and iteratively adjusting the parameters of the neural network model through a parameter optimization method to minimize the loss function value; The trained neural network model is used as the pre-trained word segmentation model to output the label probability distribution of each character position.

3. The Chinese parsing method based on artificial intelligence according to claim 2, characterized in that: The step of generating multiple candidate word segmentation results using a diversity sampling strategy based on label probability distribution includes: According to the label probability distribution, obtain all labels with non-zero probabilities at each character position; Based on the grammatical rules, starting from the beginning of the text, retrieve all label paths that meet the grammatical rules, and record the number M of retrieved label paths, where the label path consists of the word start, word middle, word end or single word label at each character position; Set an upper limit N on the number of candidate paths. If M is greater than N, use the diversity priority strategy to select N candidate paths from the M label paths. Otherwise, use the M label paths as candidate paths. Each candidate path is converted into a corresponding word segmentation result to obtain multiple candidate word segmentation results.

4. The Chinese parsing method based on artificial intelligence according to claim 3, characterized in that: The diversity-first strategy includes: Calculate the word segmentation feature vector of each candidate word segmentation result, wherein the word segmentation feature vector includes the number of word segments, average word length, longest word length, single-word ratio and word length variance; Calculating the difference between candidate word segmentation results based on the word segmentation feature vector; Set the upper limit of the number of clusters to the upper limit of the number of candidate paths N; Use a clustering algorithm to cluster the candidate word segmentation results into K categories, where K is less than or equal to N, and each category represents a different word segmentation pattern; When K is equal to N, one candidate word segmentation result is selected from each category to form the final N candidate word segmentation results; When K is less than N, the number of samples to be provided for each category is calculated according to the ratio of the number of candidate word segmentation results in each category, ensuring that at least one sample is provided for each category and the total number of samples provided by all categories is N; Within each category, randomly select a specified number of candidate word segmentation results; The samples selected from all categories are merged to form the final N candidate word segmentation results.

5. The Chinese parsing method based on artificial intelligence according to claim 4, characterized in that: The self-assessment mechanism includes: Calculate the internal consistency score of each candidate word segmentation result based on the label probability distribution; Calculate the word consensus score of each candidate word segmentation result based on the co-occurrence frequency of the words in the candidate word segmentation results; Calculate the probability rationality score of each candidate word segmentation result based on the label probability distribution; The internal consistency score, word consensus score and probability rationality score are weighted and summed to obtain the comprehensive score of each candidate word segmentation result; The candidate with the highest comprehensive score is used as the final word segmentation result.

6. The Chinese parsing method based on artificial intelligence according to claim 5, characterized in that: The steps for calculating the internal consistency score include: Get the label path corresponding to each candidate word segmentation result. For each character position in the candidate word segmentation result, calculate the ratio of the probability of the label corresponding to the character position to the probability of the highest probability label at that character position as the consistency score of that character position. The consistency scores for all character positions were averaged to obtain an internal consistency score.

7. The Chinese parsing method based on artificial intelligence according to claim 5, characterized in that: The steps for calculating the word consensus score include: Sequentially number each word in all candidate word segmentation results to determine the sequence number of each word in the word segmentation result to which it belongs; For each word in each candidate word segmentation result, count the number of times the same word appears in the same sequence number of all candidate word segmentation results; Divide the above statistical results by the total number of candidate word segmentation results to obtain the position consensus of the word; For each candidate word segmentation result, the average position consensus of all its words is calculated to obtain the word consensus score of the candidate word segmentation result.

8. The Chinese parsing method based on artificial intelligence according to claim 5, characterized in that: The calculation steps of the probability reasonableness score include: Get the label path corresponding to each candidate word segmentation result. For each character position, get the probability value of the corresponding label in the label probability distribution. Calculate the average of the label probability values ​​corresponding to all character positions to obtain the probability rationality score.

9. An artificial intelligence-based Chinese parsing system, used to implement the artificial intelligence-based Chinese parsing method according to any one of claims 1 to 8, characterized in that: include: A data acquisition module is used to obtain text data to be segmented; A label probability prediction module is used to input the text data into a pre-trained word segmentation model and output a label probability distribution for each character position in the text data through the word segmentation model; the label probability distribution includes the probability value of each character position as the beginning, middle, end and single word of a word; Diversity word segmentation generation module, which is used to generate multiple candidate word segmentation results based on the label probability distribution and adopt a diversity sampling strategy; The evaluation output module is used to evaluate the multiple candidate word segmentation results through a self-evaluation mechanism, select one candidate word segmentation result from the multiple candidate word segmentation results as the final word segmentation result, and output it.

Images