Sentence identification method, device, equipment and storage medium

Abstract

The present invention relates to the field of artificial intelligence technology, and discloses a sentence identification method, apparatus, computer equipment, and computer-readable storage medium. The method comprises: generating a word set of the text to be processed by splitting the text to be processed; determining the affiliation between each word in the word set, and constructing a corresponding language model based on the affiliation between each word; obtaining a sentence to be predicted, and obtaining a frequency value of the sentence to be predicted based on the sentence to be predicted and the language model; determining whether the sentence to be predicted is fluent based on the frequency value of the sentence to be predicted, thereby implementing an unsupervised method to model the distribution of sentences, greatly reducing labor costs, and adding prior knowledge by analyzing the affiliation structure of the sentence, thereby making full use of contextual semantics and improving the accuracy of the language model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a sentence identification method, apparatus, computer equipment, and computer-readable storage medium. Background Art

[0002] In the process of writing meeting minutes and official documents, typos and incoherent sentences are often difficult to avoid. In order to improve the writing quality of official documents and meeting minutes, reduce subsequent proofreading time and improve office efficiency, intelligent judgment of sentence fluency is very important.

[0003] Currently, there are two main methods for determining sentence fluency. The first is to train a language model using labeled training samples, enabling the trained language model to detect typos and incoherent sentences. The second method constructs a language model based on the word distribution of the input text. This model then inputs a sentence and outputs the probability of that sentence appearing, using a threshold to determine if the sentence is fluent. However, the first method often requires manual labeling of training samples, which is costly and requires complex calculations to ensure accurate detection. The second method, because it uses sequential modeling and specifies the length of the text to be trained, prevents the language model from fully utilizing the contextual semantics of the words in the text to be trained, and thus fails to capture dependencies between long-distance words, resulting in lower accuracy. Summary of the Invention

[0004] The main purpose of this application is to provide a sentence identification method, apparatus, computer equipment and computer-readable storage medium, aiming to solve the technical problems of high cost of existing language model training and inability to obtain dependencies between long-distance words, resulting in low accuracy of the language model.

[0005] In a first aspect, the present application provides a sentence identification method, the sentence identification method comprising the following steps:

[0006] Splitting the text to be processed to generate a word set of the text to be processed;

[0007] Determining the affiliation between each word in the word set, and constructing a corresponding language model according to the affiliation between each word;

[0008] Obtaining a sentence to be predicted, and obtaining a frequency value of the sentence to be predicted based on the sentence to be predicted and the language model;

[0009] Whether the sentence to be predicted is fluent is determined according to the frequency value of the sentence to be predicted.

[0010] In a second aspect, the present application further provides a sentence identification device, the sentence identification device comprising:

[0011] A generating module, configured to split the text to be processed and generate a word set of the text to be processed;

[0012] A determination and construction module, configured to determine the affiliation between the individual words in the word set and to construct a corresponding language model according to the affiliation between the individual words;

[0013] an acquisition module, configured to acquire a sentence to be predicted, and acquire a frequency value of the sentence to be predicted based on the sentence to be predicted and the language model;

[0014] The determination module is used to determine whether the sentence to be predicted is fluent according to the frequency value of the sentence to be predicted.

[0015] In a third aspect, the present application also provides a computer device comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the statement identification method as described above are implemented.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the statement identification method as described above are implemented.

[0017] The present application provides a sentence identification method, apparatus, computer device and computer-readable storage medium, which split a text to be processed to generate a word set of the text to be processed; determine the affiliation between each word in the word set, and build a corresponding language model based on the affiliation between each word; obtain a sentence to be predicted, and obtain a frequency value of the sentence to be predicted based on the sentence to be predicted and the language model; determine whether the sentence to be predicted is fluent based on the frequency value of the sentence to be predicted, thereby realizing an unsupervised method for modeling the distribution of sentences, greatly reducing labor costs, and adding prior knowledge by parsing the affiliation structure of the sentence, so as to make full use of contextual semantics and improve the accuracy of the language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 A flowchart of a sentence identification method provided in an embodiment of the present application;

[0020] Figure 2 for Figure 1 Schematic diagram of the sub-step flow chart of the sentence identification method in FIG;

[0021] Figure 3 for Figure 1 Schematic diagram of the sub-step flow chart of the sentence identification method in FIG;

[0022] Figure 4 A schematic block diagram of a sentence identification device provided in an embodiment of the present application;

[0023] Figure 5 This is a schematic block diagram of the structure of a computer device involved in one embodiment of the present application.

[0024] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0026] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0027] The present invention provides a method, apparatus, computer device, and computer-readable storage medium for identifying a sentence. The method can be applied to a computer device, such as a laptop computer, a desktop computer, or other electronic device.

[0028] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0029] Please refer to Figure 1 , Figure 1 A flowchart of a sentence identification method provided in an embodiment of the present application.

[0030] like Figure 1 As shown, the sentence identification method includes steps S101 to S104.

[0031] Step S101: Split the text to be processed to generate a word set of the text to be processed.

[0032] Exemplarily, a text to be processed is obtained and the text to be processed is split. The splitting method includes splitting the text to be processed according to preset string rules to generate a word set for the text to be processed. For example, the text to be processed is "Xiao Ming ate two breads today." Using the preset string rules, "Xiao Ming ate two breads today" is split into words such as "Xiao Ming, today, eat, two, breads." The words "Xiao Ming, today, eat, two, breads" are used as the word set for "Xiao Ming ate two breads today." The preset string rules include one string, two strings, three strings, four strings, etc.

[0033] An exemplary example is that the text to be processed is split to generate a word set of the text to be processed, including: splitting the text to be processed based on preset sentence segmentation rules to obtain multiple short sentences corresponding to the text to be processed; splitting the multiple short sentences based on preset word segmentation rules to obtain words corresponding to each of the short sentences, and generating a word set.

[0034] Exemplarily, a text to be processed is obtained, and the text to be processed is split based on a preset sentence segmentation rule to obtain multiple short sentences in the text to be processed. For example, the text to be processed obtained is "Xiao Ming ate two loaves of bread today, I am very angry! However, there is nothing I can do." According to the preset sentence segmentation rule, wherein the sentence segmentation rule is based on symbols such as period, comma, question mark and exclamation mark, "Xiao Ming ate two loaves of bread today, I am very angry! However, there is nothing I can do" is segmented to obtain multiple short sentences such as "Xiao Ming ate two loaves of bread today", "I am very angry" and "However, there is nothing I can do." When multiple short sentences corresponding to the text to be processed are obtained, each short sentence is segmented according to the preset word segmentation rule to obtain the words of each short sentence, and the words of each short sentence are grouped to generate a word set of the text to be processed, wherein the preset word segmentation rule includes multiple word combinations. For example, according to the preset word segmentation rule, the short sentence "Xiao Ming ate two loaves of bread today" is segmented to obtain the words "Xiao Ming, today, ate, two, bread." The words of each short sentence are grouped together to generate a word set of the to-be-processed text corresponding to each short sentence.

[0035] Step S102: Determine the affiliation between the words in the word set, and construct a corresponding language model according to the affiliation between the words.

[0036] Exemplarily, each word in the word set is analyzed through dependency syntactic analysis to determine the dependency relationships between each word and generate a dependency tree. This dependency syntactic analysis interprets the grammatical structure of a language unit by analyzing the dependency relationships between lexical components. It is based on the view that "the core verb in a sentence is the central component that controls other components, and it itself is not controlled by any component." Each word in the sentence, except the core verb, is controlled by the core verb as a syntactic component. The dependency relationships of each word are obtained according to the dependency tree to train a preset model. The parameters of the preset model are optimized based on the dependency relationships of each word to construct a corresponding language model, which is a 2-gram language model.

[0037] In one embodiment, specifically, referring to Figure 2 , step S102 includes: sub-step S1021 to sub-step S1023.

[0038] Sub-step S1021: compose a plurality of word pairs based on the words in the initialized descendant set and the words in the node set.

[0039] Exemplarily, the word set includes an initialization descendant set and a node set, wherein the words in the initialization descendant set and the node set are the same. For example, child is used to represent the initialization descendant set, and All is used to represent the node set, wherein child includes {W1, W2, W3...W i 、W j}, All includes {W1, W2, W3......W i 、W j}. Search for words from child and All to form corresponding word pairs. For example, search for W from child i , find W from All j , forming corresponding word pairs (W i W j ).

[0040] Sub-step S1022: Determine the weight of each word pair based on a preset entropy model.

[0041] Exemplarily, each word pair is input into a preset entropy model, which is then used to analyze each word pair to obtain a characteristic function for each word pair. Based on the characteristic function and the weight of each word pair, the weight of each word pair output by the entropy model is determined.

[0042] In one embodiment, determining the weight of each word pair based on a preset entropy model includes: determining the characteristic function of each word pair based on the preset entropy model; determining the weight of each word pair based on the characteristic function of each word pair; and determining the weight of each word pair according to the weight of each word pair and the characteristic function of each word pair.

[0043] For example, each word pair is input into a preset entropy model, and a feature function of each word pair is obtained based on the preset entropy model and each word pair. For example, any word in each word pair is input into the preset entropy model. The feature of the word pair is generated by the feature template C preset in the entropy model, and the feature template corresponding to the feature is obtained by the preset feature template formula to obtain the feature function corresponding to the feature in the feature template, wherein the number of feature templates is multiple and each feature template is different. For example, the word pair (W i W j ) is input into the entropy model, based on the preset feature template formula in the entropy model: Among them, f k (i, j) is the feature function corresponding to the k-th feature template, (W i W j ) is a word pair, and C is the number of feature templates.

[0044] When the feature function corresponding to the kth feature template of the word pair is obtained, the parameter weight of the feature function generated by the word pair in the kth feature template is obtained according to the maximum entropy model in the entropy model, and the parameter weight is used as the weight of the word pair. For example, based on the weight formula in the maximum entropy model Among them, W k The parameter weight of the feature function generated for the kth feature template, f k (i, j) is the feature function corresponding to the k-th feature template, Z is the normalization factor, is quantitative, P(ji) is the word pair (W i W j ) is a quantitative probability or frequency.

[0045] When the weight and feature function of the word pair are obtained, the value of the word pair is obtained based on the preset weight formula in the preset entropy model. For example, based on the preset weight formula W(i,j)=W×f(i,j), where W is the word pair (W i W j ), f(i,j) is the weight of the word pair (W i W j ), W(i,j) is the feature function of the word pair (W i W j )’s weight.

[0046] Sub-step S1023: determining the optimal affiliation pair among the plurality of word pairs according to the weights of the respective word pairs, wherein the optimal affiliation pair is a plurality of pairs.

[0047] Exemplarily, after obtaining the weights of each word pair, an optimal affiliation pair among the plurality of word pairs is determined based on the weights of each word pair. For example, by comparing the weights, the word pair corresponding to the maximum weight is determined as the optimal affiliation pair, wherein there are multiple optimal affiliation pairs, and each optimal affiliation pair is different.

[0048] In one embodiment, determining the optimal subsidiary relationship pair among the multiple word pairs based on the weights of the individual word pairs includes: comparing the weights of the individual word pairs composed of the same words, determining the target word pair with the largest weight; and using the target word pair as the optimal subsidiary relationship pair.

[0049] For example, the word pairs composed of the same words are compared, for example, the word pairs (W i W j ) and (W j W i ), determine the word pair with the largest weight as the target word pair, and use the target word pair as the optimal subordinate relationship pair. For example, search for words from child and All respectively to form corresponding multiple word pairs. For example, search for W from child i , find W from All j , forming corresponding word pairs (W i W j ), or, look up W from child j , find W from All i , forming corresponding word pairs (W j W i ), compare (W i W j ) and (W j W i ), and take the pair with the largest weight as the optimal affiliation pair. For example, we can obtain (W j W i ) and (W i W j ) weight, if (W j W i ) is greater than (W i W j ) weight, then determine (W j W i ) is the optimal affiliation pair; if (W j W i ) is less than (Wi W j ) weight, then determine (W i W j ) is the optimal affiliation pair.

[0050] In one embodiment, specifically, referring to Figure 3 , step S102 includes: sub-step S1024 to sub-step S1026.

[0051] Sub-step S1024: remove the optimal affiliation pairs from the initialized descendant set and the node set respectively to form an optimal affiliation pair set.

[0052] For example, when determining the optimal affiliation pair, the optimal affiliation pair is removed from the initialized descendant set and the node set to form the optimal affiliation pair set. j W i ) is the optimal affiliation pair, W j Remove from the initialized descendant set and set W i Remove from the node set; or, W j Remove from the node set and set W i Remove from the initialized descendant set. The removed optimal affiliation pairs are combined into an optimal affiliation pair set.

[0053] Sub-step S1025: Obtain the frequency value of each optimal affiliation pair in the optimal affiliation pair set to obtain the frequency value of the optimal affiliation pair set.

[0054] For example, the frequency value of the optimal adjunct relationship pair in natural language is counted. For example, (W j W i ) is the optimal affiliation pair, statistics (W j W i ) frequency value f ij and W i The frequency value f j , based on the frequency formula of the optimal affiliation pair: Get (W j W i ) frequency value. Obtain the frequency value of each optimal affiliation pair in the optimal affiliation pair set in sequence, and obtain the frequency value of the optimal affiliation pair set according to the frequency value of each optimal affiliation pair. For example, the optimal affiliation pair set is P = (W1 W2, W2 W3, .... W n-1 W n ), based on the preset optimal affiliation pair frequency formula P=P(W2|W1)×P(W3|W2)×P(W n|W n-1 ), and obtain the frequency value of the optimal affiliation pair set.

[0055] In statistics (W j W i ) frequency value f ij and W i The frequency value f j In the process, determine the frequency value f ij and f j Is it zero? If it is zero, use δ smoothing to smooth it. This smoothing method assumes that each word pair appears at least δ times. For each word pair, the original frequency f ij Change to f ij +δ, so for each word pair, its probability can be expressed as: The frequency values ​​corresponding to each optimal affiliation are obtained.

[0056] Sub-step S1026: training a preset model based on the frequency value of the optimal affiliation pair set and the optimal affiliation pair set to construct a corresponding language model.

[0057] Exemplarily, when the frequency value of the optimal affiliation pair set and the optimal affiliation pair set are obtained, the frequency value of the optimal affiliation pair set and the optimal affiliation pair set are used to train a preset model, and the weight matrix of the preset model is continuously trained through the frequency value of the optimal affiliation pair set and the optimal affiliation pair set to obtain the corresponding language model.

[0058] Step S103: Obtain a sentence to be predicted, and obtain a frequency value of the sentence to be predicted based on the sentence to be predicted and the language model.

[0059] For example, a sentence to be predicted is obtained, and the sentence to be predicted is split based on a preset word segmentation rule to obtain the words corresponding to the sentence to be predicted, and a word set is generated. The word set includes an initialized descendant set and a node set, wherein the words in the initialized descendant set and the node set are the same. For example, child is used to represent the initialized descendant set, and All is used to represent the node set, wherein child includes {W1, W2, W3...W i 、W j}, All includes {W1, W2, W3......W i 、W j}. Search for words from child and All to form corresponding word pairs. For example, search for W from child i , find W from All j , forming corresponding word pairs (W i W j ).

[0060] Each word pair is input into a preset entropy model, and each word pair is analyzed by the entropy model to obtain a characteristic function of each word pair, and a weight of each word pair is obtained based on the characteristic function. The weight of each word pair output by the entropy model is determined based on the characteristic function of each word pair and the weight of each word pair. The word pairs composed of the same words are compared, for example, the word pairs (W i W j ) and (W j W i ), determine the word pair with the largest weight as the target word pair, and use the target word pair as the optimal subordinate relationship pair. For example, search for words from child and All respectively to form corresponding multiple word pairs. For example, search for W from child i , find W from All j , forming corresponding word pairs (W i W j ), or, look up W from child j , find W from All i , forming corresponding word pairs (W j W i ), compare (W i W j ) and (W j W i ), and take the pair with the largest weight as the optimal affiliation pair. For example, we can obtain (W j W i ) and (W i W j ) weight, if (W j W i ) is greater than (W i W j ) weight, then determine (W j W i ) is the optimal affiliation pair; if (W j W i ) is less than (W i W j ) weight, then determine (W i W j ) is the optimal affiliation pair.

[0061] When determining the optimal affiliation pair, the optimal affiliation pair is removed from the initialized descendant set and the node set to form the optimal affiliation pair set. j W i ) is the optimal affiliation pair, W jRemove from the initialized descendant set and set W i Remove from the node set; or, W j Remove from the node set and set W i Remove the optimal dependency pairs from the initialized descendant set and form the optimal dependency pair set of the sentence to be predicted.

[0062] When obtaining the optimal set of subsidiary relationship pairs of the sentence to be predicted, the optimal set of subsidiary relationship pairs of the sentence to be predicted is input into a preset language model, and the optimal set of subsidiary relationship pairs of the sentence to be predicted is predicted based on the preset language model, and the frequency value of the optimal set of subsidiary relationship pairs of the sentence to be predicted output by the preset language model is obtained, and the frequency value is used as the frequency value of the sentence to be predicted.

[0063] Step S104: Determine whether the sentence to be predicted is fluent based on the frequency value of the sentence to be predicted.

[0064] Exemplarily, when the frequency value of the sentence to be predicted is obtained, whether the sentence to be predicted is fluent is determined based on the frequency value of the sentence to be predicted. For example, a preset threshold is obtained, and the frequency value of the sentence to be predicted is compared with the preset threshold. If the frequency value of the sentence to be predicted is greater than the preset threshold, the sentence to be predicted is determined to be fluent; if the frequency value of the sentence to be predicted is less than or equal to the preset threshold, the sentence to be predicted is determined to be influent.

[0065] In an embodiment of the present invention, the text to be processed is split to obtain a corresponding word set, and the word set is analyzed based on the dependency syntactic analysis to generate a dependency tree, and the weight of each pair of dependency pairs is obtained to obtain the optimal dependency pair. The optimal dependency pairs are used to generate an optimal dependency pair set, and the frequency value of each optimal dependency pair is determined to determine the frequency value of the optimal dependency set. The preset model is trained based on the frequency value of the optimal dependency set and the optimal dependency set to obtain a corresponding language model, and the frequency value of the sentence to be predicted is predicted based on the language model, thereby determining whether the sentence to be predicted is fluent based on the frequency value of the sentence to be predicted. Compared with the common supervised text classification method, the embodiment of the present invention greatly reduces the manpower cost, can make full use of external samples to improve the performance of the model, and to a certain extent avoid the problem of insufficient training samples, especially labeled samples; and generates dependency pairs to construct a language model. By analyzing the dependency structure of the sentence, prior knowledge is added, so that the context semantics can be fully utilized, avoiding the disadvantage that the traditional language model cannot model long-distance dependencies, thereby improving the accuracy of the language model.

[0066] Please refer to Figure 4 , Figure 4A schematic block diagram of a sentence identification device provided in an embodiment of the present application.

[0067] like Figure 4 As shown, the sentence identification device 400 includes: a generation module 401, a determination and construction module 402, an acquisition module 403, and a determination module 404.

[0068] A generating module 401 is used to split the text to be processed and generate a word set of the text to be processed;

[0069] A determination and construction module 402 is configured to determine the affiliation between the words in the word set and to construct a corresponding language model based on the affiliation between the words.

[0070] An acquisition module 403 is configured to acquire a sentence to be predicted and obtain a frequency value of the sentence to be predicted based on the sentence to be predicted and the language model;

[0071] The determination module 404 is configured to determine whether the sentence to be predicted is fluent according to the frequency value of the sentence to be predicted.

[0072] The generating module 401 is further configured to:

[0073] Splitting the text to be processed based on preset sentence splitting rules to obtain multiple short sentences corresponding to the text to be processed;

[0074] The multiple short sentences are split based on the preset word segmentation rules to obtain the words corresponding to each of the short sentences and generate a word set.

[0075] The determining and constructing module 402 is further configured to:

[0076] Forming a plurality of word pairs according to the words in the initialized descendant set and the words in the node set;

[0077] Determine the weight of each word pair based on the preset entropy model;

[0078] According to the weights of the respective word pairs, optimal affiliation pairs among the plurality of word pairs are determined, wherein the optimal affiliation pairs are a plurality of pairs.

[0079] The determining and constructing module 402 is further configured to:

[0080] Determine the characteristic function of each word pair based on the preset entropy model;

[0081] Determining the weight of each word pair based on the feature function of each word pair;

[0082] The weight of each word pair is determined according to the weight of each word pair and the characteristic function of each word pair.

[0083] The determining and constructing module 402 is further configured to:

[0084] Comparing the weights of the word pairs composed of the same words, and determining the target word pair with the largest weight;

[0085] The target word pair is taken as the optimal dependency relationship pair.

[0086] The determining and constructing module 402 is further configured to:

[0087] Removing the optimal affiliation pair from the initialized descendant set and the node set respectively to form an optimal affiliation pair set;

[0088] Obtaining the frequency value of each optimal affiliation pair in the optimal affiliation pair set to obtain the frequency value of the optimal affiliation pair set;

[0089] A preset model is trained according to the frequency value of the optimal affiliation pair set and the optimal affiliation pair set to construct a corresponding language model.

[0090] The determining module 404 is further configured to:

[0091] Comparing the frequency value of the sentence to be predicted with a preset threshold;

[0092] If the probability value of the sentence to be predicted is greater than the preset threshold, determining that the sentence to be predicted is fluent;

[0093] If the frequency value of the sentence to be predicted is less than or equal to the preset threshold, it is determined that the sentence to be predicted is not fluent.

[0094] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned statement judgment method embodiment and will not be repeated here.

[0095] The apparatus provided in the above embodiment can be implemented in the form of a computer program. The computer program can be used in Figure 5 Runs on the computer device shown.

[0096] See also Figure 5 , Figure 5 This is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a terminal.

[0097] like Figure 5As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0098] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any one of the statement identification methods.

[0099] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0100] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any statement identification method.

[0101] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0102] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0103] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0104] Splitting the text to be processed to generate a word set of the text to be processed;

[0105] Determining the affiliation between each word in the word set, and constructing a corresponding language model according to the affiliation between each word;

[0106] Obtaining a sentence to be predicted, and obtaining a frequency value of the sentence to be predicted based on the sentence to be predicted and the language model;

[0107] Whether the sentence to be predicted is fluent is determined according to the frequency value of the sentence to be predicted.

[0108] In one embodiment, when the processor generates a word set for the text to be processed based on the splitting of the text to be processed, it is used to implement:

[0109] Splitting the text to be processed based on preset sentence splitting rules to obtain multiple short sentences corresponding to the text to be processed;

[0110] The multiple short sentences are split based on the preset word segmentation rules to obtain the words corresponding to each of the short sentences and generate a word set.

[0111] In one embodiment, when the processor determines the affiliation relationship between the words in the word set, it is configured to implement:

[0112] Forming a plurality of word pairs according to the words in the initialized descendant set and the words in the node set;

[0113] Determine the weight of each word pair based on the preset entropy model;

[0114] According to the weights of the respective word pairs, optimal affiliation pairs among the plurality of word pairs are determined, wherein the optimal affiliation pairs are a plurality of pairs.

[0115] In one embodiment, when the processor determines the weight of each word pair based on the preset entropy model, it is used to implement:

[0116] Determine the characteristic function of each word pair based on the preset entropy model;

[0117] Determining the weight of each word pair based on the feature function of each word pair;

[0118] The weight of each word pair is determined according to the weight of each word pair and the characteristic function of each word pair.

[0119] In one embodiment, when the processor determines the optimal affiliation pair among the plurality of word pairs based on the weights of the respective word pairs, it is configured to implement:

[0120] Comparing the weights of the word pairs composed of the same words, and determining the target word pair with the largest weight;

[0121] The target word pair is taken as the optimal dependency relationship pair.

[0122] In one embodiment, when the processor constructs the corresponding language model according to the affiliation relationship between the words, it is used to implement:

[0123] Removing the optimal affiliation pair from the initialized descendant set and the node set respectively to form an optimal affiliation pair set;

[0124] Obtaining the frequency value of each optimal affiliation pair in the optimal affiliation pair set to obtain the frequency value of the optimal affiliation pair set;

[0125] A preset model is trained according to the frequency value of the optimal affiliation pair set and the optimal affiliation pair set to construct a corresponding language model.

[0126] In one embodiment, when the processor determines whether the statement is smoothly implemented based on the frequency value of the statement to be predicted, it is configured to implement:

[0127] Comparing the frequency value of the sentence to be predicted with a preset threshold;

[0128] If the probability value of the sentence to be predicted is greater than the preset threshold, determining that the sentence to be predicted is fluent;

[0129] If the frequency value of the sentence to be predicted is less than or equal to the preset threshold, it is determined that the sentence to be predicted is not fluent.

[0130] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the statement judgment method of the present application.

[0131] The computer-readable storage medium may be an internal storage unit of the computer device described in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a SmartMedia Card (SMC), a Secure Digital (SD) card, a flash memory card, etc., equipped on the computer device.

[0132] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0133] This invention refers to a novel application model for computer technologies, including storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms, in the blockchain language model. Blockchain, in essence, is a decentralized database consisting of a series of data blocks linked together using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include an underlying blockchain platform, a platform product and service layer, and an application service layer.

[0134] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0135] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments. The above description is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A sentence identification method, characterized in that: include: Splitting the text to be processed to generate a word set of the text to be processed, wherein the word set includes an initialized descendant set and a node set; A plurality of word pairs are formed based on the words in the initialized descendant set and the words in the node set using a dependency relationship syntactic analysis method; wherein the dependency relationship syntactic analysis method is based on the premise that the core verb in a sentence is the central component that controls other components and the core verb itself is not controlled by any component; Determine the weight of each word pair based on the preset entropy model; Determining an optimal affiliation pair among the plurality of word pairs according to the weights of the respective word pairs, wherein the optimal affiliation pair is a plurality of pairs; Removing the optimal affiliation pair from the initialized descendant set and the node set respectively to form an optimal affiliation pair set; Obtaining the frequency value of each optimal affiliation pair in the optimal affiliation pair set to obtain the frequency value of the optimal affiliation pair set; Building a corresponding language model based on the frequency value of the optimal affiliation pair set and training a preset model for the optimal affiliation pair set; Obtaining a sentence to be predicted, and obtaining a frequency value of the sentence to be predicted based on the sentence to be predicted and the language model; Whether the sentence to be predicted is fluent is determined according to the frequency value of the sentence to be predicted.

2. The sentence identification method according to claim 1, wherein: The step of splitting the text to be processed to generate a word set of the text to be processed includes: Splitting the text to be processed based on preset sentence splitting rules to obtain multiple short sentences corresponding to the text to be processed; The multiple short sentences are split based on the preset word segmentation rules to obtain the words corresponding to each of the short sentences and generate a word set.

3. The sentence identification method according to claim 1, wherein: Determining the weight of each word pair based on a preset entropy model includes: Determine the characteristic function of each word pair based on the preset entropy model; Determining the weight of each word pair based on the feature function of each word pair; The weight of each word pair is determined according to the weight of each word pair and the characteristic function of each word pair.

4. The sentence identification method according to claim 1, wherein: Determining the optimal affiliation pair among the plurality of word pairs according to the weights of the respective word pairs includes: Comparing the weights of the word pairs composed of the same words, and determining the target word pair with the largest weight; The target word pair is taken as the optimal dependency relationship pair.

5. The sentence identification method according to claim 1, wherein: The determining whether the sentence is fluent according to the frequency value of the sentence to be predicted includes: Comparing the frequency value of the sentence to be predicted with a preset threshold; If the probability value of the sentence to be predicted is greater than the preset threshold, determining that the sentence to be predicted is fluent; If the frequency value of the sentence to be predicted is less than or equal to the preset threshold, it is determined that the sentence to be predicted is not fluent.

6. A sentence identification device, characterized in that: include: A generating module, configured to split a text to be processed and generate a word set of the text to be processed, wherein the word set includes an initialized descendant set and a node set; a word analysis module, configured to form a plurality of word pairs based on the words in the initialized descendant set and the words in the node set using a dependency relationship syntactic analysis method; wherein the dependency relationship syntactic analysis method is based on the fact that a core verb in a sentence is a central component that controls other components and the core verb itself is not controlled by any component; A weight determination module, for determining the weight of each word pair based on a preset entropy model; a relationship pair determination module, configured to determine an optimal subordinate relationship pair among the plurality of word pairs according to the weights of the respective word pairs, wherein the optimal subordinate relationship pair is a plurality of pairs; A set forming module, configured to remove the optimal affiliation pairs from the initialized descendant set and the node set respectively to form an optimal affiliation pair set; a frequency value acquisition module, configured to acquire the frequency value of each optimal affiliation pair in the optimal affiliation pair set, to obtain the frequency value of the optimal affiliation pair set; A model construction module, configured to construct a corresponding language model based on the frequency value of the optimal affiliation pair set and the training of a preset model based on the optimal affiliation pair set; an acquisition module, configured to acquire a sentence to be predicted, and acquire a frequency value of the sentence to be predicted based on the sentence to be predicted and the language model; The determination module is used to determine whether the sentence to be predicted is fluent according to the frequency value of the sentence to be predicted.

7. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the sentence identification method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the sentence identification method according to any one of claims 1 to 5 are implemented.

Images