Method and device for constructing rationality detection model of specific Chinese text

Abstract

The embodiment of the invention provides a method and device for constructing a rationality detection model of a specific Chinese text, and the method comprises the steps: carrying out the word segmentation of existing specific Chinese text data according to two different granularities, and constructing a priori knowledge phrase library; constructing a hypothesis space containing different forms of objective functions based on the priori knowledge phrase library; fitting an optimal objective function in a hypothesis space through minimizing a model error; and constructing a Chinese text rationality detection model based on the optimal objective function so as to perform rationality detection on the generated specific Chinese text based on the Chinese text rationality detection model.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of natural language processing, and in particular, to a method and device for constructing a specific Chinese text rationality detection model and a Chinese text rationality detection method. Background Art

[0002] Existing Chinese text analysis usually relies on manually constructed dictionaries and rules for word segmentation and semantic analysis. This approach is easily affected by subjective factors, resulting in certain deviations in the analysis results, lack of consistency and universality. Whenever new words or new usages appear, the dictionary needs to be recompiled, resulting in the lack of dynamic learning ability, timeliness and quantification of analysis methods based on rules and dictionaries.

[0003] With the development of deep learning technology, analysis methods based on large-scale pre-trained models can learn the optimal word segmentation strategy from a large amount of text through self-learning, avoiding the limitations of artificial rules. However, this analysis method often relies on simple correlation analysis, for example, revealing the relationship between words based on word frequency, TF-IDF, similarity, etc. This correlation analysis method easily ignores the deep connection at the semantic level, resulting in superficial analysis results and ignoring the different possible meanings of words in different contexts. Especially for certain types of Chinese texts, the accuracy of text rationality analysis is low. Summary of the invention

[0004] The embodiments described in this article provide a method and device for constructing a specific Chinese text rationality detection model, a computer-readable storage medium storing a computer program, and a Chinese text rationality detection method. The target function is fitted through a non-parametric model in order to fit functions of different forms, and the most reasonable non-probabilistic model is found through the hypothesis space determined by prior knowledge, so as to achieve the purpose of this model to test the semantic rationality of the generated target Chinese text.

[0005] According to the first aspect of the present disclosure, a method for constructing a specific Chinese text rationality detection model is provided, comprising: performing word segmentation on existing specific Chinese text data according to two different granularities to construct a priori knowledge phrase library; constructing a hypothesis space containing objective functions of different forms based on the priori knowledge phrase library; fitting an optimal objective function in the hypothesis space by minimizing the model error; and constructing a specific Chinese text rationality detection model based on the optimal objective function, so as to perform rationality detection on the generated Chinese text based on the Chinese text rationality detection model.

[0006] In some embodiments of the present disclosure, existing specific Chinese text data are segmented according to two different granularities, and constructing a priori knowledge phrase library includes: preprocessing the existing specific Chinese text data with data cleaning, denoising and standardization; performing maximum granularity segmentation and minimum granularity segmentation on the preprocessed Chinese text data to convert the text into phrases; and statistically analyzing the frequency and distribution of phrase occurrences to construct a priori knowledge phrase library.

[0007] In some embodiments of the present disclosure, constructing a hypothesis space containing different forms of objective functions based on a priori knowledge phrase library includes: converting the priori knowledge phrase library into vectorized text feature data; selecting different kernel functions according to the relationship between the text feature data, the kernel functions including linear kernel functions, polynomial kernel functions, and Gaussian kernel functions; and fitting the kernel function through non-parametric regression to obtain a hypothesis space containing different forms of objective functions, the non-parametric regression including kernel density estimation, local weighted scatter plot smoothing, and spline regression.

[0008] In some embodiments of the present disclosure, fitting the optimal objective function in the hypothesis space by minimizing the model error includes: Find the objective function with the smallest generalization error in the hypothesis space, where l(h(x), y) is the loss function of the selected objective function h(x), H is the hypothesis space, and P(x, y) is the joint probability distribution function of the input x and the label y.

[0009] In some embodiments of the present disclosure, fitting the optimal objective function in the hypothesis space by minimizing the model error also includes: Find the objective function with the minimum empirical error, where (x (i) ,y (i) ) is the i-th training sample, m is the size of the training set, Represents empirical error.

[0010] In some embodiments of the present disclosure, fitting the optimal objective function in the hypothesis space by minimizing the model error also includes: Regularized optimization is performed to obtain the optimal objective function, where λ is the regularization parameter and Ω(h) is the penalty term related to the complexity of the model.

[0011] According to the second aspect of the present disclosure, a Chinese text rationality detection method is provided, including: generating a specific type of Chinese text based on a generation model; inputting the generated specific type of text into a specific Chinese text rationality detection model to perform text meaning rationality detection to obtain a text rationality detection result, and the specific Chinese text rationality detection model is constructed based on the construction method of the specific Chinese text rationality detection model of the first aspect of the present disclosure.

[0012] In some embodiments of the present disclosure, the generated specific type of text is input into a specific Chinese text rationality detection model to perform text semantic rationality detection, and the text rationality detection result includes: extracting features of the generated specific type of text, and calculating the rationality data deviation of the text features based on the optimal objective function; if the rationality data deviation is within the error range preset by the model, the generated specific type of Chinese text is determined to be reasonable.

[0013] According to a third aspect of the present disclosure, a device for constructing a rationality detection model for a specific Chinese text is provided. The device includes at least one processor; and at least one memory storing a computer program. When the computer program is executed by at least one processor, the device: performs word segmentation on existing specific Chinese text data according to two different granularities to construct a priori knowledge phrase library; constructs a hypothesis space containing objective functions of different forms based on the priori knowledge phrase library; fits the optimal objective function in the hypothesis space by minimizing the model error; and constructs a specific Chinese text rationality detection model based on the optimal objective function, so as to perform rationality detection on the generated Chinese text based on the Chinese text rationality detection model.

[0014] In some embodiments of the present disclosure, when the computer program is executed by at least one processor, the device causes the device to perform the following operations to segment existing specific Chinese text data according to two different granularities to construct a priori knowledge phrase library: perform data cleaning, denoising and standardization preprocessing on the existing specific Chinese text data; perform maximum granularity word segmentation and minimum granularity word segmentation on the preprocessed Chinese text data to convert the text into phrases; and perform statistics on the frequency and distribution of occurrence of phrases to construct a priori knowledge phrase library.

[0015] In some embodiments of the present disclosure, when the computer program is executed by at least one processor, the device constructs a hypothesis space containing different forms of objective functions based on the prior knowledge phrase library through the following operations: converting the prior knowledge phrase library into vectorized text feature data; selecting different kernel functions according to the relationship between the text feature data, the kernel functions including linear kernel functions, polynomial kernel functions, and Gaussian kernel functions; and fitting the kernel function through non-parametric regression to obtain a hypothesis space containing different forms of objective functions, the non-parametric regression including kernel density estimation, local weighted scatter plot smoothing, and spline regression.

[0016] In some embodiments of the present disclosure, when the computer program is executed by at least one processor, the device performs the following operations to fit the optimal objective function:

[0017] pass Find the objective function with the smallest generalization error in the hypothesis space, where l(h(x), y) is the loss function of the selected objective function h(x), H is the hypothesis space, and P(x, y) is the joint probability distribution function of the input x and the label y;

[0018] pass Find the objective function with the minimum empirical error, where (x (i) ,y (i) ) is the i-th training sample, m is the size of the training set, represents the empirical error;

[0019] pass Regularized optimization is performed to obtain the optimal objective function, where λ is the regularization parameter and Ω(h) is the penalty term related to the complexity of the model.

[0020] According to a fourth aspect of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein the computer program, when executed by a processor, implements the steps of the method for constructing a specific Chinese text rationality detection model according to the first aspect of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly described below. It should be noted that the drawings described below only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure, wherein:

[0022] Figure 1 An exemplary flow chart showing a method 100 for constructing a specific Chinese text rationality detection model according to an embodiment of the present disclosure;

[0023] Figure 2 A schematic diagram of a process for detecting rationality of Chinese text according to an embodiment of the present disclosure is shown;

[0024] Figure 3 It is a schematic block diagram of a device 300 for constructing a specific Chinese text rationality detection model according to an embodiment of the present disclosure.

[0025] It should be noted that the elements in the drawings are schematic and not drawn to scale. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work also fall within the scope of protection of the present disclosure.

[0027] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the specification and the relevant art, and will not be interpreted in an idealized or overly formal manner unless expressly so defined herein.

[0028] Specific Chinese texts are Chinese language texts with specific content, style, or goals according to different fields, contexts, forms, emotions, or task requirements. For example, technical documents, legal texts, medical texts, etc. usually contain professional terms and require knowledge of the relevant fields to understand. Existing Chinese text analysis often relies on simple correlation analysis, resulting in problems such as thin data quantification and shallow analysis.

[0029] In view of the problems of thin data quantification and shallow data in specific Chinese text analysis, embodiments of the present disclosure propose a method for constructing a rationality detection model for specific Chinese texts. This method improves the accuracy and depth of text analysis by fitting the objective function with a non-parametric model. Figure 1 An exemplary flowchart of a method 100 for constructing a rationality detection model for specific Chinese texts according to an embodiment of the present disclosure is shown.

[0030] At Figure 1 block S102, the existing specific Chinese text data is segmented according to two different granularities to construct a prior knowledge phrase library.

[0031] First, the quality of the prior knowledge base is crucial for subsequent analysis. Prior knowledge mainly comes from the processing of existing specific types of Chinese texts. It is necessary to clarify the type of text (such as news, novels, academic papers, social media, etc.). The language features, structures, and styles of each type of text are different. By collecting specific types of Chinese text data, representative Chinese text data can be obtained.

[0032] Before segmentation, preprocessing such as data cleaning, denoising, and standardization is performed on the collected specific type of Chinese text data. For example, punctuation marks, special symbols, irrelevant characters, etc. are removed, high-frequency words without substantial meaning such as "de", "le", "shi" are removed, and different spellings or formats are unified.

[0033] Then, the preprocessed text data is segmented according to the maximum granularity and the minimum granularity, and the text is converted into phrases. Among them, the minimum granularity is the character-level granularity, and each word is analyzed as an independent unit. Extracting single words or words from the text can capture more detailed information and provide more phrase combination possibilities. The maximum granularity is the vocabulary-level granularity, and the text is segmented into words or phrases according to semantic units. For example, long phrases, idioms, phrases, proper nouns, etc. are recognized, and this type of word segmentation can retain more contextual information. Word segmentation tools (such as jieba, THULAC, HanLP, etc.) can be used to segment the text into phrases of different granularities, and part-of-speech tagging is performed to form a labeled phrase library. According to an embodiment of the present disclosure, according to task requirements, the phrase library can be further expanded to add some professional field vocabulary, synonyms, antonyms, entity information, etc.

[0034] Finally, the frequency and distribution of phrases are counted to construct a priori knowledge phrase library. For example, common collocation phrases and their frequencies are counted to provide data support for the design of the objective function.

[0035] By performing word segmentation of Chinese text data at the maximum and minimum granularity, more comprehensive phrase information can be covered, laying the foundation for building a diverse prior knowledge phrase library. Ultimately, these phrases can not only support effective text generation, but also provide rich corpus and contextual knowledge for various natural language processing tasks.

[0036] Later in Figure 1 In block S104 , a hypothesis space containing different forms of objective functions is constructed based on the prior knowledge phrase library.

[0037] Since the prior knowledge phrase library contains multi-granular phrases obtained through word segmentation, which cover different levels and expressions of the text, the design of the hypothesis space depends on the characteristics of the prior knowledge. In other words, the hypothesis space H can be regarded as a function space that represents all possible mappings from input text to output labels, which can be designed based on text structure, vocabulary features, contextual information, etc.

[0038] In some embodiments of the present disclosure, the prior knowledge phrase library can be converted into vectorized text feature data. Different kernel functions are selected according to the relationship between the text feature data to construct a hypothesis space containing different forms of objective functions, and the kernel functions include linear kernel functions, polynomial kernel functions, and Gaussian kernel functions. The kernel function is fitted by non-parametric regression to obtain a hypothesis space containing different forms of objective functions, and the non-parametric regression includes kernel density estimation, local weighted scatter plot smoothing, and spline regression.

[0039] Specifically, the word segmentation of the text can be used as an independent feature based on the bag-of-words model to calculate the frequency of occurrence of each word. The importance of words in the text can be measured based on the TF-IDF weighting method. For the phrases in the prior knowledge phrase library, the TF-IDF value of each phrase in different texts can be calculated to obtain the weight of each phrase. You can also use a pre-trained word embedding model (such as Word2Vec) to map each word or phrase in the prior knowledge phrase library into a vector, which can capture the semantic similarity of words.

[0040] Select different kernel functions based on the relationship between text feature data. For example, the linear kernel function is suitable for situations where there is a linear relationship between text feature data. For vectorized text features, if the distribution of the data is relatively simple, using a linear kernel may bring good results. The polynomial kernel function is suitable for situations where there is a polynomial relationship between data. It can capture high-order relationships between data. The Gaussian kernel function is usually used when the relationship between data is very complex or difficult to model with a low-order polynomial, and can capture local similarities between data.

[0041] Once a suitable kernel function is selected, the next task is to fit the data using nonparametric regression. Nonparametric regression does not assume the specific form of the target function, but learns the law of the target function through the data distribution. Common nonparametric regression methods include kernel density estimation, local weighted scatter plot smoothing, and spline regression. Among them, kernel density estimation (KDE) estimates the probability density distribution of data through the kernel function, which can provide a smooth transition between data points and is suitable for estimating the distribution of word frequencies in the text generation process. Local weighted scatter plot smoothing can capture local nonlinear relationships by weighted fitting of data in local areas, which is suitable for fitting local changes in text generation. Spline regression fits data in each interval by means of piecewise polynomials to ensure the smoothness of the fitting function.

[0042] Through the above process, a set of objective functions in different forms can be obtained, which define the hypothesis space for generating data. The hypothesis space is the space for model selection, and different kernel functions and regression methods correspond to different forms of objective functions. Therefore, the embodiment of the present disclosure can effectively model text data and learn a variety of different objective functions by converting the prior knowledge phrase library into vector feature data, combining the selection of kernel functions and the fitting process of non-parametric regression.

[0043] Next, in block S106 , an optimal objective function is fitted in the hypothesis space by minimizing the model error.

[0044] In some embodiments of the present disclosure, the objective function is used to measure the rationality of the meaning of the generated Chinese text, and its core is to ensure that the generated text conforms to the basic rules of the Chinese language in terms of grammar, semantics, logic and contextual consistency. The most appropriate objective function is selected by comparing the fit and generalization ability of these non-probabilistic models in the hypothesis space. If some models perform poorly, the model can be optimized by adjusting the parameters of the kernel function, adding correction terms or introducing regularization methods.

[0045] Specifically, it can be done by Find the objective function with the smallest generalization error in the hypothesis space, where l(h(x), y) is the loss function of the selected objective function h(x), H is the hypothesis space, and P(x, y) is the joint probability distribution function of the input x and the label y.

[0046] Since we do not always know the complete probability distribution P(x,y) in practice, we can estimate the error through the training set. The empirical error is an error metric based on the training set, which estimates the performance of the model by calculating the average loss in the training data. Find the objective function with the minimum empirical error, where (x (i) ,y (i) ) is the i-th training sample, m is the size of the training set, Represents empirical error.

[0047] Furthermore, in order to avoid overfitting, Regularized optimization is performed to obtain the optimal objective function, where λ is the regularization parameter and Ω(h) is the penalty term related to the complexity of the model.

[0048] Through the above process, the final model can be used to test and analyze the rationality of the semantics of a specific type of Chinese text generation. The output of the model can be used as an indicator of text generation quality to help determine whether the generated text conforms to the language norms and whether it has a reasonable grammatical and semantic structure.

[0049] Finally, in block S108 , a specific Chinese text rationality detection model is constructed based on the optimal objective function, so as to perform rationality detection on the generated Chinese text based on the Chinese text rationality detection model.

[0050] The embodiment of the present disclosure also provides a Chinese text rationality detection method, comprising: generating a specific type of Chinese text based on a generation model, inputting the generated specific type of text into the Chinese text rationality detection model to perform text meaning rationality detection, and obtaining a text rationality detection result.

[0051] Figure 2FIG. 1 is a flow chart showing a process for detecting the rationality of Chinese text according to an embodiment of the present disclosure. Figure 2 As shown, firstly, word segmentation of different granularities is performed based on the existing specific Chinese text data to obtain a priori knowledge phrase library, and a hypothesis space containing objective functions of different forms is fitted through kernel functions and non-parametric regression. The optimal objective function in the hypothesis space is determined through model optimization, so as to construct a specific Chinese text rationality detection model.

[0052] Then, the model application phase is entered, and a specific type of Chinese text is generated based on the generation model. The generated specific type of text is input into the specific Chinese text rationality detection model to perform text meaning rationality detection, and obtain the text rationality detection result. Specifically, feature extraction is performed on the generated specific type of text, and the rationality data deviation of the text feature is calculated based on the optimal objective function. If the rationality data deviation is within the error range preset by the model, the generated specific type of Chinese text is judged to be rational.

[0053] Figure 3 3 is a schematic block diagram of a device 300 for constructing a specific Chinese text rationality detection model according to an embodiment of the present disclosure. Figure 3 As shown, the device 300 may include a processor 310 and a memory 320 storing a computer program. When the computer program is executed by the processor 310, the device 300 may perform the following operations: Figure 1 The steps of the construction method 100 of the rationality detection model of a specific Chinese text shown. In one example, the device 300 can be a computer device or a cloud computing node. The device 300 can segment the existing specific Chinese text data according to two different granularities to construct a priori knowledge phrase library; then, the device 300 can construct a hypothesis space containing different forms of objective functions based on the priori knowledge phrase library; by minimizing the model error, the device 300 can fit the optimal objective function in the hypothesis space; finally, the device 300 can construct a specific Chinese text rationality detection model based on the optimal objective function, so as to perform rationality detection on the generated Chinese text based on the Chinese text rationality detection model.

[0054] In some embodiments of the present disclosure, the device 300 can perform data cleaning, denoising and standardization preprocessing on existing specific Chinese text data; perform maximum granularity word segmentation and minimum granularity word segmentation on the preprocessed Chinese text data to convert the text into phrases; and perform statistics on the frequency and distribution of occurrence of phrases to construct a priori knowledge phrase library.

[0055] In some embodiments of the present disclosure, the device 300 can convert the prior knowledge phrase library into vectorized text feature data. After the text feature data is converted into a vector, the kernel method can be used to perform nonlinear mapping on the text data, thereby improving the complexity and expression ability of the model. The device 300 can select different kernel functions according to the relationship between the text feature data, and the kernel functions include linear kernel functions, polynomial kernel functions, and Gaussian kernel functions. Among them, the linear kernel function is suitable for the case where the data itself is already linearly separable, the polynomial kernel function can expand the ability of the linear kernel, so that the data becomes linearly separable in a higher dimensional space, and is suitable for scenarios where there is a polynomial relationship between the data. The Gaussian kernel function is often used in situations where there is a complex nonlinear relationship between the data, and can make the data linearly separable by mapping to an infinite dimensional space. By selecting a suitable kernel function, the text data can be better separated and represented in a high dimensional space, thereby providing strong support for the next step of nonparametric regression modeling.

[0056] The nonparametric regression method does not assume a specific distribution form of the data and can flexibly model complex relationships. The nonparametric regression method combined with the kernel function can be used to fit the kernel function to obtain the hypothesis space of the target function. The device 300 can fit the kernel function through nonparametric regression to obtain a hypothesis space containing different forms of the target function. The nonparametric regression includes kernel density estimation, local weighted scatter plot smoothing (LOESS), and spline regression. Among them, kernel density estimation can be used to estimate the density of the target function and infer the form of the target function through these density estimates. LOWESS smoothes data through local weighted least squares and is suitable for situations with strong nonlinearity and noise. In the regression analysis of text data, this method can better fit complex relationships. Spline regression can capture local changes in data and avoid overfitting problems.

[0057] By choosing appropriate kernel functions and nonparametric regression methods, a hypothesis space containing different forms of the objective function can be obtained. For example, in kernel density estimation, the hypothesis space of the objective function can be defined by different kernel functions (such as Gaussian kernel, Epanechnikov kernel, etc.) and adjustments with different bandwidth parameters. In local weighted scatter plot smoothing or spline regression, the fitting results can be affected by adjusting the regression window size, smoothing parameters, etc.

[0058] In some embodiments of the present disclosure, the apparatus 300 may fit the optimal objective function by the following operations:

[0059] pass Find the objective function with the smallest generalization error in the hypothesis space, where l(h(x), y) is the loss function of the selected objective function h(x), H is the hypothesis space, and P(x, y) is the joint probability distribution function of the input x and the label y;

[0060] pass Find the objective function with the minimum empirical error, where (x (i) ,y (i) ) is the i-th training sample, m is the size of the training set, represents the empirical error;

[0061] pass Regularized optimization is performed to obtain the optimal objective function, where λ is the regularization parameter and Ω(h) is the penalty term related to the complexity of the model.

[0062] In an embodiment of the present disclosure, the processor 310 may be, for example, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a processor based on a multi-core processor architecture, etc. The memory 320 may be any type of memory implemented using data storage technology, including but not limited to random access memory, read-only memory, semiconductor-based memory, flash memory, disk storage, etc.

[0063] In addition, in the embodiment of the present disclosure, the device 300 may also include an input device 330, such as a keyboard, a mouse, etc., for inputting existing specific Chinese text data. In addition, the device 300 may also include an output device 340, such as a display, etc., for outputting text rationality detection results.

[0064] In other embodiments of the present disclosure, a computer-readable storage medium storing a computer program is further provided, wherein the computer program can achieve the following when executed by a processor: Figure 1 The steps of the method 100 for constructing a specific Chinese text rationality detection model are shown.

[0065] In summary, according to the construction method and device of the rationality detection model of a specific Chinese text of the embodiment of the present disclosure, the text is segmented by two different granularity word segmentations, which can improve the understanding ability of the specific Chinese text, so that the model can capture more levels of language information, enhance the semantic expression ability of the text, and thus improve the accuracy and robustness of rationality detection. By fitting a hypothesis space with a non-parametric model to various forms of objective functions, the model can be trained under multiple different objective functions to obtain a more comprehensive generalization ability. This diverse hypothesis space can help the model adapt to a wider range of text types, solve complex rationality detection problems, and improve the flexibility and accuracy of the model. Through the rationality detection model constructed based on the optimal objective function, it is possible to accurately perform rationality detection on the text after it is generated, not only to evaluate the rationality of the grammatical structure, but also to check the logic of the text, the consistency of the context, and whether it conforms to common sense, which can greatly improve the output quality of the text generation system and ensure its grammatical and logical accuracy.

[0066] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device and method according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0067] Unless the context clearly indicates otherwise, the singular form of the words used herein and in the appended claims includes the plural and vice versa. Thus, when referring to the singular, the plural form of the corresponding term is generally included. Similarly, the words "comprise" and "include" are to be interpreted as inclusive rather than exclusive. Likewise, the terms "include" and "or" should be interpreted as inclusive unless such interpretation is expressly prohibited herein. Where the term "example" is used herein, particularly when it is located after a group of terms, the "example" is merely exemplary and illustrative and should not be considered exclusive or comprehensive.

[0068] Further aspects and scopes of adaptability become apparent from the description provided herein. It should be understood that various aspects of the present application can be implemented individually or in combination with one or more other aspects. It should also be understood that the description and specific embodiments herein are intended for purposes of illustration only and are not intended to limit the scope of the present application.

[0069] Several embodiments of the present disclosure are described in detail above, but it is obvious that those skilled in the art can make various modifications and variations to the embodiments of the present disclosure without departing from the spirit and scope of the present disclosure. The protection scope of the present disclosure is defined by the attached claims.

Claims

1. A method for constructing a specific Chinese text rationality detection model, characterized in that: include: Segment the existing specific Chinese text data according to two different granularities and build a priori knowledge phrase library; Constructing a hypothesis space containing different forms of objective functions based on the prior knowledge phrase library; Fitting the optimal objective function in the hypothesis space by minimizing the model error; as well as A specific Chinese text rationality detection model is constructed based on the optimal objective function, so as to perform rationality detection on the generated Chinese text based on the Chinese text rationality detection model.

2. The method for constructing a specific Chinese text rationality detection model according to claim 1, characterized in that: The word segmentation of the existing specific Chinese text data according to two different granularities and the construction of a priori knowledge phrase library include: Perform data cleaning, denoising and standardization preprocessing on existing specific Chinese text data; Performing maximum granularity word segmentation and minimum granularity word segmentation on the preprocessed Chinese text data to convert the text into phrases; and The occurrence frequency and distribution of the phrases are statistically analyzed to construct a priori knowledge phrase library.

3. The method for constructing a specific Chinese text rationality detection model according to claim 1, characterized in that: The construction of a hypothesis space containing different forms of objective functions based on the prior knowledge phrase library includes: Converting the prior knowledge phrase library into vectorized text feature data; Select different kernel functions according to the relationship between the text feature data, the kernel functions including linear kernel functions, polynomial kernel functions, and Gaussian kernel functions; and The kernel function is fitted by nonparametric regression to obtain a hypothesis space containing target functions in different forms, wherein the nonparametric regression includes kernel density estimation, local weighted scatter plot smoothing, and spline regression.

4. The method for constructing a specific Chinese text rationality detection model according to claim 1, characterized in that: The step of fitting the optimal objective function in the hypothesis space by minimizing the model error comprises: pass Find the objective function with the smallest generalization error in the hypothesis space, where l(h(x), y) is the loss function of the selected objective function h(x), H is the hypothesis space, and P(x, y) is the joint probability distribution function of the input x and the label y.

5. The method for constructing a specific Chinese text rationality detection model according to claim 4, characterized in that: The step of fitting the optimal objective function in the hypothesis space by minimizing the model error further comprises: pass Find the objective function with the minimum empirical error, where (x (i) ,y (i) ) is the i-th training sample, m is the size of the training set, Represents empirical error.

6. The method for constructing a specific Chinese text rationality detection model according to claim 5, characterized in that: The step of fitting the optimal objective function in the hypothesis space by minimizing the model error further comprises: pass Regularized optimization is performed to obtain the optimal objective function, where λ is the regularization parameter and Ω(h) is the penalty term related to the complexity of the model.

7. A Chinese text rationality detection method, characterized in that: include: Generate specific types of Chinese text based on the generative model; as well as The generated specific type of text is input into a specific Chinese text rationality detection model to perform text meaning rationality detection to obtain a text rationality detection result. The specific Chinese text rationality detection model is constructed based on the construction method of the specific Chinese text rationality detection model as described in any one of claims 1-6.

8. The Chinese text rationality detection method according to claim 7 is characterized in that: The specific type of text generated is input into the specific Chinese text rationality detection model to perform text meaning rationality detection, and the text rationality detection result obtained includes: Extracting features of the generated specific type of text, and calculating the rationality data deviation of the text features based on the optimal objective function; If the rationality data deviation is within the error range preset by the model, it is determined that the generated Chinese text of the specific type is rational.

9. A device for constructing a specific Chinese text rationality detection model, characterized in that: The device comprises: at least one processor; and at least one memory storing a computer program; Wherein, when the computer program is executed by the at least one processor, the device executes the steps of the method for constructing a specific Chinese text rationality detection model according to any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the method for constructing a specific Chinese text rationality detection model according to any one of claims 1 to 6.

Images