Text information amount evaluation method and device, equipment and medium

Abstract

This disclosure relates to a method, apparatus, device, and medium for evaluating the information content of text. The method includes: acquiring the target text to be evaluated; determining the occurrence probability of each segmented text in the target text, and determining the original information entropy of the target text based on the occurrence probability of each segmented text; correcting the original information entropy based on the text length and correction coefficient of the target text to obtain the target information entropy; and standardizing the target information entropy according to a predetermined evaluation threshold. This disclosure can improve the computational efficiency and accuracy of information content evaluation.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for evaluating the information content of text. Background Technology

[0002] With the rapid development of Artificial Intelligence (AI) technology, data has become a key element supporting the training and optimization of AI models. Among various types of data, text data plays an important role in fields such as natural language processing and machine learning due to its wide range and diversity. The quality of text data directly affects the training effect and practical application performance of models. Therefore, effectively evaluating the quality of text data, especially the assessment of the amount of information contained in the text, is particularly crucial, as the amount of information directly reflects the value density and usability of the data.

[0003] Currently, existing methods for assessing text information content all have their limitations. For example, manual assessment is time-consuming, labor-intensive, and highly subjective; language model-based assessment has high computational costs and poor interpretability; and word frequency statistics-based methods are too simplistic and cannot accurately reflect the actual information content of the text.

[0004] Therefore, there is an urgent need to develop a text information content evaluation method that is based on solid theoretical foundations, highly efficient, and well interpretable. Summary of the Invention

[0005] To address the aforementioned technical problems, this disclosure provides a method, apparatus, device, and medium for evaluating the information content of text, which can achieve an objective, efficient, and interpretable evaluation of the information content of text.

[0006] According to one aspect of this disclosure, a method for evaluating the information content of a text is provided, comprising:

[0007] Obtain the target text to be evaluated;

[0008] Determine the occurrence probability of each segmented text in the target text, and determine the original information entropy of the target text based on the occurrence probability of each segmented text;

[0009] Based on the text length and correction coefficient of the target text, the original information entropy is corrected to obtain the target information entropy;

[0010] The target information entropy is standardized according to a predetermined evaluation threshold.

[0011] According to another aspect of this disclosure, a text information content evaluation device is also provided, comprising:

[0012] The text acquisition module is used to acquire the target text to be evaluated;

[0013] The information entropy determination module is used to determine the occurrence probability of each segmented text in the target text, and to determine the original information entropy of the target text based on the occurrence probability of each segmented text;

[0014] The information entropy correction module is used to correct the original information entropy based on the text length and correction coefficient of the target text to obtain the target information entropy;

[0015] The information entropy standardization module is used to standardize the target information entropy according to a predetermined evaluation threshold.

[0016] According to another aspect of this disclosure, an electronic device is also provided, the electronic device comprising:

[0017] processor;

[0018] Memory used to store the processor's executable instructions;

[0019] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the above method.

[0020] According to another aspect of this disclosure, a computer-readable storage medium is also provided, the storage medium storing a computer program for performing the above-described method.

[0021] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0022] The technical solution provided in this disclosure includes: acquiring the target text to be evaluated; determining the occurrence probability of each segmented text in the target text, and determining the original information entropy of the target text based on the occurrence probability of each segmented text; correcting the original information entropy based on the text length and correction coefficient of the target text to obtain the target information entropy; and standardizing the target information entropy according to a predetermined evaluation threshold.

[0023] In this technical solution, the original information entropy of the target text can be determined relatively efficiently by utilizing the occurrence probability of each segmented text. Then, considering the impact of text length on the accuracy of information entropy, the original information entropy is corrected based on text length and a correction coefficient, thus improving the accuracy of the target information entropy. Finally, by standardizing the target information entropy, the standardized information entropy becomes easier to understand and use, exhibiting good interpretability and verifiability. Therefore, the text information content evaluation method provided in this disclosure is objective and reliable, computationally efficient and accurate, requires no complex language models, and can adapt to the evaluation needs of different types of text. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0025] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of the information content evaluation method for text described in the embodiments of this disclosure;

[0027] Figure 2 This is a structural block diagram of the text information evaluation device described in the embodiments of this disclosure;

[0028] Figure 3 This is a schematic diagram of the structure of the electronic device described in an embodiment of this disclosure. Detailed Implementation

[0029] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0030] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0031] In existing methods for assessing text information content, manual assessment is time-consuming, labor-intensive, and highly subjective, lacking objective quantitative standards; language model-based assessments are computationally expensive and yield unstable results; and word frequency statistics-based methods are too simplistic and cannot accurately reflect the actual information content of the text. To address at least one of these problems, embodiments of this disclosure provide a method, apparatus, device, and medium for assessing text information content. Through rigorous mathematical theory and statistical analysis, this method achieves an objective, efficient, and interpretable assessment of text information content. For ease of understanding, embodiments of this disclosure are described in detail below.

[0032] Figure 1This flowchart illustrates a method for evaluating the information content of text, provided as an embodiment of this disclosure. This method is applicable to multiple fields requiring text information content evaluation, such as answer quality evaluation in intelligent question-answering systems, information integrity evaluation of machine translation results, summary quality evaluation in automatic summarizing systems, output quality control of text generation models, dataset screening, and quality control. This text information content evaluation method can be executed by a text information content evaluation device, which can be implemented using software and / or hardware. Figure 1 As shown, a method for evaluating the information content of text may include the following steps.

[0033] Step S102: Obtain the target text to be evaluated.

[0034] In this embodiment, the target text can be any text whose information content needs to be evaluated, such as question-and-answer pairs containing questions and answers, translated text, text from journals or books, etc. This embodiment can obtain the target text to be evaluated through various methods, such as downloading from the network, local storage, or manual uploading.

[0035] Step S104: Determine the occurrence probability of each segmented text in the target text, and determine the original information entropy of the target text based on the occurrence probability of each segmented text.

[0036] In one embodiment, the probability of occurrence of each segmented text in the target text can be determined as follows.

[0037] The target text is classified into languages ​​using a pre-trained text classification model to obtain the target language. For example, the fasttext model is used to detect and classify the target text into languages ​​such as Chinese, English, or other languages, thereby obtaining the target language corresponding to the target text.

[0038] The target text is segmented into multiple segmented texts based on a segmentation algorithm that matches the target language; and text content in languages ​​other than the target language is filtered out from the target text.

[0039] The probability of each segmented text appearing in the target text is determined based on the length of the target text and the number of times each segmented text appears in the target text.

[0040] In a specific example, the probability of the i-th segmented text (denoted as xi) appearing in the target text (denoted as T) can be determined by the following formula (denoted as p(xi)):

[0041] p(xi)=ni / N (1)

[0042] Where ni is the number of times the segmented text xi appears in the target text T, and N is the text length (i.e., the total number of characters) of the target text T.

[0043] Then, referring to the following formula (2), the original information entropy of the target text is determined based on the occurrence probability of each segmented text:

[0044] E 原始 =-Σp(xi)log(p(xi)) (2)

[0045] Information entropy (IE) is a metric used to measure the average amount of information contained in text, and log() refers to the logarithm to the base 2.

[0046] Step S106: Based on the text length and correction coefficient of the target text, the original information entropy is corrected to obtain the target information entropy.

[0047] This embodiment may include: determining target relevance based on the text length of the target text and a correction coefficient; wherein, target relevance is used to represent the correlation between the original information entropy and the text length of the target text; and correcting the original information entropy based on the target relevance to obtain the target information entropy.

[0048] In practical applications, for short texts (i.e., short texts), such as question-and-answer texts, the limited length restricts the diversity of characters, words, and sentence structures. Furthermore, the small sample size fails to fully represent all the features of the language, leading to unstable estimation of information entropy. In this situation, the following problems arise:

[0049] Limited character and vocabulary diversity: The number of unique characters or words that a short text may contain is limited, and information entropy increases rapidly with the increase of text length.

[0050] Significant statistical fluctuations: Due to the small amount of data, the impact of random factors on information entropy is more significant, resulting in a strong correlation between information entropy and text length, with the Spearman correlation coefficient possibly approaching 1.

[0051] Unlike short texts, for longer texts, such as entire papers, journal articles, or books, the types of characters and words contained in the text tend to be stable, and the probability distribution more accurately reflects the characteristics of the language. In this case, the following problems may arise:

[0052] Diversity tends to saturate: As text length increases, the frequency of new and unique characters or words decreases, and the rate of increase in information entropy slows down.

[0053] Information entropy tends to stabilize: Due to the stable probability distribution, the information entropy value also tends to a stable level, and further increases in text length have little impact on information entropy.

[0054] Decreased correlation: The correlation between information entropy and text length decreases, and Spearman correlation coefficient and p-value test may indicate that there is no significant association between the two.

[0055] Therefore, to avoid the information entropy score being strongly influenced by text length when calculating the information entropy of short texts, this embodiment can use a secondary calculation of information entropy to correct the original information entropy.

[0056] During implementation, if the target text length is less than a preset length threshold, the target text is determined to be short text. In this case, the original information entropy is corrected according to the following correction formula to obtain the target information entropy.

[0057]

[0058] Where E represents the target information entropy, E 原始 Let represent the original information entropy, L represent the text length of the target text, and γ represent the correction coefficient. Indicates target relevance.

[0059] Of course, according to the above modified formula (3), it can also be understood as: calculating the original information entropy E 原始 The product of this value and the text length L yields an intermediate value. Dividing this intermediate value by the correction coefficient γ gives the corrected target information entropy E. The correction coefficient γ was determined through analysis of a large amount of internal enterprise data; the specific determination process will not be detailed here.

[0060] Step S108: Standardize the target information entropy according to a predetermined evaluation threshold.

[0061] In this embodiment, the process of determining the evaluation threshold is first described, as shown below.

[0062] (1) Obtain the sample information entropy of multiple text samples. Here, a text sample is text of the same type as the target text, such as a question-and-answer pair or a book text. Methods for determining the sample information entropy of a text sample may include:

[0063] Determine the word segmented text y in the text sample i The probability of occurrence p(y) i And refer to the following formula based on the occurrence probability p(y) of each segmented text. i Determine the sample information entropy H(Y) of the text sample:

[0064]

[0065] (2) Determine the distribution characteristics of the sample information entropy; wherein, the distribution characteristics include: the skewness, kurtosis and normality test results of the sample information entropy.

[0066] As an example, the analysis of the distribution characteristics of information entropy from multiple samples can mainly include the following:

[0067] The skewness and kurtosis of the information entropy of multiple samples are calculated to understand the symmetry and tail characteristics of the sample information entropy distribution. Skewness describes the degree to which the probability distribution deviates from symmetry; positive skewness indicates a longer right tail, and negative skewness indicates a longer left tail. Kurtosis describes the steepness of the probability distribution curve; compared to a normal distribution, high kurtosis indicates a steeper distribution, and low kurtosis indicates a flatter distribution.

[0068] The Shapiro-Wilk normality test is performed on the information entropy of multiple samples to determine whether the sample information entropy conforms to a normal distribution. The Shapiro-Wilk normality test is a statistical test method used to determine whether the sample information entropy originates from a normally distributed population by calculating the degree of fit between the sample information entropy and a normal distribution.

[0069] (3) Determine the various distribution types that the sample information entropy conforms to based on the distribution characteristics. Specifically, based on the above skewness, kurtosis and normality test results, determine the top N distribution types that the sample information entropy best conforms to or is closest to, such as normal distribution, log-normal distribution, skewed normal distribution and Gamma distribution, etc.

[0070] The Gamma distribution is a continuous probability distribution determined by shape and scale parameters, and is often used to describe waiting time or positively skewed data. The Log-normal distribution: If the logarithm of a random variable follows a normal distribution, then the random variable follows a log-normal distribution; it is often used to describe data in fields such as finance and biology.

[0071] (4) Use probability distribution fitting functions of various distribution types to fit the sample information entropy to obtain various fitting results.

[0072] Each distribution type has its own probability distribution function. Therefore, the probability distribution fitting function of each distribution type can be used to fit the sample information entropy. Accordingly, each probability distribution function can yield its own fitting result.

[0073] (5) Use PP plots and QQ plots to test each fitting result in order to determine the target probability distribution function and its corresponding target fitting result among multiple probability distribution functions.

[0074] The PP plot (Probability-Probability Plot) is a graph drawn based on the relationship between the cumulative proportion of a variable and the cumulative proportion of a specified distribution. The PP plot can be used to examine whether the fitting result of the sample information entropy conforms to a certain theoretical distribution. When the fitting result conforms to a certain theoretical distribution, the points in the PP plot approximately form a straight line.

[0075] The QQ plot (quantile-quantile plot) and PP plot serve the same purpose, but differ in their testing methods. To use a QQ plot to test whether the fitting result of the sample information entropy conforms to a certain theoretical distribution, one can observe whether the points on the QQ plot are approximately near a straight line, and whether the slope of this line is the standard deviation and the intercept is the mean. The QQ plot can also provide rough information about sample skewness and kurtosis.

[0076] Taking the Probability Graph (PP) plot as an example, for a fitting result obtained using the probability distribution function of a normal distribution, a PP plot is constructed using this fitting result. If all points in the constructed PP plot approximately form a straight line, then the fitting result is determined to conform to a normal distribution. The verification of other fitting results is similar and will not be described in detail here.

[0077] Each fitting result is tested using the above embodiments. Based on the test results of the PP plot and QQ plot, if it is determined that the fitting result matches its corresponding theoretical distribution, then the fitting result is determined as the target fitting result, and the probability distribution function used by the target fitting result is determined as the target probability distribution function.

[0078] (6) Based on the target fitting results and different statistical algorithms, various candidate thresholds are determined; among them, the statistical algorithms include: interquartile range, 3sigma model and confidence interval.

[0079] In this embodiment, examples of determining multiple candidate thresholds based on the target fitting results and different statistical algorithms are shown below.

[0080] Example 1. The first candidate threshold is determined based on the target fitting results and the interquartile range (IQR). Specifically, the IQR is used to determine the difference between the third quartile and the first quartile; in other words, the IQR equals the upper quartile Q3 minus the lower quartile Q1. After arranging the sample information entropy in the target fitting results from smallest to largest, 25% of the samples have an information entropy less than or equal to the lower quartile Q1. Based on this, the first candidate threshold is determined using the following formula:

[0081] T1 = Q1 - 1.5 × IQR (5)

[0082] Wherein, IQR is the interquartile range, the difference between the upper quartile Q3 and the lower quartile Q1: IQR = Q3 - Q1.

[0083] Example 2. Refer to the following formula to determine the second candidate threshold based on the target fitting result and the 3sigma model:

[0084] T2=μ-3σ (6)

[0085] Where μ is the mean and σ is the standard deviation. The mean μ is the sum of the information entropy of all samples divided by the number of data points, representing the average level of the sample information entropy; the standard deviation σ represents the dispersion of the sample information entropy, and is calculated as the square root of the sum of the squares of the differences between the information entropy of each sample and the mean.

[0086] Example 3. The third and fourth candidate thresholds are determined based on the target fit results and confidence intervals. Specifically, the third candidate threshold is determined by calculating the lower bound of the first confidence interval (CI) (e.g., 95%), and the fourth candidate threshold is determined by calculating the lower bound of the second confidence interval (e.g., 99%). The confidence interval is used to estimate the range of information entropy of the overall sample.

[0087] In one possible approach, when determining the mean M and standard deviation ST of the sample information entropy, the lower bound of the confidence interval is: a = Mn * ST. When calculating the 95% confidence interval, n = 1.96, and the third candidate threshold is calculated based on this; when calculating the 99% confidence interval, n = 2.576, and the fourth candidate threshold is calculated based on this.

[0088] In another possible approach, when obtaining the estimated value distribution using the Monte Carlo method, all estimated value samples are first sorted, with the lower confidence interval defined as 'a' being the lower % percentile value after sorting. When calculating the 95% confidence interval, lower = 2.5, and the third candidate threshold is calculated based on this; when calculating the 99% confidence interval, lower = 0.5, and the fourth candidate threshold is calculated based on this.

[0089] When the amount of sample information entropy data is large enough, the thresholds obtained by the two possible methods mentioned above are basically equal.

[0090] Based on the above embodiments, four candidate thresholds can be obtained.

[0091] (7) Determine the evaluation threshold from the candidate thresholds as the criterion for judging outliers.

[0092] This embodiment allows for manual review and verification of the various candidate thresholds, selecting the most suitable candidate threshold as the evaluation threshold for outlier detection. Alternatively, this embodiment can determine if any of the candidate thresholds are consistent with the others; if so, the consistent candidate threshold is selected as the most suitable evaluation threshold for outlier detection.

[0093] In this embodiment, the target information entropy can be standardized based on the evaluation threshold determined in the above embodiments. Different standardization methods are used for cases where the target information entropy is less than the evaluation threshold and cases where it is not less than the evaluation threshold.

[0094] In one approach, when the target information entropy is less than a predetermined evaluation threshold, a preset first algorithm is used, and the target information entropy is standardized based on the evaluation threshold and the preset minimum information entropy to obtain the first standardized information entropy.

[0095] In this embodiment, the determined evaluation threshold can be fixedly mapped to 60 points, and normalized using a piecewise linear transformation function (i.e., the first algorithm) to map the target information entropy to the score range of [0, 100]. Assuming that after the evaluation threshold is determined as described above, the evaluation threshold is represented as E_th, where E is the actual score of the target information entropy of the target text, and S is the percentage score after the target information entropy is mapped, i.e., the first standardized information entropy. In this case, the first algorithm shown in the following formula (7) is used to standardize the target information entropy to obtain the first standardized information entropy:

[0096] S1=60 / (E_th–E_min)*(E–E_min) (7)

[0097] Where S1 represents the first standardized information entropy, E_th represents the evaluation threshold, E_min represents the preset minimum information entropy, usually E_min = 0, and E represents the target information entropy.

[0098] In another approach, when the target information entropy is not less than a predetermined evaluation threshold, a pre-defined second algorithm is used, and the target information entropy is standardized based on the evaluation threshold and the pre-defined maximum information entropy to obtain the second standardized information entropy.

[0099] In this embodiment, to ensure that the remaining 40 percentage units are allocated to the remaining entropy value range, the target information entropy is standardized using the second algorithm shown in formula (8) to obtain the second standardized information entropy:

[0100] S2=60+(E–E_th) / (E_max–E_th)*40 (8)

[0101] Wherein, S2 represents the second standardized information entropy, E_th represents the evaluation threshold, E_max represents the preset maximum information entropy, which is the maximum information entropy in a set of information entropies, and E represents the target information entropy.

[0102] In summary, the text information content evaluation method provided in this disclosure includes: obtaining the target text to be evaluated; determining the occurrence probability of each segmented text in the target text, and determining the original information entropy of the target text based on the occurrence probability of each segmented text; correcting the original information entropy based on the text length and correction coefficient of the target text to obtain the target information entropy; and standardizing the target information entropy according to a predetermined evaluation threshold.

[0103] In this technical solution, the original information entropy of the target text can be determined relatively efficiently by utilizing the occurrence probability of each segmented text. Then, considering the impact of text length on the accuracy of information entropy, the original information entropy is corrected based on text length and a correction coefficient, thus improving the accuracy of the target information entropy. Finally, by standardizing the target information entropy, the standardized information entropy becomes easier to understand and use, exhibiting good interpretability and verifiability. Therefore, the text information content evaluation method provided in this disclosure is objective and reliable, computationally efficient and accurate, requires no complex language models, and can adapt to the evaluation needs of different types of text.

[0104] Figure 2 This is a structural block diagram of a text information content evaluation device provided in an embodiment of this disclosure. This device can be used to implement the text information content evaluation method provided in the above embodiments. Figure 2 As shown, a text information content evaluation device may include the following modules.

[0105] Text acquisition module 210 is used to acquire the target text to be evaluated;

[0106] Information entropy determination module 220 is used to determine the occurrence probability of each segmented text in the target text, and to determine the original information entropy of the target text based on the occurrence probability of each segmented text;

[0107] Information entropy correction module 230 is used to correct the original information entropy based on the text length and correction coefficient of the target text to obtain the target information entropy;

[0108] Information entropy standardization module 240 is used to standardize the target information entropy according to a predetermined evaluation threshold.

[0109] The device provided in this embodiment has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0110] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Figure 3 As shown, the electronic device 300 includes one or more processors 301 and memory 302.

[0111] The processor 301 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device 300 to perform desired functions.

[0112] The memory 302 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 301 may execute the program instructions to implement the text information content evaluation method of the embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0113] In one example, the electronic device 300 may also include an input device 303 and an output device 304, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0114] In addition, the input device 303 may also include, for example, a keyboard, a mouse, etc.

[0115] The output device 304 can output various information to the outside, including determined distance information, direction information, etc. The output device 304 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0116] Of course, for the sake of simplicity, Figure 3 Only some of the components of the electronic device 300 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 300 may include any other suitable components depending on the specific application.

[0117] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program for executing the above-described text information content evaluation method.

[0118] The present disclosure provides a computer program product for evaluating the information content of text, including a method, apparatus, electronic device, and medium. The program product includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0119] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0120] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating the information content of text, characterized in that, include: Obtain the target text to be evaluated; Determine the occurrence probability of each segmented text in the target text, and determine the original information entropy of the target text based on the occurrence probability of each segmented text; Based on the text length and correction coefficient of the target text, the original information entropy is corrected to obtain the target information entropy; The target information entropy is standardized according to a predetermined evaluation threshold; The process of determining the evaluation threshold includes: Obtain the sample information entropy of multiple text samples; Determine the distribution characteristics of the sample information entropy; wherein, the distribution characteristics include: the skewness, kurtosis, and normality test results of the sample information entropy; Based on the distribution characteristics, determine the various distribution types that the sample information entropy conforms to; The sample information entropy is fitted using probability distribution fitting functions of various distribution types to obtain multiple fitting results; The fitting results were tested using PP plots and QQ plots to determine the target probability distribution function and its corresponding target fitting result among multiple probability distribution functions. Multiple candidate thresholds are determined based on the target fitting results and different statistical algorithms; wherein, the statistical algorithms include: interquartile range, 3sigma model, and confidence interval; The evaluation threshold, which serves as the criterion for outlier judgment, is determined from the candidate thresholds. The standardization of the target information entropy based on a predetermined evaluation threshold includes: When the target information entropy is less than a predetermined evaluation threshold, a preset first algorithm is used and the target information entropy is standardized according to the evaluation threshold and the preset minimum information entropy to obtain a first standardized information entropy. The standardization of the target information entropy based on a predetermined evaluation threshold includes: When the target information entropy is not less than a predetermined evaluation threshold, a preset second algorithm is used, and the target information entropy is standardized according to the evaluation threshold and the preset maximum information entropy to obtain a second standardized information entropy. The first algorithm includes: Wherein, S1 represents the first standardized information entropy. This represents the evaluation threshold. E represents the preset minimum information entropy, and E represents the target information entropy. The second algorithm includes: Where S2 represents the second standardized information entropy, This represents the evaluation threshold. E represents the preset maximum information entropy, and E represents the target information entropy.

2. The method according to claim 1, characterized in that, The step of correcting the original information entropy based on the text length and correction coefficient of the target text to obtain the target information entropy includes: The target relevance is determined based on the text length of the target text and the correction coefficient; wherein, the target relevance is used to represent the correlation between the original information entropy and the text length of the target text; The original information entropy is corrected based on the target relevance to obtain the target information entropy.

3. The method according to claim 1, characterized in that, Determining the probability of occurrence of each segmented text in the target text includes: The target text is classified into languages ​​using a pre-trained text classification model to obtain the target language. The target text is segmented according to a word segmentation algorithm that matches the target language, resulting in multiple segmented texts. The probability of each segmented text appearing in the target text is determined based on the text length of the target text and the number of times each segmented text appears in the target text.

4. A device for evaluating the information content of text, characterized in that, include: The text acquisition module is used to acquire the target text to be evaluated; The information entropy determination module is used to determine the occurrence probability of each segmented text in the target text, and to determine the original information entropy of the target text based on the occurrence probability of each segmented text; The information entropy correction module is used to correct the original information entropy based on the text length and correction coefficient of the target text to obtain the target information entropy; An information entropy standardization module is used to standardize the target information entropy according to a predetermined evaluation threshold; The apparatus includes: a process for determining the evaluation threshold, the process including: Obtain the sample information entropy of multiple text samples; Determine the distribution characteristics of the sample information entropy; wherein, the distribution characteristics include: the skewness, kurtosis, and normality test results of the sample information entropy; Based on the distribution characteristics, determine the various distribution types that the sample information entropy conforms to; The sample information entropy is fitted using probability distribution fitting functions of various distribution types to obtain multiple fitting results; The fitting results were tested using PP plots and QQ plots to determine the target probability distribution function and its corresponding target fitting result among multiple probability distribution functions. Multiple candidate thresholds are determined based on the target fitting results and different statistical algorithms; wherein, the statistical algorithms include: interquartile range, 3sigma model, and confidence interval; The evaluation threshold, which serves as the criterion for outlier judgment, is determined from the candidate thresholds. The information entropy standardization module is also used for: When the target information entropy is less than a predetermined evaluation threshold, a preset first algorithm is used and the target information entropy is standardized according to the evaluation threshold and the preset minimum information entropy to obtain a first standardized information entropy. The standardization of the target information entropy based on a predetermined evaluation threshold includes: When the target information entropy is not less than a predetermined evaluation threshold, a preset second algorithm is used, and the target information entropy is standardized according to the evaluation threshold and the preset maximum information entropy to obtain a second standardized information entropy. The first algorithm includes: Wherein, S1 represents the first standardized information entropy. This represents the evaluation threshold. E represents the preset minimum information entropy, and E represents the target information entropy. The second algorithm includes: Where S2 represents the second standardized information entropy, This represents the evaluation threshold. E represents the preset maximum information entropy, and E represents the target information entropy.

5. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method as described in any one of claims 1-3.

Images