Text information amount evaluation method and device, equipment and medium

Abstract

The invention relates to a text information amount evaluation method and device, equipment and a medium. The method comprises the steps of obtaining a to-be-evaluated target text; determining the occurrence probability of each word segmentation text in the target text, and determining the original information entropy of the target text based on the occurrence probability of each word segmentation text; based on the text length and the correction coefficient of the target text, correcting the original information entropy to obtain a target information entropy; and standardizing the target information entropy according to a predetermined evaluation threshold. According to the invention, the calculation efficiency and accuracy of information amount evaluation can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for evaluating the information content of a text. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, data has become a key factor in supporting AI model training and optimization. Among all types of data, text data plays an important role in natural language processing, machine learning and other fields due to its extensiveness and diversity. The quality of text data directly affects the training effect and actual application performance of the model. Therefore, it is particularly important to effectively evaluate the quality of text data, especially the amount of information contained in the text, which directly reflects the value density and use value of the data.

[0003] At present, some existing text information evaluation methods have their own limitations. For example, manual evaluation is time-consuming, labor-intensive and highly subjective; evaluation based on language models has high computational costs and poor interpretability; and methods based on word frequency statistics are too simple and cannot accurately reflect the actual information content of the text.

[0004] Therefore, it is urgent to develop a text information evaluation method based on a solid theoretical foundation, with high efficiency and good interpretability. Summary of the invention

[0005] In order to solve the above technical problems, the present disclosure provides a method, device, equipment and medium for evaluating the information content of a text, which can achieve an objective, efficient and explainable evaluation of the information content of the text.

[0006] According to one aspect of the present disclosure, a method for evaluating the amount of information in a text is provided, comprising:

[0007] Get the target text to be evaluated;

[0008] Determine the occurrence probability of each segmented text in the target text, and determine the original information entropy of the target text based on the occurrence probability of each segmented text;

[0009] Based on the text length and correction coefficient of the target text, the original information entropy is corrected to obtain the target information entropy;

[0010] The target information entropy is normalized according to a predetermined evaluation threshold.

[0011] According to another aspect of the present disclosure, there is also provided a device for evaluating the amount of information in a text, comprising:

[0012] A text acquisition module is used to acquire the target text to be evaluated;

[0013] An information entropy determination module, used to determine the occurrence probability of each segmented text in the target text, and determine the original information entropy of the target text based on the occurrence probability of each segmented text;

[0014] An information entropy correction module, used to correct the original information entropy based on the text length and correction coefficient of the target text to obtain a target information entropy;

[0015] The information entropy normalization module is used to normalize the target information entropy according to a predetermined evaluation threshold.

[0016] According to another aspect of the present disclosure, an electronic device is further provided, the electronic device comprising:

[0017] processor;

[0018] a memory for storing instructions executable by the processor;

[0019] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above method.

[0020] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the above method.

[0021] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0022] The technical solution provided by the embodiment of the present disclosure includes: obtaining a target text to be evaluated; determining the occurrence probability of each segmented text in the target text, and determining the original information entropy of the target text based on the occurrence probability of each segmented text; correcting the original information entropy based on the text length and correction coefficient of the target text to obtain a target information entropy; and normalizing the target information entropy according to a predetermined evaluation threshold.

[0023] In this technical solution, the original information entropy of the target text can be determined more efficiently by using the probability of occurrence of each word segmentation text; then, considering the influence of text length on the accuracy of information entropy, the original information entropy is corrected based on the text length and the correction coefficient, thereby improving the accuracy of the target information entropy. Finally, by standardizing the target information entropy, the standardized information entropy can be easy to understand and use, and has good interpretability and verifiability. Therefore, the information volume evaluation method of the text provided by the present disclosure is objective and reliable, with high computational efficiency and accuracy, without the need to use complex language models, and can adapt to the evaluation needs of different types of texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0026] Figure 1 A flowchart of a method for evaluating the amount of information in a text according to an embodiment of the present disclosure;

[0027] Figure 2 A structural block diagram of a device for evaluating the amount of information in a text according to an embodiment of the present disclosure;

[0028] Figure 3 It is a schematic diagram of the structure of the electronic device described in the embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0031] In some existing text information volume evaluation methods, manual evaluation is time-consuming, labor-intensive, and highly subjective, lacking objective quantitative standards; evaluation based on language models has high computational costs and unstable results; methods based on word frequency statistics are too simple and cannot accurately reflect the actual information volume of the text. In order to improve at least one of the above problems, the embodiments of the present disclosure provide a text information volume evaluation method, device, equipment, and medium, which achieve objective, efficient, and interpretable evaluation of the text information volume through rigorous mathematical theory and statistical analysis. For ease of understanding, the embodiments of the present disclosure are described in detail below.

[0032] Figure 1The flowchart of a method for evaluating the information content of a text provided in an embodiment of the present disclosure is applicable to multiple fields that require text information content evaluation, such as answer quality evaluation of intelligent question-answering systems, information integrity evaluation of machine translation results, summary quality evaluation of automatic summary systems, output quality control of text generation models, data set screening and quality control, etc. The text information content evaluation method can be executed by a text information content evaluation device, which can be implemented in software and / or hardware. Figure 1 As shown, a method for evaluating the information content of a text may include the following steps.

[0033] Step S102: obtaining a target text to be evaluated.

[0034] The target text in this embodiment can be any text whose information quantity needs to be evaluated, such as: a question-answer text containing questions and answers, a translated text, a text in a journal or book, etc. In this embodiment, the target text to be evaluated can be obtained by various methods such as network download, local storage or manual upload.

[0035] Step S104, determining the occurrence probability of each segmented text in the target text, and determining the original information entropy of the target text based on the occurrence probability of each segmented text.

[0036] In one embodiment, the method of determining the occurrence probability of each segmented text in the target text may be as shown below.

[0037] The target language is classified by a pre-trained text classification model to obtain the target language. For example, the fasttext model is used to detect and classify the target text into languages ​​such as Chinese, English or other languages, thereby obtaining the target language corresponding to the target text.

[0038] The target text is segmented according to a segmentation algorithm that matches the target language to obtain a plurality of segmented texts; and text contents in languages ​​other than the target language are filtered out from the target text.

[0039] According to the text length of the target text and the number of occurrences of each segmented text in the target text, the occurrence probability of each segmented text in the target text is determined.

[0040] In a specific example, the probability of occurrence (expressed as p(xi)) of the i-th segmented text (expressed as xi) in the target text (expressed as T) can be determined according to the following formula:

[0041] p(xi)=ni / N (1)

[0042] Among them, ni is the number of times the segmented text xi appears in the target text T, and N is the text length of the target text T (i.e. the total number of characters).

[0043] Then, referring to the following formula (2), the original information entropy of the target text is determined based on the occurrence probability of each segmented text:

[0044] E 原始 =-Σp(xi)log(p(xi)) (2)

[0045] Among them, Information Entropy (IE) is an indicator used to measure the average amount of information contained in a text, and log() refers to the logarithm with base 2.

[0046] Step S106, based on the text length of the target text and the correction coefficient, the original information entropy is corrected to obtain the target information entropy.

[0047] This embodiment may include: determining a target relevance based on a text length of a target text and a correction coefficient; wherein the target relevance is used to represent the correlation between the original information entropy and the text length of the target text; and correcting the original information entropy based on the target relevance to obtain a target information entropy.

[0048] In practical applications, for short texts (i.e. short texts), such as question-and-answer texts, the diversity of characters, words, or sentences in the texts is limited due to the short length of the texts. In addition, the sample size of the texts is small and cannot fully reflect all the characteristics of the language, resulting in unstable estimation of information entropy. In this case, the following problems may occur:

[0049] Limited character and vocabulary diversity: A short text may contain a limited number of unique characters or words, and information entropy increases rapidly with the length of the text.

[0050] Large statistical fluctuations: Due to the small amount of data, the impact of random factors on information entropy is more significant, resulting in a strong correlation between information entropy and text length, and the Spearman correlation coefficient may be close to 1.

[0051] Unlike short texts, for longer texts, such as a whole paper, journal, or book, the characters and vocabulary contained in the text tend to be stable, and the probability distribution more accurately reflects the characteristics of the language. In this case, the following problems may exist:

[0052] Diversity saturates: As the length of the text increases, the frequency of new unique characters or words decreases, and the growth rate of information entropy slows down.

[0053] Information entropy tends to be stable: Due to the stability of probability distribution, the information entropy value also tends to a stable level, and further increase in text length has little effect on information entropy.

[0054] Weakened correlation: The correlation between information entropy and text length decreases, and the Spearman correlation coefficient and p-value test may show that there is no significant association between the two.

[0055] Based on this, in order to avoid the information entropy score being strongly affected by the text length when calculating the information entropy of a text with a shorter text length, this embodiment can use a method of calculating the information entropy twice to correct the original information entropy.

[0056] In the implementation process, when the text length of the target text is less than the preset length threshold, the target text is determined to be a short text. In this case, the original information entropy is corrected by referring to the following correction formula to obtain the target information entropy.

[0057]

[0058] Among them, E represents the target information entropy, E 原始 represents the original information entropy, L represents the text length of the target text, γ represents the correction coefficient, Indicates target relevance.

[0059] Of course, according to the above modified formula (3), it can also be understood as: Calculate the original information entropy E 原始 The product of the length of the text L is an intermediate value. The intermediate value is divided by the correction coefficient γ to obtain the corrected target information entropy E. The above correction coefficient γ is determined by analyzing a large amount of internal enterprise data, and the specific determination process will not be described in detail.

[0060] Step S108, normalizing the target information entropy according to a predetermined evaluation threshold.

[0061] In this embodiment, the process of determining the evaluation threshold is first described as shown below.

[0062] (1) Obtaining sample information entropy of multiple text samples. The text samples are texts of the same type as the target text, such as question-answer texts or book texts. Methods for determining the sample information entropy of the text samples may include:

[0063] Determine the text y for each word segmentation in the text sample i The probability of occurrence p(y i ), and refer to the following formula based on the probability of occurrence of each word segmentation text p(y i )Determine the sample information entropy H(Y) of the text sample:

[0064]

[0065] (2) Determine the distribution characteristics of the sample information entropy; wherein the distribution characteristics include: the skewness, kurtosis and normality test results of the sample information entropy.

[0066] As an example, the analysis of the distribution characteristics of information entropy of multiple samples may mainly include the following items:

[0067] Calculate the skewness and kurtosis of multiple sample information entropies, and understand the symmetry and tail characteristics of the distribution of sample information entropy through skewness and kurtosis. Among them, skewness is used to describe the degree to which the probability distribution deviates from symmetry. Positive skewness means that the right tail of the distribution is longer, and negative skewness means that the left tail is longer. Kurtosis is used to describe the steepness of the probability distribution curve. Compared with the normal distribution, high kurtosis means that the distribution is steeper, and low kurtosis means that the distribution is flatter.

[0068] The Shapiro-Wilk normality test is performed on multiple sample information entropies, and the normality test results are used to determine whether the sample information entropy conforms to the normal distribution. The Shapiro-Wilk normality test is a statistical test method used to determine whether the sample information entropy comes from a normal distribution population by calculating the degree of fit between the sample information entropy and the normal distribution.

[0069] (3) Determine the multiple distribution types that the sample information entropy conforms to based on the distribution characteristics. Specifically, based on the above skewness, kurtosis and normality test results, determine the top N distribution types that the sample information entropy best conforms to or is closest to, such as normal distribution, lognormal distribution, skewed normal distribution and Gamma distribution.

[0070] Among them, Gamma distribution is a continuous probability distribution, which is determined by shape parameters and scale parameters, and is often used to describe waiting time or positive skewness data. Log-normal distribution: If the logarithm of a random variable follows a normal distribution, then the random variable follows a log-normal distribution, which is often used to describe data in finance, biology and other fields.

[0071] (4) The probability distribution fitting functions of various distribution types are used to fit the sample information entropy data and obtain various fitting results.

[0072] Each distribution type corresponds to its own probability distribution function. Therefore, the probability distribution fitting function of each distribution type can be used to fit the sample information entropy data. Correspondingly, each probability distribution function can obtain its own corresponding fitting result.

[0073] (5) Use PP plots and QQ plots to test each fitting result in order to determine the target probability distribution function and its corresponding target fitting result among multiple probability distribution functions.

[0074] Among them, the PP graph (Probability-Probability Plot) is a graph drawn based on the relationship between the cumulative proportion of the variable and the cumulative proportion of the specified distribution. The PP graph can be used to test whether the fitting result of the sample information entropy conforms to a certain theoretical distribution. When the fitting result conforms to a certain theoretical distribution, the points in the PP graph are approximately a straight line.

[0075] The QQ plot (Quantile-Quantile Plot) and the PP plot have the same purpose, but the test methods are different. To use the QQ plot to test whether the fitting result of the sample information entropy conforms to a certain theoretical distribution, you can see whether the points on the QQ plot are approximately near a straight line, and the slope of the line is the standard deviation and the intercept is the mean. The QQ plot can also be used to obtain rough information on the sample skewness and kurtosis.

[0076] Taking the PP graph as an example, for the fitting result obtained using the probability distribution function of the normal distribution, the PP graph is constructed using the fitting result. If the points in the constructed PP graph are approximately in a straight line, it is determined that the fitting result conforms to the normal distribution. The tests for other fitting results are similar and will not be described in detail here.

[0077] Through the above embodiments, each fitting result is tested respectively. According to the test results of the PP graph and the QQ graph, if it is determined that the fitting result is consistent with the corresponding theoretical distribution, the fitting result is determined as the target fitting result, and the probability distribution function used by the target fitting result is determined as the target probability distribution function.

[0078] (6) Determine a plurality of candidate thresholds according to the target fitting results and different statistical algorithms; wherein the statistical algorithms include: interquartile range, 3sigma model and confidence interval.

[0079] In this embodiment, examples of determining various candidate thresholds according to target fitting results and different statistical algorithms are shown below.

[0080] Example 1. Determine the first candidate threshold value based on the target fitting result and the interquartile range (IQR). Specifically, the interquartile range is used to determine the difference between the third quartile and the first quartile, or in other words, the interquartile range is equal to the upper quartile Q3 minus the lower quartile Q1. After arranging the sample information entropy in the target fitting result from small to large, 25% of the sample information entropy is less than or equal to the lower quartile Q1. Based on this, the first candidate threshold value is determined by referring to the following formula:

[0081] T1=Q1-1.5×IQR (5)

[0082] Among them, IQR is the interquartile range, the difference between the upper quartile Q3 and the lower quartile Q1: IQR = Q3-Q1.

[0083] Example 2. Refer to the following formula to determine the second candidate threshold value based on the target fitting result and the 3sigma model:

[0084] T2=μ-3σ (6)

[0085] Among them, μ is the mean and σ is the standard deviation. The mean μ is the sum of all sample information entropies divided by the number of data, which represents the average level of sample information entropy; the standard deviation σ represents the degree of dispersion of sample information entropy, which is calculated as the square root of the mean of the sum of the squares of the differences between the information entropy of each sample and the mean.

[0086] Example 3. Determine the third candidate threshold and the fourth candidate threshold according to the target fitting result and the confidence interval. Specifically, the third candidate threshold is determined by calculating the lower bound of the confidence interval of the first confidence interval (Confidence Interval, CI) (for example, 95%), and the fourth candidate threshold is determined by calculating the lower bound of the confidence interval of the second confidence interval (for example, 99%). The confidence interval is used to estimate the interval range of the overall sample information entropy.

[0087] In one possible manner, when determining the mean M and standard deviation ST of the sample information entropy, the lower bound of the confidence interval is: a=Mn*ST. When obtaining a 95% confidence interval, n=1.96, based on which the third candidate threshold is calculated; when obtaining a 99% confidence interval, n=2.576, based on which the fourth candidate threshold is calculated.

[0088] In another possible manner, when obtaining the estimated value distribution by using the Monte Carlo method, all estimated value samples are first sorted, and the confidence interval lower limit: a is the lower% percentile value after sorting. When obtaining the 95% confidence interval, lower = 2.5, based on which the third candidate threshold is calculated; when obtaining the 99% confidence interval, lower = 0.5, based on which the fourth candidate threshold is calculated.

[0089] When the data volume of sample information entropy is large enough, the thresholds obtained by the above two possible methods are basically equal.

[0090] Based on the above embodiments, four candidate thresholds can be obtained.

[0091] (7) An evaluation threshold value serving as an outlier judgment standard is determined among the candidate threshold values.

[0092] This embodiment can manually review and verify the above multiple candidate thresholds, and select the most suitable candidate threshold as the evaluation threshold of the abnormal value judgment standard. Alternatively, this embodiment can also determine whether there is a threshold that is consistent with the above multiple candidate thresholds, and if so, the candidate threshold that is consistent with the comparison is used as the evaluation threshold that is most suitable for the abnormal value judgment standard.

[0093] In this embodiment, the target information entropy can be normalized according to the evaluation threshold determined in the above embodiment. When normalization is implemented, different methods are used for normalization in the two cases where the target information entropy is less than the evaluation threshold and not less than the evaluation threshold.

[0094] In one embodiment, when the target information entropy is less than a predetermined evaluation threshold, a preset first algorithm is used to normalize the target information entropy according to the evaluation threshold and a preset minimum information entropy to obtain a first normalized information entropy.

[0095] In this embodiment, the determined evaluation threshold can be fixedly mapped to 60 points, and a piecewise linear transformation function (i.e., the first algorithm) is used for normalization to map the target information entropy to a score interval of [0,100]. Assume that after the above evaluation threshold is determined, the evaluation threshold is represented by E_th, where E is the actual score of the target information entropy of the target text, and S is the percentage score after the target information entropy is mapped, i.e., the first standardized information entropy. In this case, the target information entropy is standardized using the first algorithm shown in the following formula (7) to obtain the first standardized information entropy:

[0096] S1=60 / (E_th–E_min)*(E–E_min) (7)

[0097] Wherein, S1 represents the first standardized information entropy, E_th represents the evaluation threshold, E_min represents the preset minimum information entropy, usually E_min=0, and E represents the target information entropy.

[0098] In another manner, when the target information entropy is not less than a predetermined evaluation threshold, a preset second algorithm is used to normalize the target information entropy according to the evaluation threshold and a preset maximum information entropy to obtain a second normalized information entropy.

[0099] In order to ensure that the remaining 40 percentage units are allocated to the remaining entropy value range, this embodiment uses the second algorithm shown in the following formula (8) to normalize the target information entropy to obtain the second normalized information entropy:

[0100] S2=60+(E–E_th) / (E_max–E_th)*40 (8)

[0101] Among them, S2 represents the second standardized information entropy, E_th represents the evaluation threshold, E_max represents the preset maximum information entropy, which is the maximum information entropy in a set of information entropies, and E represents the target information entropy.

[0102] In summary, the information content evaluation method of the text provided by the embodiment of the present disclosure includes: obtaining the target text to be evaluated; determining the occurrence probability of each segmented text in the target text, and determining the original information entropy of the target text based on the occurrence probability of each segmented text; based on the text length and correction coefficient of the target text, correcting the original information entropy to obtain the target information entropy; and standardizing the target information entropy according to a predetermined evaluation threshold.

[0103] In this technical solution, the original information entropy of the target text can be determined more efficiently by using the probability of occurrence of each word segmentation text; then, considering the influence of text length on the accuracy of information entropy, the original information entropy is corrected based on the text length and the correction coefficient, thereby improving the accuracy of the target information entropy. Finally, by standardizing the target information entropy, the standardized information entropy can be easy to understand and use, and has good interpretability and verifiability. Therefore, the information volume evaluation method of the text provided by the present disclosure is objective and reliable, with high computational efficiency and accuracy, without the need to use complex language models, and can adapt to the evaluation needs of different types of texts.

[0104] Figure 2 This is a structural block diagram of a text information volume evaluation device provided in an embodiment of the present disclosure, and the device can be used to implement the text information volume evaluation method provided in the above embodiment. Figure 2 As shown, a device for evaluating the information content of a text may include the following modules.

[0105] A text acquisition module 210 is used to acquire a target text to be evaluated;

[0106] An information entropy determination module 220 is used to determine the occurrence probability of each segmented text in the target text, and determine the original information entropy of the target text based on the occurrence probability of each segmented text;

[0107] An information entropy correction module 230 is used to correct the original information entropy based on the text length of the target text and the correction coefficient to obtain a target information entropy;

[0108] The information entropy normalization module 240 is used to normalize the target information entropy according to a predetermined evaluation threshold.

[0109] The implementation principle and technical effects of the device provided in this embodiment are the same as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0110] Figure 3 The structure diagram of an electronic device provided by the embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the electronic device 300 includes one or more processors 301 and a memory 302 .

[0111] The processor 301 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 300 to perform desired functions.

[0112] The memory 302 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 301 may run the program instructions to implement the text information quantity evaluation method of the embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0113] In one example, the electronic device 300 may further include: an input device 303 and an output device 304 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0114] In addition, the input device 303 may also include, for example, a keyboard, a mouse, and the like.

[0115] The output device 304 can output various information to the outside, including the determined distance information, direction information, etc. The output device 304 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0116] Of course, to simplify, Figure 3 Only some of the components related to the present disclosure in the electronic device 300 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 300 may also include any other appropriate components.

[0117] Furthermore, this embodiment also provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned text information volume evaluation method.

[0118] The embodiments of the present disclosure provide a method, device, electronic device, and medium for evaluating the amount of information in a text, including a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. For specific implementation, please refer to the method embodiments, which will not be described in detail here.

[0119] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0120] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating the information content of a text, characterized in that: include: Get the target text to be evaluated; Determine the occurrence probability of each segmented text in the target text, and determine the original information entropy of the target text based on the occurrence probability of each segmented text; Based on the text length and correction coefficient of the target text, the original information entropy is corrected to obtain the target information entropy; The target information entropy is normalized according to a predetermined evaluation threshold.

2. The method according to claim 1, characterized in that The method of correcting the original information entropy based on the text length and the correction coefficient of the target text to obtain the target information entropy includes: Determining a target correlation according to the text length of the target text and the correction coefficient; wherein the target correlation is used to represent the correlation between the original information entropy and the text length of the target text; The original information entropy is modified according to the target correlation to obtain the target information entropy.

3. The method according to claim 1, characterized in that The process of determining the evaluation threshold includes: Obtain sample information entropy of multiple text samples; Determine the distribution characteristics of the sample information entropy; wherein the distribution characteristics include: skewness, kurtosis and normality test results of the sample information entropy; Determine, according to the distribution characteristics, a plurality of distribution types to which the sample information entropy conforms; Using probability distribution fitting functions of multiple distribution types to perform data fitting on the sample information entropy, respectively, to obtain multiple fitting results; Using the PP graph and the QQ graph to test each of the fitting results, so as to determine a target probability distribution function and its corresponding target fitting result among the multiple probability distribution functions; Determining a plurality of candidate thresholds according to the target fitting results and different statistical algorithms; wherein the statistical algorithms include: interquartile range, 3sigma model and confidence interval; The evaluation threshold value as an abnormal value judgment standard is determined among the candidate threshold values.

4. The method according to claim 1, characterized in that: The step of normalizing the target information entropy according to a predetermined evaluation threshold value includes: When the target information entropy is less than a predetermined evaluation threshold, a preset first algorithm is used to normalize the target information entropy according to the evaluation threshold and a preset minimum information entropy to obtain a first normalized information entropy.

5. The method according to claim 1, characterized in that The step of normalizing the target information entropy according to a predetermined evaluation threshold value includes: When the target information entropy is not less than a predetermined evaluation threshold, a preset second algorithm is used to normalize the target information entropy according to the evaluation threshold and a preset maximum information entropy to obtain a second normalized information entropy.

6. The method according to claim 4, characterized in that The first algorithm comprises: S1=60 / (E_th–E_min)*(E–E_min) Among them, S1 represents the first standardized information entropy, E_th represents the evaluation threshold, E_min represents the preset minimum information entropy, and E represents the target information entropy.

7. The method according to claim 5, characterized in that The second algorithm comprises: S2=60+(E–E_th) / (E_max–E_th)*40 Among them, S2 represents the second standardized information entropy, E_th represents the evaluation threshold, E_max represents the preset maximum information entropy, and E represents the target information entropy.

8. The method according to claim 1, characterized in that The determining the occurrence probability of each segmented text in the target text includes: Classifying the language of the target text by using a pre-trained text classification model to obtain the target language; Segmenting the target text according to a segmentation algorithm matching the target language to obtain a plurality of segmented texts; The occurrence probability of each of the segmented texts in the target text is determined according to the text length of the target text and the number of occurrences of each of the segmented texts in the target text.

9. A device for evaluating the information content of a text, characterized in that: include: A text acquisition module is used to acquire the target text to be evaluated; An information entropy determination module, used to determine the occurrence probability of each segmented text in the target text, and determine the original information entropy of the target text based on the occurrence probability of each segmented text; An information entropy correction module, used to correct the original information entropy based on the text length and correction coefficient of the target text to obtain a target information entropy; The information entropy normalization module is used to normalize the target information entropy according to a predetermined evaluation threshold.

10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device implements the method according to any one of claims 1 to 8.

Images