A method for evaluating the quality of large model input text in vertical fields

Abstract

The present invention discloses a method for evaluating the quality of large-model input text in a vertical field, which is applied to the field of large-model generative AI technology. The method comprises the following steps: obtaining the content of the input text; determining the prompt word score D1 of the input text; determining the text structure score D2 of the input text; determining the content relevance score D3 of the input text; and generating the final input text quality score Q by calculating the prompt word score D1, the text structure score D2, and the content relevance score D3 of the input text. The present invention proposes to improve the quality of the generation effect of the generative AI in the vertical field by analyzing the text quality of the input text. The higher the quality of the input text, the higher the generation effect.

Description

Technical Field

[0001] The present invention relates to the field of large-model generative AI technology, and more specifically to a large-model input text quality evaluation method in a vertical field. Background Art

[0002] With the rise of generative AI based on the Transformer architecture, people are increasingly interested in research on generative large language models, mainly focusing on natural language processing applications. At the same time, generative large language models have also shown great potential in vertical fields ranging from education and history to mathematics, medicine, information processing, and network security.

[0003] Nowadays, countless research focuses have turned to the study of generative AI. In the research of AI application fields, it is found that the quality of text generated by generative AI has become the core focus. However, the research on the quality of input text is still a neglected focus.

[0004] Therefore, it is an urgent problem for those skilled in the art to propose a large-model input text quality evaluation method in a vertical field to solve the difficulties existing in the prior art. Summary of the invention

[0005] In view of this, the present invention provides a method for evaluating the quality of input text of a large model in a vertical field, which realizes the study of the quality of the input text and enables the large model to generate better answers while obtaining higher quality scores.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A method for evaluating the quality of large model input text in a vertical field comprises the following steps:

[0008] Get the input text content;

[0009] Determine the prompt word score D1 of the input text;

[0010] Determine a text structure score D2 of the input text;

[0011] Determine a content relevance score D3 of the input text;

[0012] The final input text quality score Q is generated by calculating the prompt word score D1, text structure score D2, and content relevance score D3 of the input text.

[0013] Optionally, the method described above may include determining the prompt word score D1 of the input text as follows: the prompt word score D1 is used to analyze whether the input text contains the prompt word. If so, D1=1; if not, D1=0.

[0014] In the above method, optionally, the text structure score D2 includes: text vocabulary richness s1, text complexity s2, and the number of logical connectives s3.

[0015] In the above method, optionally, the text vocabulary richness s1 is obtained by dividing the text into words to calculate the number of occurrences of each different word n1, and comparing the scores of the number of different words and the total number of words n to obtain s1;

[0016] The text complexity s2 is obtained by the ratio of the average sentence length l1 of the input text and the maximum Chinese sentence length l;

[0017] The number of logical connectives s3 measures the degree of use of logical connectives in sentences and designs a dictionary of multiple types of connectives including causal relationships, parallel relationships, transitional relationships, comparative relationships, progressive relationships, and conclusions.

[0018] In the above method, optionally, the formula for calculating the text structure score D2 of the input text is:

[0019] D2=1 / 3(s1+s2+s3)

[0020] Among them, s1 is the vocabulary richness of the text, s2 is the complexity of the text, and s3 is the number of logical connectives.

[0021] In the above method, optionally, the specific process of determining the content relevance score D3 of the input text is:

[0022] It includes professional vocabulary module, genetic algorithm module, word vector comparison module and acquisition module; among them,

[0023] Professional vocabulary module: After obtaining the input text, it performs text preprocessing, removes prompt words and stop words, and performs text segmentation. The segmentation results are matched with the words in the vectorized vocabulary. The number of successfully matched words is recorded as M1, and the corresponding word vector representation x is extracted. i ,i=1,2,3M1, at the same time, record the number of words that failed to match successfully, recorded as M2;

[0024] Genetic algorithm module: Genetic algorithm is used to generate the initial population of vectorized vocabulary and filter out irrelevant vectors x e ;

[0025] Word vector comparison module: normalize the obtained word vector to obtain the final input text vector X input ;

[0026] Get the module: Use X input The cosine similarity is calculated with each vector in the vectorized vocabulary, and the arithmetic mean of the cosine similarities is taken to obtain the content relevance score D3.

[0027] The above method, optionally, the final input text vector X input The calculation formula is:

[0028]

[0029] Among them, M1 is the number of words that match successfully, M2 is the number of words that do not match successfully, and x i is the word vector representation, x e is an independent vector.

[0030] In the above method, optionally, the calculation formula of the content relevance score D3 is:

[0031]

[0032] Where n is the number of vectors in the vertical vocabulary, X input is the final input text vector, x i is the word vector representation.

[0033] In the above method, optionally, the calculation formula of the input text quality score Q is:

[0034]

[0035] Among them, D1 is the prompt word score, D2 is the text structure score, and D3 is the content relevance score.

[0036] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method for evaluating the quality of large-model input text in a vertical field, and its beneficial effects are:

[0037] 1) The present invention proposes to improve the quality of the generation effect of vertical field generative AI by analyzing the text quality of the input text. The higher the quality of the input text, the higher the generation effect;

[0038] 2) The present invention solves the problem of quantifying the quality of generative AI input for the same type of questions. By quantifying the quality of input text, different users can optimize their own input problems to improve the efficiency of generative AI in vertical fields.

[0039] 3) The present invention realizes the quality analysis of generative AI input text in vertical fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0041] Figure 1 A flowchart of a method for evaluating the quality of large-scale model input text in a vertical field provided by the present invention;

[0042] Figure 2 A schematic diagram of the prompt word score provided by the present invention;

[0043] Figure 3 A schematic diagram of the text structure score provided by the present invention;

[0044] Figure 4 A schematic diagram of content relevance scores provided by the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] See also Figure 1 As shown, the present invention discloses a method for evaluating the quality of large-model input text in a vertical field, comprising the following steps:

[0047] Get the input text content;

[0048] Determine the prompt word score D1 of the input text;

[0049] Determine a text structure score D2 of the input text;

[0050] Determine a content relevance score D3 of the input text;

[0051] The final input text quality score Q is generated by calculating the prompt word score D1, text structure score D2, and content relevance score D3 of the input text.

[0052] For further information, see Figure 2 As shown, the specific content of determining the prompt word score D1 of the input text is: the prompt word score D1 is used to analyze whether the input text contains the prompt word, if yes, D1=1; if no, D1=0.

[0053] Furthermore, the text structure score D2 includes: text vocabulary richness s1, text complexity s2, and the number of logical connectives s3.

[0054] Furthermore, the lexical richness s1 of the text is obtained by dividing the text into words to calculate the number of occurrences of each different word n1, and comparing the scores of the number of different words and the total number of words n to obtain s1;

[0055] The text complexity s2 is obtained by the ratio of the average sentence length l1 of the input text and the maximum Chinese sentence length l;

[0056] The number of logical connectives s3 measures the degree of use of logical connectives in sentences and designs a dictionary of multiple types of connectives including causal relationships, parallel relationships, transitional relationships, comparative relationships, progressive relationships, and conclusions.

[0057] Specifically, the input text is cut into individual sentences by punctuation marks, thereby obtaining the total number of sentences, and further determining the maximum number z of logical connectives in each sentence.

[0058] Then all logical connectives appearing in the input text are checked, which are derived from a predefined connective dictionary. Through this process, the number of logical connectives z1 in each sentence can be calculated.

[0059] For further information, see Figure 3 As shown, the formula for calculating the text structure score D2 of the input text is:

[0060] D2=1 / 3(s1+s2+s3)

[0061] Among them, s1 is the vocabulary richness of the text, s2 is the complexity of the text, and s3 is the number of logical connectives.

[0062] For further information, see Figure 4 As shown, the specific process of determining the content relevance score D3 of the input text is:

[0063] It includes professional vocabulary module, genetic algorithm module, word vector comparison module and acquisition module; among them,

[0064] Professional vocabulary module: After obtaining the input text, it performs text preprocessing, removes prompt words and stop words, and performs text segmentation. The segmentation results are matched with the words in the vectorized vocabulary. The number of successfully matched words is recorded as M1, and the corresponding word vector representation x is extracted. i ,i=1,2,3M1, at the same time, record the number of words that failed to match successfully, recorded as M2;

[0065] Genetic algorithm module: Genetic algorithm is used to generate the initial population of vectorized vocabulary and filter out irrelevant vectors x e ;

[0066] Word vector comparison module: normalize the obtained word vector to obtain the final input text vector X input ;

[0067] Get the module: Use X input The cosine similarity is calculated with each vector in the vectorized vocabulary, and the arithmetic mean of the cosine similarities is taken to obtain the content relevance score D3.

[0068] Specifically, based on the existing vertical field related text data, the Skip-gram model of the Word2vec library is used to train the existing vertical field vectorized vocabulary V.

[0069] Based on the trained vectorized vocabulary V, the genetic algorithm is used to filter out irrelevant vectors x e , where 200 200-dimensional random vectors are randomly generated to form the initial population, and the adaptive function F(v) is designed, where

[0070]

[0071]

[0072] v|=n

[0073] Each individual in the randomly generated initial population is compared with all the word vectors in the vectorized vocabulary V, and the fitness score of each population individual is calculated using the adaptive function F(v). Subsequently, these scores are sorted, and the top 80 individuals with the highest adaptive scores are selected as the parents of the next generation population. New population individuals are generated by combining the 100-dimensional features of both parents. This process generates a total of 120 new individuals, which together with the original parent individuals constitute a new population. Five dimensions of genetic variation are introduced into the offspring population, and it is iterated 50 times on this basis, and finally the irrelevant vector x is screened out. e .

[0074] Furthermore, the final input text vector X input The calculation formula is:

[0075]

[0076] Among them, M1 is the number of words that match successfully, M2 is the number of words that do not match successfully, and x i is the word vector representation, x e is an independent vector.

[0077] Furthermore, the calculation formula of the content relevance score D3 is:

[0078]

[0079] Where n is the number of vectors in the vertical vocabulary, X input is the final input text vector, x i is the word vector representation.

[0080] Furthermore, the calculation formula of the input text quality score Q is:

[0081]

[0082] Among them, D1 is the prompt word score, D2 is the text structure score, and D3 is the content relevance score.

[0083] Specifically, when using the generative AI big model, when a question text is input, it can be scored using evaluation indicators. If the score is low, the answer can be modified until the score becomes higher. Use the higher-scoring answer to ask the generative big model questions in order to obtain a better answer.

[0084] When using generative AI big models in many fields, they will face problems such as vague professional terms or inaccurate definitions. Therefore, more logical and professional questions will help the generative AI big model generate better answers to assist users and use the generative AI big model more efficiently to complete the corresponding tasks.

[0085] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0086] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating the quality of large model input text in a vertical field, characterized in that: The following steps are involved: Get the input text content; Determine the prompt word score D1 of the input text; Determine a text structure score D2 of the input text; Determine a content relevance score D3 of the input text; The final input text quality score Q is generated by calculating the prompt word score D1, text structure score D2, and content relevance score D3 of the input text; The specific process of determining the content relevance score D3 of the input text is as follows: It includes professional vocabulary module, genetic algorithm module, word vector comparison module and acquisition module; among them, Professional vocabulary module: After obtaining the input text, it performs text preprocessing, removes prompt words and stop words, and performs text segmentation. The segmentation results are matched with the words in the vectorized vocabulary. The number of successfully matched words is recorded as M1, and the corresponding word vector representation x is extracted. i ,i=1,2,3,···,M1, at the same time, record the number of words that failed to match successfully, recorded as M2; Genetic algorithm module: Genetic algorithm is used to generate the initial population of vectorized vocabulary and filter out irrelevant vectors x e ; Word vector comparison module: normalize the obtained word vector to obtain the final input text vector X input ; Get the module: Use X input Calculate the cosine similarity with each vector in the vectorized vocabulary, take the arithmetic mean of the cosine similarity, and thus obtain the content relevance score D3; The final input text vector X input The calculation formula is: Among them, M1 is the number of words that match successfully, M2 is the number of words that do not match successfully, and x i is the word vector representation, x e is an independent vector; The calculation formula for content relevance score D3 is: Where n is the number of vectors in the vertical vocabulary, X input is the final input text vector, x i is the word vector representation; The specific content of determining the prompt word score D1 of the input text is: The prompt word score D1 is used to analyze whether the input text contains the prompt word. If yes, D1 = 1; if no, D1 = 0; The text structure score D2 includes: text vocabulary richness s1, text complexity s2, and the number of logical connectives s3; The text vocabulary richness s1 is obtained by dividing the text into words, calculating the number of occurrences of each different word n1, and comparing the scores of the number of different words and the total number of words n to obtain s1; The text complexity s2 is obtained by the ratio of the average sentence length l1 of the input text and the maximum Chinese sentence length l; The number of logical connectives s3 measures the degree of use of logical connectives in sentences and designs a dictionary of multiple types of connectives including causal relationships, parallel relationships, transitional relationships, comparative relationships, progressive relationships, and conclusion relationships.

2. According to the method for evaluating the quality of large model input text in a vertical field in claim 1, it is characterized in that: The formula for calculating the text structure score D2 of the input text is: Among them, s1 is the vocabulary richness of the text, s2 is the complexity of the text, and s3 is the number of logical connectives.

3. According to the method for evaluating the quality of large model input text in a vertical field in claim 1, it is characterized in that: The calculation formula of input text quality score Q is: Among them, D1 is the prompt word score, D2 is the text structure score, and D3 is the content relevance score.

Images