Method, device and equipment for evaluating AI generation academic summary ability and medium

Abstract

The invention relates to an evaluation method and device for AI academic summary generation ability, equipment and a medium. The method comprises the steps of obtaining an original full text and a reference abstract of an academic literature; a to-be-tested generative AI model is adopted, and a detection abstract is generated according to the full original text; performing discussion unit segmentation and function role classification on the detection abstract to obtain detection information, and performing discussion unit segmentation and function role classification on the full original text or the reference abstract to obtain reference information; determining a matching degree score of each category dimension between the reference information and the detection information; and performing weighted summation on the matching degree score of each category dimension to obtain a total score of the evaluation capability. According to the method, the specific structure and characteristics of the academic summary are considered, the accuracy of evaluating the academic literature summary ability of the generative AI model is improved, and a practical tool is provided for evaluating the academic literature summary ability of the generative AI model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and text generation technology, and in particular to a method, device, equipment and medium for evaluating the ability of AI to generate academic summaries. Background Art

[0002] The academic summarization ability of generative AI models is reflected in the generation of abstracts, the generation of reviews of multiple articles, the generation of the central idea of ​​paragraphs or chapters, etc. At present, the evaluation of academic summarization ability in the field of generative AI model evaluation mostly focuses on general content summarization ability, and does not focus on the structure of academic literature.

[0003] The main difference between academic research and general content (newspapers, news, etc.) is that the former has an article structure consisting of research background, research purpose, research methods and results analysis. When making an academic summary, the research methods, conclusions obtained, goals achieved, problems solved, etc. should all be reflected, and the importance also varies. The existing evaluation method for academic literature summary ability is consistent with the evaluation method for general content summary ability, and it is impossible to accurately evaluate whether the academic research structure mentioned above has been summarized completely and accurately. Summary of the invention

[0004] The present application provides a method, device, equipment and medium for evaluating the ability of AI to generate academic summaries, so as to overcome or at least partially overcome the defects of the prior art.

[0005] First, a method for evaluating AI’s ability to generate academic summaries is provided, including:

[0006] Get the original full text and reference abstracts of academic literature;

[0007] Using the generative AI model to be tested, a test summary is generated based on the full text of the original text;

[0008] Performing argument unit segmentation and functional role classification on the detection summary to obtain detection information, and performing argument unit segmentation and functional role classification on the original full text or reference summary to obtain reference information, wherein the detection information includes at least one reference argument unit of at least one category, and the reference information includes at least one generated argument unit of at least one category;

[0009] A matching module, used to determine the matching degree scores of each category dimension between the reference information and the detection information;

[0010] The scoring module is used to perform weighted summation on the matching degree scores of each category dimension to obtain a total score of the assessment capability.

[0011] Optionally, in the above method, the discussion unit segmentation is performed according to the following method:

[0012] The argument unit segmenter is used to segment the content to be processed into at least one part based on the argument integrity principle, so that each segmented part forms an argument or evidence.

[0013] Optionally, in the above method, the functional role classification is performed according to the following method:

[0014] Using the functional role classifier, the parts obtained after segmenting the argument unit are divided into any of the following categories: background, purpose, method, and conclusion;

[0015] Treat each classified part as a discussion unit.

[0016] Optionally, in the above method, determining the matching degree scores of each category dimension between the reference information and the detection information includes:

[0017] For any reference argument unit in the reference information under any category dimension, respectively determine the sub-scores of the matching degree between the generated argument units under the same category as the reference argument unit;

[0018] Selecting the generated argument unit with the highest matching degree sub-score to form an argument unit pair with the reference argument unit;

[0019] The matching degree sub-scores of all argument unit pairs under the category dimension are accumulated to obtain the total matching degree score of the category dimension;

[0020] Based on the length of the reference argument unit, the average score of the matching degree of each argument unit pair is determined as the matching degree score of the category dimension.

[0021] Optionally, in the above method, determining the average score of the matching degree of each argument unit pair based on the length of the reference argument unit includes:

[0022] The following formula is used to determine the average score of the matching degree of each argument unit pair:

[0023]

[0024] Where C represents the category, MatchScore(S,X) represents the sub-score of the matching degree between the reference argument unit S and the generated argument sentence X, and ReferenceSentences C Reference Sentences represents the reference sentence of this category. c |Represents the length of the reference argument unit for this category.

[0025] Optionally, in the above method, the matching degree score can be calculated by any one of ROUGE-N calculation, semantic similarity calculation, and relational reasoning.

[0026] Optionally, in the above method, the weighted sum of the matching degree scores of the category dimensions to obtain the total score of the assessment capability includes:

[0027] The following formula is used to determine the total score of the evaluation capability of the generative AI model to be tested:

[0028]

[0029] Among them, C represents the category, wC represents the weight of the category, and FumctionClass represents the category set. Indicates the matching degree score of a category.

[0030] In a second aspect, the present application also provides a device for evaluating the ability of AI to generate academic summaries, the device comprising:

[0031] The acquisition module is used to obtain the original full text and reference abstracts of academic documents;

[0032] A generation module, used to generate a detection summary based on the full text of the original text using a generative AI model to be tested;

[0033] a segmentation and classification module, configured to segment the detection summary into argument units and classify the functional roles to obtain detection information, and segment the original full text or reference summary into argument units and classify the functional roles to obtain reference information, wherein the detection information includes at least one reference argument unit of at least one category, and the reference information includes at least one generated argument unit of at least one category;

[0034] A matching module, used to determine the matching degree scores of each category dimension between the reference information and the detection information;

[0035] The scoring module is used to perform weighted summation on the matching degree scores of each category dimension to obtain a total score of the assessment capability.

[0036] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the above-mentioned method for evaluating the ability of AI to generate academic summaries are implemented.

[0037] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned method for evaluating the ability of AI to generate academic summaries are implemented.

[0038] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0039] This application provides an evaluation method for AI's ability to generate academic literature summaries. This method takes into account the importance differences of argument units with different functions in academic literature, and assigns weights based on the importance of arguments with different functions. This method can be used to improve the academic structural integrity of the score in any method, and ultimately achieve the technical effect of objectively evaluating the academic summary ability of the generative AI model. The evaluation method for AI's ability to generate academic literature summaries in this application takes into account the unique structure and characteristics of academic summaries, improves the accuracy of evaluating the academic literature summary ability of generative AI models, and provides a practical tool for evaluating the academic literature summary ability of generative AI models. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 It is a flowchart of a method for evaluating the ability of AI to generate academic summaries in one embodiment of the present application;

[0042] Figure 2 It is a structural schematic diagram of an evaluation device for the ability of AI to generate academic summaries in one embodiment of the present application;

[0043] Figure 3 is a schematic diagram of the structure of a computer device in one embodiment of the present application;

[0044] Figure 4 It is a structural diagram of a computer device in another embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] With the development of artificial intelligence technology, large-scale language models (LLMs) have become increasingly mature in the field of natural language processing, and have shown great potential in text generation, translation, question answering, etc. At present, various open-source and closed-source generative AI models are in a stage of contention, and a unified capability evaluation standard is particularly important. At present, the automatic evaluation of generative AI models mainly focuses on the model's language, understanding, reasoning, code and other capabilities, while the automatic evaluation method for evaluating academic summary capabilities is still imperfect.

[0047] At present, the method of automatic evaluation of academic literature summaries is consistent with that of ordinary content summaries. The widely used evaluation method is to use the ROUGE indicator to measure the consistency between the summary content generated by the model to be tested and the standard answer. The specific process is as follows:

[0048] First, develop a standard answer: use the article's abstract, or have it written by a human expert.

[0049] Then, the ROUGE score of the summary content generated by the tested model and the standard answer is calculated.

[0050] Among them, a brief introduction and calculation process of ROUGE score is as follows:

[0051] ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a commonly used metric for evaluating the quality of text summarization. It measures the similarity between the generated summary and the reference summary by comparing the degree of overlap between them, such as ROUGE-N (considering n-gram overlap), ROUGE-L (considering the longest common subsequence), and ROUGE-W (considering word-level overlap).

[0052] Taking the most widely used ROUGE-N as an example, its definition is: calculate the overlap of n-grams (n-tuples) between the generated summary and the reference summary.

[0053] The formula is shown in formula (1):

[0054]

[0055] Among them, Count match (gram n ) indicates the number of matches between the generated summary and the reference summary n-gram, Count(gram n ) represents the total number of n-grams in the reference summary.

[0056] In addition, Chinese patents CN112528598B and CN110427483B have also studied the evaluation methods of the model's ability to generate summaries. However, these two methods both solve the problem of calculating Rouge scores relying on reference summaries, focusing on the matching of information theory dimensions and the degree of semantic similarity. They are mainly suitable for text summarization tasks in general fields, such as news and newspaper summaries, but the application effect in academic papers is not very ideal.

[0057] In response to the above situation, this application provides an evaluation method for AI's ability to generate academic summaries, which mainly focuses on the evaluation of automatically generated summaries in the academic structure dimension. It uses the functional role classification of argument units and weights the scores calculated using any scoring criteria according to the category of the summary sentences, thereby obtaining a score that can reflect AI's ability to generate summaries in the academic dimension.

[0058] Figure 1 is a flow chart of a method for evaluating the ability of AI to generate academic summaries in one embodiment of the present application, from Figure 1 It can be seen that this embodiment includes steps S110 to S150:

[0059] Step S110, obtaining the original full text and reference abstract of the academic document.

[0060] First, obtain the original full text and reference abstract of the academic document, where the reference abstract can be considered the "standard answer" of the abstract. The reference abstract can come from the "Abstract" section of the original full text, or it can be written by experts in the relevant field, which is not limited in this application.

[0061] An academic document usually includes but is not limited to four main parts, namely research background, research purpose, research methods, and research conclusions.

[0062] Step S120, using the generative AI model to be tested to generate a detection summary according to the full text of the original text.

[0063] Select or build one or more generative AI models to be tested to generate summaries for a given academic document, which is referred to as "test summaries". Any model can be processed in the following manner, so the following description will take a generative AI model to be tested as an example.

[0064] The generative AI model to be tested in this application can be any text generation model in the prior art, and this application does not limit this.

[0065] Step S130, performing argument unit segmentation and functional role classification on the detection summary to obtain detection information, and performing argument unit segmentation and functional role classification on the original full text or reference summary to obtain reference information, wherein the detection information includes at least one reference argument unit of at least one category, and the reference information includes at least one generated argument unit of at least one category.

[0066] The reference abstract (or the full text of the original article) and the detection abstract are processed separately. The detection abstract is first used as an example to illustrate. The detection abstract is segmented into discussion units and classified into functional roles in turn to obtain detection information, which includes at least one reference discussion unit.

[0067] Specifically, argument unit segmentation can be performed according to the following method: using an argument unit segmentor, based on the argument integrity principle, the content to be processed is segmented into at least one part, so that each segmented part forms an argument or evidence.

[0068] The argument unit segmenter is used to segment the input text fragment (detection information) according to the argument completeness to obtain one or more parts, usually multiple parts in actual scenarios, and then check whether the segmented parts are fluent and semantically complete. If the sentences are fluent and the semantics are complete, the semantic integrity ensures that the segmented unit is a relatively complete argument or evidence.

[0069] The specific implementation method of the unit segmenter can be implemented by using any semantic segmenter in the prior art.

[0070] Then, the parts obtained after the segmentation are classified. Specifically, in some embodiments, a functional role classifier is used to classify the parts obtained after the segmentation of the argument unit into any one of the background, purpose, method, and conclusion; each of the classified parts is respectively regarded as an argument unit. For the detection information, the argument unit obtained in this application is recorded as a generated argument unit.

[0071] For the "standard answer", it can be processed using the reference summary, or it can directly use the full text of the original text to enter this step. Specifically, the argument unit segmentation and functional role classification are performed in sequence to obtain reference information, which includes at least one reference argument unit. For the reference information, the argument unit obtained in this application is recorded as the reference argument unit.

[0072] Step S140, determining the matching degree scores of each category dimension between the reference information and the detection information.

[0073] The category dimension refers to the four categories of background, purpose, method, and conclusion mentioned above. There is at least one reference discussion unit in the reference information, and a reference discussion unit is classified as any one of the background, purpose, method, and conclusion, which is the output result of the functional role classifier; similarly, there is at least one reference discussion unit in the detection information, and a reference discussion unit is classified as any one of the background, purpose, method, and conclusion, which is also the output result of the functional role classifier.

[0074] Then, the matching degree between the reference information and the detection information in each category dimension is calculated, which can be reflected by a matching degree score. The matching degree score can be any scoring method in the prior art that can reflect the matching degree between the information, and ROUGE-N calculation, semantic similarity calculation, and relational reasoning are recommended but not limited to.

[0075] Specifically, in some embodiments, determining the matching degree scores of each category dimension between the reference information and the detection information includes: for any reference argument unit in the reference information under any category dimension, respectively determining the matching degree sub-scores between each generated argument unit under the same category as the reference argument unit; selecting the generated argument unit with the highest matching degree sub-score to form an argument unit pair with the reference argument unit; accumulating the matching degree sub-scores of all argument unit pairs under the category dimension to obtain a total matching degree score for the category dimension; and determining the average matching degree score of each argument unit pair based on the length of the reference argument unit as the matching degree score for the category dimension.

[0076] That is to say, for a reference argument unit under a category in the reference information, first find the most matching generated argument unit, and the two form an argument unit pair, with a matching degree sub-score between them. Under a category, there may be one or more argument unit pairs; the matching degree sub-scores of these argument unit pairs are accumulated and summed to obtain the total matching degree score of the category dimension. Finally, the total matching degree score is divided by the length of the reference argument unit to obtain the matching degree score of the category dimension. The calculation method of the matching degree score of a category dimension can be expressed as follows:

[0077]

[0078] Where C represents the category, MatchScore(S,X) represents the sub-score of the matching degree between the reference argument unit S and the generated argument sentence X, and ReferenceSentences C Reference Sentences represents the reference sentence of this category. C|Represents the length of the reference discussion unit of this category, which mainly reflects the number of words.

[0079] Step S150, performing weighted summation on the matching degree scores of each category dimension to obtain a total assessment capability score.

[0080] The weight value of each category is assigned weight w according to the importance of the category C , this weight is set in advance. Specifically, in some embodiments, the weighted sum of the matching degree scores of each category dimension is performed to obtain the total score of the assessment ability, including:

[0081] The following formula is used to determine the total score of the evaluation capability of the generative AI model to be tested:

[0082]

[0083] Among them, C represents the category, w C Represents the weight of the category, FunctionClass represents the category set, Indicates the matching degree score of a category obtained above.

[0084] Finally, the total evaluation ability score of the generative AI model to be tested is obtained. The total evaluation ability score can reflect the strength of the academic summary ability of the generative AI model to be tested. The higher the total evaluation ability score, the stronger the academic summary ability of the generative AI model to be tested.

[0085] This application considers that the argument unit in academic writing scenarios has the functions of research background, research purpose, research method, and research conclusion, and the importance of different functional roles is different. This can avoid the situation where when important arguments (such as research methods) account for a small proportion of the length, the generated summary that summarizes important content but does not fully summarize non-important structures will not get the due score. Assume that the existing reference summary and two generated summaries have the following structure:

[0086] Reference Abstract: Research background 1. Research background 2. Research purpose 1. Research background 3. Research methods 1. Research conclusions 1.

[0087] Generate abstract 1: Research background 1. Research background 1. Research background 2. Research background 3. Research background 3. Research conclusion 1.

[0088] Generate summary2: Research purpose1. Research methods1. Research conclusion1.

[0089] If sequence similarity or semantic similarity is used for calculation, the generated abstract 1 has a higher similarity score with the reference abstract and is considered to be a better abstract. In non-academic writing scenarios, it is reasonable that the generated abstract 1 is considered to be a better abstract because it is more similar to the reference abstract. In academic writing scenarios, since the research purpose and research methods are important components of academic research, the generated abstract 2 is considered to be a better abstract. Using this application for weighting based on the calculated similarity can reflect the importance of academic writing structure.

[0090] Depend on Figure 1 It can be seen from the method shown that this application provides an evaluation method for AI's ability to generate academic literature summaries. This method takes into account the importance differences of discussion units with different functions in academic literature, and assigns weights based on the importance of discussions with different functions. This method can be used in the score calculation of any method to improve the academic structural integrity of the score, and ultimately achieve the technical effect of objectively evaluating the academic summary ability of the generative AI model. The evaluation method for AI's ability to generate academic literature summaries in this application takes into account the unique structure and characteristics of academic summaries, improves the accuracy of evaluating the academic literature summary ability of generative AI models, and provides a practical tool for evaluating the academic literature summary ability of generative AI models.

[0091] In one embodiment, a device for evaluating the ability of AI to generate academic summaries is provided, and the device for evaluating the ability of AI to generate academic summaries corresponds one-to-one to the method for evaluating the ability of AI to generate academic summaries in the above embodiment. Figure 2 As shown, the AI-generated academic summary capability evaluation device 200 includes:

[0092] The acquisition module 210 is used to acquire the original full text and reference abstract of the academic document.

[0093] The generation module 220 is used to generate a detection summary based on the full text of the original text using the generative AI model to be tested.

[0094] The segmentation and classification module 230 is used to perform argument unit segmentation and functional role classification on the detection summary to obtain detection information, and to perform argument unit segmentation and functional role classification on the original full text or reference summary to obtain reference information, wherein the detection information includes at least one reference argument unit of at least one category, and the reference information includes at least one generated argument unit of at least one category.

[0095] The matching module 240 is used to determine the matching degree scores of each category dimension between the reference information and the detection information.

[0096] The scoring module 250 is used to perform weighted summation on the matching degree scores of each category dimension to obtain a total assessment capability score.

[0097] In one embodiment, in the above-mentioned device, the segmentation and classification module 230 is used to use a discussion unit segmenter to segment the content to be processed into at least one part based on the principle of discussion integrity, so that each segmented part forms an argument or evidence.

[0098] In one embodiment, in the above-mentioned device, the segmentation and classification module 230 is used to use a functional role classifier to divide the parts obtained after segmenting the argument unit into any one of the following categories: background, purpose, method, and conclusion; and each classified part is respectively regarded as a argument unit.

[0099] In one embodiment, in the above-mentioned device, the matching module 240 is used to determine the matching degree sub-scores between each of the generated argument units in the same category as any reference argument unit in the reference information under any category dimension; select the generated argument unit with the highest matching degree sub-score to form an argument unit pair with the reference argument unit; add up the matching degree sub-scores of all argument unit pairs under the category dimension to obtain the total matching degree score of the category dimension; and determine the average matching degree score of each argument unit pair based on the length of the reference argument unit as the matching degree score of the category dimension.

[0100] In one embodiment, in the above device, the matching module 240 is used to determine the average score of the matching degree of each argument unit pair using the following formula:

[0101]

[0102] Where C represents the category, MatchScore(S,X) represents the sub-score of the matching degree between the reference argument unit S and the generated argument sentence X, and ReferenceSentences C Reference Sentences represents the reference sentence of this category. C |Represents the length of the reference argument unit for this category.

[0103] In one embodiment, in the above device, the matching module 240 is used to calculate the matching degree score by any one of ROUGE-N calculation, semantic similarity calculation, and relational reasoning.

[0104] In one embodiment, in the above device, the scoring module 250 is used to determine the total score of the evaluation capability of the generative AI model to be tested using the following formula:

[0105]

[0106] Among them, C represents the category, w C Represents the weight of the category, FunctionClass represents the category set, Represents the matching degree score of a category. For the specific limitations of the evaluation device for the ability of AI to generate academic summaries, please refer to the limitations of the evaluation method for the ability of AI to generate academic summaries above, which will not be repeated here. Each module in the above-mentioned evaluation device for the ability of AI to generate academic summaries can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0107] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the server side of a method for evaluating the ability of AI to generate academic summaries.

[0108] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the client side of a method for evaluating the ability of AI to generate academic summaries

[0109] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for evaluating the ability of AI to generate academic summaries of the present application are implemented.

[0110] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the following method for evaluating the ability of AI to generate academic summaries of the present application are implemented.

[0111] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0112] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0113] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0114] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for evaluating the ability of AI to generate academic summaries, characterized in that: include: Get the original full text and reference abstracts of academic literature; Using the generative AI model to be tested, a test summary is generated based on the full text of the original text; Performing argument unit segmentation and functional role classification on the detection summary to obtain detection information, and performing argument unit segmentation and functional role classification on the original full text or reference summary to obtain reference information, wherein the detection information includes at least one reference argument unit of at least one category, and the reference information includes at least one generated argument unit of at least one category; Determine the matching degree score of each category dimension between the reference information and the detection information; The matching degree scores of each category dimension are weighted and summed to obtain the total assessment ability score.

2. The method according to claim 1, characterized in that The discussion unit segmentation is performed according to the following method: The argument unit segmenter is used to segment the content to be processed into at least one part based on the argument integrity principle, so that each segmented part forms an argument or evidence.

3. The method according to claim 2, characterized in that The functional role classification is performed according to the following method: Using the functional role classifier, the parts obtained after segmenting the argument unit are divided into any of the following categories: background, purpose, method, and conclusion; Treat each classified part as a discussion unit.

4. The method according to claim 1, characterized in that The determining of the matching degree scores of each category dimension between the reference information and the detection information includes: For any reference argument unit in the reference information under any category dimension, respectively determine the sub-scores of the matching degree between the generated argument units under the same category as the reference argument unit; Selecting the generated argument unit with the highest matching degree sub-score to form an argument unit pair with the reference argument unit; The matching degree sub-scores of all argument unit pairs under the category dimension are accumulated to obtain the total matching degree score of the category dimension; Based on the length of the reference argument unit, the average score of the matching degree of each argument unit pair is determined as the matching degree score of the category dimension.

5. The method according to claim 4, characterized in that The step of determining the average score of the matching degree of each argument unit pair based on the length of the reference argument unit comprises: The following formula is used to determine the average score of the matching degree of each argument unit pair: Where C represents the category, MatchScore(S,X) represents the sub-score of the matching degree between the reference argument unit S and the generated argument sentence X, and ReferenceSentences C Reference Sentences represents the reference sentence of this category. C |Represents the length of the reference argument unit for this category.

6. The method according to claim 4, characterized in that The matching degree score may be calculated by any one of ROUGE-N calculation, semantic similarity calculation, and relational reasoning.

7. The method according to claim 1, characterized in that The weighted sum of the matching degree scores of each category dimension is performed to obtain the total score of the assessment ability, including: The following formula is used to determine the total score of the evaluation capability of the generative AI model to be tested: Among them, C represents the category, w C Represents the weight of the category, FunctionClass represents the category set, Indicates the matching degree score of a category.

8. A device for evaluating the ability of AI to generate academic summaries, characterized in that: The device comprises: The acquisition module is used to obtain the original full text and reference abstracts of academic documents; A generation module, used to generate a detection summary based on the full text of the original text using a generative AI model to be tested; a segmentation and classification module, configured to segment the detection summary into argument units and classify the functional roles to obtain detection information, and segment the original full text or reference summary into argument units and classify the functional roles to obtain reference information, wherein the detection information includes at least one reference argument unit of at least one category, and the reference information includes at least one generated argument unit of at least one category; A matching module, used to determine the matching degree scores of each category dimension between the reference information and the detection information; The scoring module is used to perform weighted summation on the matching degree scores of each category dimension to obtain a total score of the assessment capability.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for evaluating the ability of AI to generate academic summaries as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for evaluating the ability of AI to generate academic summaries as described in any one of claims 1 to 7 are implemented.

Images