Computing device, method, computer readable storage medium and computer program product for document meta-analysis

Abstract

The invention relates to a computer system utilizing a computer model. A computing device, a method, a computer-readable storage medium and a computer program product for document meta-analysis are disclosed. A computing system for file meta-analysis includes a computing resource and a plurality of agents executed by the computing resource. The multiple agents include a first agent, a second agent, a third agent and a fourth agent. The first agent is configured to perform a document evaluation. The second agent is configured to perform data extraction. The third agent is configured to perform data integration. The fourth agent is configured to perform a data check. Wherein a report of document meta-analysis is generated based on a single set of data passing the inspection. According to the computing device, the limitation that the document element analysis technology based on deep learning dispersedly processes all links of document element analysis is overcome, and the automation degree and efficiency of the whole processing flow are improved.

Description

Technical Field

[0001] The present invention generally relates to computer systems using computational models, and more particularly to computing devices, methods, computer-readable storage media, and computer program products for literature meta-analysis. Background Art

[0002] Meta-analysis is a statistical analysis of a large number of existing documents to extract the relationship between variables in the documents and discover new relationships or knowledge. Meta-analysis is widely used in scientific research fields such as medicine, agriculture, and ecological environment. Since the documents analyzed are independent, the relationship obtained through meta-analysis is more scientific and accurate, which can make up for the shortcomings of a single study and discover potential laws that cannot be found in a single study.

[0003] The implementation of literature meta-analysis usually includes the following key steps: first, clarify the research problem and set clear research objectives and hypotheses; second, perform a systematic literature search to ensure that all relevant studies are covered; then, screen out eligible literature based on preset criteria; then, extract key inputs in the study, such as sample size and effect size; on this basis, select an appropriate statistical model, usually deciding whether to use a fixed model or a random effects model based on the heterogeneity between studies; finally, conduct a sensitivity analysis to test the robustness of the results, and write a detailed interpretation and report of the results.

[0004] Commonly used literature meta-analysis methods can be divided into two categories: one is the traditional quantitative meta-analysis method, and the other is the modern data integration technology. Traditional methods (such as fixed effect models and random effect models) evaluate the overall effect by calculating the weighted average effect size of each study. Modern methods are more flexible and usually use Bayesian methods or machine learning techniques to handle more complex consistency and data structure.

[0005] The traditional literature meta-analysis method is very time-consuming and labor-intensive in the process of literature collection and screening, and requires a lot of manpower to organize and evaluate literature data one by one. For the literature that has been screened, the process of extracting and organizing data is also cumbersome, which further increases the workload. Secondly, the experimental methods and standards used in different literature may be significantly different, and simply integrating literature from different sources may lead to erroneous conclusions. The lack of unified evaluation standards makes the comparability between different research results low, thus affecting the reliability of meta-analysis. Finally, if the quality of the analyzed literature is low, then the final analysis conclusions will inevitably be affected, resulting in inaccurate or misleading results. These problems make the traditional literature meta-analysis technology need to be improved in terms of efficiency and reliability of results, and more advanced automation and standardization solutions are urgently needed to optimize this process.

[0006] In recent years, the application of deep learning technology in literature meta-analysis has gradually attracted attention, especially the use of large language models (LLM) to analyze and summarize research literature. Through a systematic review of existing literature, the corresponding analysis model is designed to improve the analysis efficiency and the accuracy of the results, thereby achieving a comprehensive evaluation of existing literature and data extraction.

[0007] Literature meta-analysis based on deep learning technology is mainly based on text analysis methods based on large language models. Its typical applications include using simple prompts and application programming interface (API) calls to perform specific links in literature meta-analysis, such as literature screening and data extraction. These methods usually focus on a certain link in literature meta-analysis, such as automatically evaluating literature quality or extracting table data through large language models, thereby improving work efficiency.

[0008] Existing literature meta-analysis methods based on deep learning technology have limitations at least on two levels.

[0009] First, each link of literature meta-analysis still requires separate manual intervention to complete separately, and there is a lack of automated design for the entire process of literature meta-analysis. For example, some studies focus on using large language models to accelerate the literature screening process to cope with the high labor costs of the literature screening process; while other studies focus on the extraction of tabular data in an attempt to simplify the data collation steps through large language models. This fragmented processing method cannot achieve complete automation and still requires manual intervention in multiple links, which limits the overall efficiency and accuracy of literature meta-analysis.

[0010] Second, there are still some limitations in each link involved in the process of literature meta-analysis.

[0011] For example, the literature meta-analysis process involves the literature evaluation link. Deep learning-based methods have limitations in literature evaluation. First, due to the context window length limit of large language models, these models often cannot process the complete content of the literature. Traditional large language model evaluation methods usually only read the abstract part of the literature. This incomplete reading leads to a one-sided evaluation of the literature, which may miss key experimental results and important conclusions. Secondly, there is a "hallucination" phenomenon in large language models, that is, the scores and evaluations output by the model may not be true. This situation may lead to a misjudgment of the quality of the literature, thereby affecting subsequent research decisions and the quality of literature screening.

[0012] For example, the literature meta-analysis process involves literature data mining. Literature data mining technology also faces multiple challenges, especially in the application of converting documents in a specific format (e.g., Portable Document Format (PDF)) to text based on Optical Character Recognition (OCR). Some OCR models cannot effectively adapt to the diverse data formats in different documents, resulting in poor recognition results. For complex tables, especially those containing multiple sub-tables, OCR technology often misrecognizes, thus affecting the accuracy of the data. In addition, when simply extracting tabular data, it is easy to cause information ambiguity. This is mainly because the information contained in the table may not be complete, especially when abbreviations are used for different variables, these abbreviations are not clearly stated, resulting in misunderstandings and incorrect analysis in the subsequent data mining process.

[0013] There is a need in the art for improved literature meta-analysis methods on at least one of the dimensions discussed above. Summary of the invention

[0014] The present invention is provided in order to further improve the literature meta-analysis technology based on deep learning.

[0015] One aspect of the present invention provides a computing device for document meta-analysis, comprising: computing resources; and multiple agents, the multiple agents being executed by the computing resources, the multiple agents comprising: a first agent configured to call a first large language model to: receive multiple documents and a first user input, the multiple documents comprising documents, the first user input comprising a user requirement; score the document corresponding to each document based on the user requirement; filter the multiple documents based on the scoring result to determine a subset of documents; a second agent configured to call a multimodal large model to: receive a subset of the documents; convert the multimodal data in each document in the subset of the documents into multiple groups of data in a tabular form, each group of data The data include entries and data for the entries; a third agent is configured to call a second large language model to: receive the multiple sets of data and a second user input, wherein the second user input includes entries that the user cares about; integrate the multiple sets of data into a single set of data in a tabular form based on the entries that the user cares about; and a fourth agent is configured to call a third large language model to: receive the multiple sets of data and the single set of data; check the single set of data against the multiple sets of data based on at least one of user interest relevance and accuracy; and iteratively provide an indication to update the single set of data based on a result of the check until the updated single set of data passes the check, wherein a report of the literature meta-analysis is generated based on the single set of data that passes the check.

[0016] In the computing device as described above, the multiple documents include a first number of documents, and the first agent is configured to call the first large language model to: determine an independent score for each document in the first number of documents based on the user requirement, and determine a relative score for each document in the first number of documents as a whole; determine a comprehensive score for each document based on the independent score and relative score of each document in the first number of documents; and filter the multiple documents based on the comprehensive score of each document in the first number of documents.

[0017] A computing device as described in any of the above items, wherein the user requirement includes at least one of topic relevance, innovation, and feasibility, and / or is characterized in that the first user inputs a reason for requiring the first large language model to output a score.

[0018] In a computing device as described in any of the above items, the first agent is configured to call the first large language model to: read a portion at a predetermined position of each document to score each document, wherein the portion at the predetermined position includes an abstract of the document and a portion of the body of the document, and the amount of text in the portion at the predetermined position meets the context window length limit of the first large language model.

[0019] In a computing device as described in any of the above items, the second agent is configured to call the multimodal big model to: generate a hierarchical data description based on the multimodal data in each document, wherein different levels of data description describe the multimodal data with different degrees of structuring; and generate the multiple sets of data based on the hierarchical data description.

[0020] In a computing device as described in any of the above items, the hierarchical data description includes a description of the overall content of the multimodal data.

[0021] In the computing device as described in any one of the above items, the hierarchical data description includes a data description in a dictionary form, and the data description in the dictionary form includes a description of row items and / or a description of column items associated with a table form.

[0022] In a computing device as described in any of the above items, the third agent is configured to call the second large language model to: analyze the semantic similarity of the descriptions of the row items and / or the descriptions of the column items to perform a consistency check on the multiple sets of data, wherein the descriptions of the row items and / or the descriptions of the column items include the full name and abbreviation of the variable, and the multiple sets of data are integrated based on the results of the consistency check.

[0023] In a computing device as described in any of the above items, the multimodal data includes a picture, and the second agent is configured to call the multimodal large model to: output the description content of the picture point by point.

[0024] In a computing device as described in any of the above items, the third agent is configured to call the second large language model to: determine whether each group of data in the multiple groups of data includes target data associated with the items that the user cares about; and based on the result of the determination, integrate the groups of data in the multiple groups of data that include the target data into the single group of data.

[0025] In a computing device as described in any of the above items, the third agent is configured to call the second large language model to: output the source of each data in the single set of data, the source being used to indicate which set of data among the multiple sets of data each data comes from and which document among the multiple documents.

[0026] In a computing device as described in any of the above items, the fourth agent is configured to call a third language model to: compare each data in the single set of data with corresponding data in the multiple sets of data; and determine the accuracy of the data based on the result of the comparison.

[0027] In a computing device as described in any of the above items, the fourth agent is configured to call the third language model to: determine an overall score based on at least one of the user interest relevance and accuracy; compare the overall score with a preset threshold; and based on the result of the comparison, iteratively provide the indication until the overall score reaches the threshold.

[0028] In a computing device as described in any of the above items, the fourth agent is configured to call the third language model to: provide the indication in response to the result of the comparison indicating that the overall score is lower than the threshold; and not provide the indication in response to the result of the comparison indicating that the overall score is greater than or equal to the threshold.

[0029] A computing device as described in any of the above items, wherein the multiple agents also include: a fifth agent, configured to: receive multiple original documents including documents; perform page parsing on each of the multiple original documents to determine multiple parsed elements; merge the multiple parsed elements to form each parsed document, wherein each parsed document formed constitutes the multiple documents.

[0030] As described in any of the above computing devices, the plurality of original documents include a diagram, and the fifth agent is configured to: call a multimodal language model to convert the diagram into corresponding parsed elements among the plurality of parsed elements.

[0031] A computing device as described in any of the above items, wherein the chart includes a statistical chart and a table, the multimodal language model includes a first multimodal language model and a second multimodal language model, and the fifth agent is configured to: call the first multimodal language model to convert the statistical chart into a corresponding parsed element among the multiple parsed elements; and call the second multimodal language model to convert the table into a corresponding parsed element among the multiple parsed elements.

[0032] In a computing device as described in any of the above items, the multiple original documents include text and formulas, and the fifth agent is configured to: use optical character recognition technology to convert the text into corresponding parsed elements among the multiple parsed elements; call the fourth language model to convert the formula into corresponding parsed elements among the multiple parsed elements.

[0033] A computing device as described in any of the above items, wherein the fifth agent is configured to: for each document to be formed: input the multiple parsed elements into the page of the document in multiple candidate layouts respectively; record the size of each parsed element and the position in the page for each candidate layout, and calculate the fitness under the current candidate layout based on the area of ​​the region occupied by the multiple parsed elements in the page and the total area of ​​the page; and merge the multiple parsed elements in the candidate layout with the largest fitness.

[0034] Another aspect of the present invention provides a method for document meta-analysis, comprising the following steps:

[0035] S1: The first agent calls the first large language model to perform a literature review, and the step S1 includes: S11: receiving a plurality of documents and a first user input, the plurality of documents including documents, and the first user input including user requirements; S12: scoring the documents corresponding to each document based on the user requirements; S13: screening the plurality of documents based on the scoring results to determine a subset of documents; S2: The second agent calls the multimodal large model to perform data extraction, and the step S2 includes: S21: receiving a subset of the documents; S22: converting the multimodal data in each document in the subset of the documents into multiple sets of data in a tabular form, Each set of data includes entries and data for the entries; S3: the third agent calls the second largest language model to perform data integration, and the step S3 includes: S31: receiving the multiple sets of data and the second user input, the second user input includes the entries that the user cares about; S32: integrating the multiple sets of data into a single set of data in a tabular form based on the entries that the user cares about; S4: the fourth agent calls the third largest language model to perform data inspection, and the step S4 includes: S41: receiving the multiple sets of data and the single set of data; S42: checking the single set of data against the multiple sets of data based on at least one of user interest relevance and accuracy; S43: based on the result of the inspection, iteratively providing an indication to update the single set of data until the updated single set of data passes the inspection, wherein a report of the literature meta-analysis is generated based on the single set of data that passes the inspection.

[0036] As described above, the step S22 comprises the following steps: S221: generating a hierarchical data description based on the multimodal data in each document, wherein each level of data description describes the multimodal data with a different degree of structural hierarchy; and S222: generating the multiple groups of data based on the hierarchical data description.

[0037] As described in any of the methods above, step S32 includes the following steps: S321: determining whether each group of data in the multiple groups of data includes target data associated with the items that the user is concerned about; and S322: integrating the groups of data in the multiple groups of data that include the target data into the single group of data.

[0038] As described in any of the methods above, step S42 includes the following steps: S421: determining an overall score based on at least one of the user interest relevance and accuracy; S422: comparing the overall score with a preset threshold; and S423: based on the result of the comparison, iteratively providing the indication until the overall score reaches the threshold.

[0039] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the method described above when executed by a processor.

[0040] Another aspect of the present invention provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the above methods are implemented.

[0041] The computing device and method according to the present invention overcome the limitation of decentralized processing of various links in document meta-analysis based on deep learning technology, and utilize multiple intelligent agents to form a workflow that can automatically perform complete document meta-analysis, thereby increasing the degree of automation of the entire process and improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Embodiments of the present invention are described with reference to the accompanying drawings.

[0043] Figure 1 is a block diagram of a computing device for literature meta-analysis according to some embodiments of the present invention.

[0044] Figure 2 A first example graph showing data processed using literature meta-analysis techniques according to some embodiments of the present invention to extract data.

[0045] Figure 3 A second example graph is shown of data processed using literature meta-analysis techniques according to some embodiments of the present invention to extract data.

[0046] Figure 4 is a flow chart of a processing pipeline for performing literature meta-analysis according to some embodiments of the present invention.

[0047] Figure 5 is a flowchart of a method for literature meta-analysis according to some embodiments of the present invention.

[0048] Figure 6 is a flow chart of a first process associated with a method for literature meta-analysis according to some embodiments of the present invention.

[0049] Figure 7 is a flow chart of a second process associated with a method for literature meta-analysis according to some embodiments of the present invention.

[0050] Figure 8 is a flow chart of a third process associated with a method for literature meta-analysis according to some embodiments of the present invention.

[0051] Fig. 9 is a flow chart of a fourth process associated with a method for literature meta-analysis according to some embodiments of the present invention.

[0052] Fig.10 is a flow chart of a fifth process associated with a method for literature meta-analysis according to some embodiments of the present invention.

[0053] Fig.11 is a flow chart of a sixth process associated with a method for literature meta-analysis according to some embodiments of the present invention.

[0054] Fig.12 is a flow chart of a seventh process associated with a method for literature meta-analysis according to some embodiments of the present invention.

[0055] Fig.13 is a flow chart of an eighth process associated with a method for literature meta-analysis according to some embodiments of the present invention.

[0056] Fig.14 is a flow chart of a ninth process associated with a method for literature meta-analysis according to some embodiments of the present invention.

[0057] Fig.15 is a flow chart of a tenth process associated with a method for literature meta-analysis according to some embodiments of the present invention.

[0058] Fig.16 is a flow chart of an eleventh process associated with a method for literature meta-analysis according to some embodiments of the present invention.

[0059] Fig.17 is a flow chart of a twelfth process associated with a method for literature meta-analysis according to some embodiments of the present invention.

[0060] Fig.18 is a block diagram of a computer-readable storage medium according to some embodiments of the present invention.

[0061] Fig.19 is a block diagram of a computer program product according to some embodiments of the invention. DETAILED DESCRIPTION

[0062] In this application, the term "agent" refers to an agent that can perceive the environment and take actions to perform specific goals. The agent mainly refers to software code. The agent can be executed by the computing resources of the computing device. The agent can call the corresponding model through the API interface and call the corresponding tools (for example, PDF reader, Python interpreter, calculator, etc.) to interact with various forms of input or implement corresponding functions.

[0063] In this application, ordinal numbers "first", "second", "third", etc. are used to distinguish different instances of objects with the same name. Ordinal numbers "first", "second", "third", etc. do not indicate the relative order of the objects indicated in time, space, sorting, or other aspects.

[0064] According to one aspect of the present invention, a computing device for literature meta-analysis is provided.

[0065] Figure 1 is a block diagram of a computing device 100 for literature meta-analysis according to some embodiments of the present invention.

[0066] The computing device 100 may be a local or remote computer, server, etc. The computing device 100 may include a computing resource 110 and a plurality of agents. The plurality of agents may be executed by the computing resource 110 to enable the computing resource to perform corresponding operations.

[0067] In some embodiments, computing resources may include a central processing unit (CPU), a graphics processing unit (GPU), and various other processing units or cores (e.g., arithmetic logic units, integer units, floating point units, tensor units, ray tracing cores, etc.), etc.

[0068] In some embodiments, each of the multiple agents can call a corresponding model through an API. As an example, the model may include a large language model, a multimodal model, a multimodal language model, etc. In some embodiments, multiple models that can be called by multiple agents can be deployed locally on the computing device 100. In some embodiments, multiple models that can be called by multiple agents can be deployed remotely from the computing device 100, for example, in the cloud. In some embodiments, some of the multiple models that can be called by multiple agents can be deployed locally on the computing device 100, and other models can be deployed remotely from the computing device 100. In some embodiments, each of the multiple agents can call various tools to interact with various forms of input or implement corresponding functions. For example, an agent can call a PDF reader to read a PDF document. An agent can call a python interpreter to interact with python code. An agent can call a calculator to implement calculation functions.

[0069] The plurality of agents may include a first agent 121. The first agent 121 may be configured to call a first large language model to perform operations related to document evaluation.

[0070] The first agent 121 may be configured to receive a plurality of documents and a first user input, wherein the plurality of documents may include literature, and the first user input may include user requirements. The first agent 121 may be configured to score the literature corresponding to each document based on the user requirements. The first agent 121 may be configured to screen a plurality of documents based on the scoring results to determine a subset of the documents. Through the literature evaluation performed by the first agent 121, the literature that meets the user requirements is screened out for subsequent further analysis, which improves the quality of the literature to be analyzed and helps to improve the reliability of the literature meta-analysis.

[0071] The plurality of agents may include a second agent 122. The second agent 122 may be configured to invoke the multimodal macro model to perform extraction of data from the document.

[0072] The second agent 122 may be configured to receive a subset of documents (determined by the first agent 121). The second agent 122 may be configured to convert the multimodal data in the subset of the document into multiple sets of data in a tabular form, wherein each set of data may include entries and data for the entries. As an example, each set of data may correspond to data of a table, which may include a header (which contains entries) and a body (which contains data). The second agent 122 may extract the multimodal data therein into multiple sets of data in a tabular form based on the documents filtered by the first agent 121, thereby providing structured data to downstream agents to facilitate further processing.

[0073] The plurality of agents may include a third agent 123. The third agent 123 may be configured to call the second largest language model to perform data integration of a plurality of sets of data in a table form.

[0074] The third agent 123 may be configured with multiple sets of data (in a tabular form converted by the second agent 122) and a second user input, wherein the second user input may include items of interest to the user. The third agent 123 may be configured to integrate multiple sets of data into a single set of data in a tabular form based on the items of interest to the user. The third agent integrates multiple sets of data in a tabular form provided by the upstream second agent 122 based on the second user input to provide structured data in a single table for downstream analysis.

[0075] The plurality of agents may include a fourth agent 124. The fourth agent 124 may be configured to invoke a fourth language model to perform data checking on a single set of data in a tabular form.

[0076] In some embodiments, the fourth agent 124 may be configured to receive multiple sets of data (in the form of a table provided by the second agent 122) and a single set of data (in the form of a table synthesized by the third agent 123). The fourth agent 124 may be configured to check the single set of data against the multiple sets of data based on at least one of user interest relevance and accuracy. As an example, the user interest relevance may indicate whether the relevant content belongs to the field, aspect, etc. of concern to the user and / or the degree of association between the relevant content and the field, aspect, etc. of concern to the user. The fourth agent 124 may be configured to iteratively provide instructions for updating the single set of data based on the results of the detection until the updated single set of data passes the inspection. Based on the single set of data that passes the inspection, a report of literature meta-analysis may be generated, for example, by a downstream agent. The fourth agent 124 checks the single set of data (synthesized data) by comparing it with multiple sets of data (the source of the synthesized data), and iteratively provides instructions for updating the single set of data based on the results of the detection. The instruction may be fed back to the third agent 123 located upstream. The third agent 123 may modify one or more data based on the received instructions to generate an updated single set of data. Through the above iteration and feedback process, the accuracy and reliability of the literature meta-analysis results were further improved.

[0077] Some embodiments integrate the first agent 121, the second agent 122, the third agent 123 and the fourth agent 124 to integrate the literature evaluation stage, the data extraction stage, the data integration stage and the data inspection stage in the literature meta-analysis process, and provide a solution for the full process of efficient and automated execution of literature meta-analysis. In the process of performing literature meta-analysis, the results generated or output by the upstream agent are helpful to improve the accuracy and reliability of the data processed or analyzed by the downstream agent, and the downstream agent can also iteratively check the synthesized data to iteratively improve the synthesized data through feedback, thereby increasing the accuracy and reliability of the literature meta-analysis.

[0078] With respect to the first agent 121, its implementation details are further described.

[0079] In some embodiments, the plurality of documents may be a plurality of documents downloaded by searching a document database. In some embodiments, the plurality of documents may be documents after document processing is performed on the plurality of documents downloaded (e.g., as described below in conjunction with the fifth agent 125).

[0080] In some embodiments, user requirements may include at least one of subject relevance, innovation, and feasibility. As an example, subject relevance may be judged based on factors such as whether the subject of the document and the subject of interest to the user belong to the same field, whether there is a direct connection, whether it is related, and how many requirements of the user are met. As an example, innovation may be judged based on whether the document proposes a new method or new discovery, the significance or value of the proposed new method or new discovery, etc. As an example, feasibility may be judged based on whether the document contains experimental or data support, the completeness, adequacy, and repeatability of the experiments and data, etc.

[0081] In the prior art, traditional corpus analysis techniques are often used to evaluate the quality of documents, which may have several limitations. On the one hand, traditional corpus analysis relies on word frequency analysis, which cannot accurately reflect the novelty and innovation of the paper, because even if there are many names of new methods and new problems in the paper, these words are likely to be over-packaging of existing names, and the document itself does not present real new technologies. In addition, word frequency analysis may be accidental for shorter documents. On the other hand, traditional corpus analysis relies on the number of document citations. The number of document citations is an important factor in measuring the influence of documents, but the number of citations is affected by the time of publication and the field in which the document is located. Therefore, the document scores obtained by analyzing citations cannot be compared horizontally. For example, a document published in a popular field in 2010 and a paper published in an unpopular field in 2020 may have much more citations than the latter, but this is not directly useful to conclude that the former is more useful than the latter.

[0082] Some embodiments can overcome the above limitations of traditional corpus analysis to a certain extent by selecting at least one of topic relevance, innovation and feasibility as a criterion for literature evaluation.

[0083] In addition, different evaluation dimensions and / or evaluation criteria for literature review can be selected according to different fields. For example, in the medical field, the rigor of experimental design may be emphasized more, and accordingly, feasibility is more important for literature evaluation and may have a higher weight in literature evaluation; alternatively, the weights of other evaluation dimensions may be reduced or ignored. For example, the social science field may pay more attention to theoretical re-experimentation, and accordingly, it may have other weights in literature evaluation; alternatively, the weights of other evaluation dimensions may be reduced or ignored.

[0084] In some embodiments, scores may be given based on two or three of topic relevance, novelty, and feasibility, respectively, and a composite score may be determined based on the individual scores.

[0085] In some embodiments, the first user input may request a reason for the first language model output score.

[0086] In some embodiments, the first user input may include a prompt word input. The following is an example of a prompt word input that may be used as the first user input:

[0087] ===========;

[0088] Prompt word example 1:

[0089] You are a professional reviewer and your area of ​​expertise is atmospheric sciences. Please review and rate the following paper.

[0090] You will be asked to evaluate responses on the following dimensions: topic relevance, innovation, and feasibility.

[0091] When replying, please strictly follow the following rules:

[0092] 1. Evaluate the paper along different dimensions, indicate its strengths or weaknesses in each dimension, and assign a score from 1 to 10 to each dimension.

[0093] 2. Finally, provide an overall score of 1 to 10 for the paper based on the evaluation of each dimension.

[0094] Generally speaking, the higher the quality of the paper and the more it meets the user requirements, the higher the score will be. Papers that do not meet the user requirements will receive a lower score.

[0095] Scoring Rules:

[0096] Topic Relevance:

[0097] When the paper is not relevant to user needs, it is scored 1-2.

[0098] Papers that are in the same field as the user's topic of interest, but not directly related, are scored 3-4.

[0099] Papers that are relevant to the topic of interest to the user but do not meet specific requirements (such as time, place, method) will be scored 5-6.

[0100] Papers that are closely related to the topic of interest to the user and meet most of the requirements (such as time, place, and method) are scored 7-8.

[0101] Papers that are closely related to the topic of interest to the user and meet all requirements (e.g., time, location, methods) are scored 9-10.

[0102] Innovation:

[0103] Papers that do not present new methods or discoveries are scored 1-2.

[0104] The innovation proposed in the paper is very small or incremental and is scored 3-4.

[0105] Papers that present new methods or findings but whose significance is limited are scored 5-6.

[0106] If the new method or discovery proposed in the paper is very valuable, the score is 7-8.

[0107] A score of 9-10 indicates that the paper proposes a very important new method or discovery, which has a huge impact on the relevant field.

[0108] feasibility:

[0109] 1-2 points means that the paper does not have any experimental or data support.

[0110] A score of 3-4 indicates that the paper has little experimental or data support.

[0111] 5-6 points means that there are some experiments and data in the paper, but they are not complete and sufficient.

[0112] 7-8 points indicate that the paper provides sufficient experiments and data and is reproducible.

[0113] 9-10 points means that the experiments and data in the paper are very complete, including experimental details and data descriptions, and can serve as the basis for subsequent work.

[0114] Total score:

[0115] A score of 1-2 means the paper is not worth reading.

[0116] 3-4 points means that you only need to read the abstract of the paper without reading it carefully.

[0117] A score of 5-6 indicates that the paper is worth a cursory read.

[0118] A score of 7-8 indicates that the paper requires careful reading.

[0119] A score of 9-10 indicates that the paper is valuable and worth reading repeatedly.

[0120] ===========;

[0121] The following is another example of a prompt word input that may be used as the first user input:

[0122] ===========;

[0123] Prompt word example 2:

[0124] You are an expert in the field of agriculture and are good at literature analysis.

[0125] Please judge whether the following papers are relevant to the topic of interest to the user.

[0126] Please strictly follow the following rules when replying:

[0127] 1. Please use 1 or 0 to indicate whether each paper is related to the topic of interest, where 1 means yes and 0 means no.

[0128] Please do not give any other responses.

[0129] 2. When judging whether it is relevant, please strictly follow every request made by the user in the topic of interest.

[0130] Including location, time, etc. If not, please judge it as irrelevant, that is, 0.

[0131] 3. Only papers that strictly meet all the user's requirements can be judged as relevant, that is, 1.

[0132] 4. Please express your final answer in the form of a list. Do not reply in other words.

[0133] Example:

[0134] [1, 0, 1, 0, 1, ...];

[0135] ===========;

[0136] In some embodiments, the first agent 121 may be configured to read a portion of a predetermined position of each document to screen multiple documents, wherein the portion of the predetermined position includes an abstract of the document and a portion of the body of the document, and the amount of text in the portion of the predetermined position meets the context window length limit of the first large language model. Limited by the context length limit of the large language model, it is impossible to input too much text into the window of the large language model for evaluation. In order to meet the requirements of the context window length limit, some existing document analysis methods only input the abstract portion of the document into the dialogue window of the large language model for analysis. However, analysis based only on the abstract of the document may omit key information in the body of the document, thereby making the evaluation results incomplete.

[0137] In some embodiments, the first agent 121 may input the abstract of each document and the first paragraph of each chapter into the first large language model. In some embodiments, when inputting the abstract of the document and the first paragraph of each chapter into the first large language model, if it is prompted that the length limit of the context window of the first large language model has been reached, the first agent 121 may instruct the first large language model to stop receiving further input. In some embodiments, after inputting the abstract of the document and the first paragraph of each chapter into the first large language model, if the length limit of the context window of the first large language model has not been reached, the first agent 121 may instruct to continue inputting another part of each chapter (e.g., the second paragraph, the third paragraph, ..., the last paragraph, etc.) into the first large language model.

[0138] Some embodiments take into account both the comprehensiveness of the analysis corpus and the length limit of the context window of the large language model by simultaneously inputting the abstract and part of the main text of the document into the large language model, so that the large language model can understand the details of the document as much as possible while understanding the overall content of the document, thereby providing more accurate and comprehensive evaluation results.

[0139] In some embodiments, the plurality of documents may include a first number of documents. The first agent 121 may be configured to determine an independent score for each document in the first number of documents based on user requirements, and to determine a relative score for each document in the first number of documents as a whole. The first agent 121 may be configured to determine a comprehensive score for each document based on the independent score and the relative score of each document in the first number of documents. The first agent may be configured to filter the plurality of documents based on the comprehensive score of each document in the first number of documents.

[0140] In some embodiments, the plurality of documents may include a first document and a second document. The first agent 121 may be configured to determine a first independent score for the first document, determine a second independent score for the second document, and determine a first relative score for the first document and a second relative score for the second document for both the first document and the second document based on user requirements. The first agent 121 may be configured to determine a first comprehensive score based on the first independent score and the second relative score. The first agent 121 may be configured to determine a second comprehensive score based on the second independent score and the second relative score. The first agent 121 may be configured to filter the plurality of documents based on the first comprehensive score and the second comprehensive score.

[0141] As an example, assume that the first agent 121 can call the first language model to determine the independent scores of document i and document j according to the following formula (1) and formula (2) respectively: , :

[0142] (1);

[0143] (2);

[0144] Among them, R i , R j is the topic relevance score, I i ,I j is the Innovation score, F i 、F j It is the feasibility score.

[0145] The first agent 121 can call the first language model to take document i and document j as input, evaluate them as a whole, and score each document with a number between 0 and 1 to calculate the relative score of each document. , .

[0146] The first agent 121 may call the first language model to calculate the independent scores of document i and document j based on their respective independent scores. , and relative scores , , the comprehensive scores are determined according to the following formulas (3) and (4) respectively: , :

[0147] (3);

[0148] (4);

[0149] As an example, evaluating document i individually can determine its independent score ={8, 9, 6}, evaluating document j individually can determine its independent score ={6, 6, 6}, inputting document i and document j into the first language model simultaneously can determine their relative scores =0.6, =0.9. From this, the comprehensive score of document i can be determined ={8×0.6=4.8,9×0.6=5.4, 6×0.6=3.6}, ={6×0.9=5.4, 6×0.9=5.4, 6×0.9=5.4}. From this example, we can see that if the individual documents are evaluated separately, the score of document i is better than that of document j. However, through mixed evaluation, the comprehensive score of document j is higher than that of document i. This shows that the addition of mixed evaluation helps to unify the scoring criteria, thereby making a more accurate horizontal comparison of documents.

[0150] In some embodiments, the plurality of documents may include three or more documents. The first agent 121 may be configured to determine a comprehensive score of each of the documents in a similar manner, and filter the plurality of documents based on the comprehensive score of each document.

[0151] In some embodiments, independent scores may be determined for a portion of the multiple documents, and comprehensive scores may be determined for another portion of the multiple documents. Finally, the multiple documents may be screened based on the determined independent scores and comprehensive scores.

[0152] With respect to the second agent 122, its implementation details are further described.

[0153] In some embodiments, the second agent 122 may be configured to generate a hierarchical data description based on the multimodal data in each document, wherein the data descriptions at different levels describe the multimodal data with different degrees of structure. As an example, the multimodal data may include text, pictures, and tables. The second agent 122 may be configured to generate multiple groups of data based on the hierarchical data description. The hierarchical data description with data descriptions of different degrees of structure can prevent ambiguity in the extracted data during the data extraction process, thereby further avoiding errors in the extracted data during the subsequent integration process.

[0154] In some embodiments, the hierarchical data description includes a description of the overall content of the multimodal data. In some embodiments, the hierarchical data description includes a data description in dictionary form, wherein the data description in dictionary form may include a description of row indexes and row entries associated with the table form, and / or the data description in dictionary form may include a description of column indexes and column entries associated with the table form. Providing a data description in dictionary form helps downstream agents (e.g., the third agent 123) to integrate tabular data through semantic similarity rather than string matching during the data integration stage, thereby enabling more data to be integrated while ensuring accuracy.

[0155] In some embodiments, the second agent 122 may also be configured to generate a data description with more levels. In some embodiments, the second agent 122 may also be configured to have a data description with a higher degree of structure. For example, a data description in the form of a dictionary may include more data elements. In some embodiments, the second agent 122 may also be configured to have a data description with a lower degree of structure (e.g., between a description of the overall content and a data description in the form of a dictionary).

[0156] The following Table 1 is an example table. As an example, Table 1 may be converted into a markdown format by the second agent 122, but the scope of the present invention is not limited thereto.

[0157]

[0158] With respect to Table 1 above, the second agent 122 may be configured to output the following two-level data descriptions (eg, caption descriptions):

[0159] ===========;

[0160] [Start of level 1 caption]:

[0161] This table provides clay mineralogy results and inferred thermal excursions for different fault gouge samples and host rock samples, which aid in understanding their thermal history and mineral transformations.

[0162] [End of level 1 caption];

[0163] [Start of level 2 caption]:

[0164] Sample: An identification number for each sample in fault gouge or host rock.

[0165] Type: Indicates whether the sample is fault gouge or host rock, which helps classify the source of the sample.

[0166] Illite Moisture (Relative): The relative moisture of the illite, which is an indication of aqueous conditions.

[0167] Chlorite (Peak Ratio*): The peak ratio indicated by the 001 reflection and the 002 reflection from the chlorite measurement, which is necessary to evaluate the mineral structure.

[0168] Inferred Thermal Excursion (Illite): The estimated range of temperatures experienced by an illite sample that is indicative of historical thermal events.

[0169] Inferred Thermal Excursions (Chlorite): Estimated range of temperatures experienced by chlorite samples that are indicative of historical thermal events.

[0170] [End of level 2 caption];

[0171] ===========;

[0172] In some embodiments, in order to enable the multimodal large model called by the second agent 122 to generate a hierarchical data description for each table, the multimodal answer model can be fine tuned. For example, a large number of documents containing literature can be collected, and the tables therein can be divided into different parts. In some embodiments, each table can be divided into a triple containing three parts. For example, the table can be divided into a table body, a table title, and a table footer. After collecting a large number of table triplets, they can be used as a fine-tuning data set for the multimodal large model to fine-tune the multimodal large model.

[0173] When fine-tuning the multimodal large model, the input of the multimodal large model may be a table body, and the target of the multimodal large model output may be a table title and a table footer in a triple. As an example, the multimodal large model may be optimized by a gradient descent algorithm, so that the multimodal large model can learn the output including the table title and the table footer.

[0174] In some embodiments, in order to speed up the fine-tuning training of multimodal large models, a low-rank adaptation (LoRA) algorithm can be used. LoRA is a technology for fine-tuning large-scale pre-trained models, which aims to reduce the update scale of parameters by introducing low-rank matrices, thereby improving training efficiency and reducing computational costs. The basic idea of ​​LoRA is to add a set of low-rank adaptive adjustments to the original weights of the model, allowing effective fine-tuning of the model with only a small number of parameters updated.

[0175] In order to implement the LoRA algorithm, in some embodiments, the parameter decomposition of the model (e.g., a multimodal large model) may be performed first. For the weight matrix W of some layers, assuming that the size of W is m×n, it can be decomposed into two low-rank matrices A and B, as shown in the following equation (5), where the size of matrix A can be m×r, and the size of matrix B can be r×n, where r is the rank of the low rank.

[0176] (5);

[0177] Then, during fine-tuning, the optimization goal is to make the loss function Minimize. In some embodiments, the loss function It can be the cross entropy loss, which can be calculated by the following formula (6):

[0178] (6);

[0179] in, are the parameters of the original model, f() is the forward propagation function of the model, and x i is the input sample, y i is the corresponding label.

[0180] During the optimization training process, only parameters A and B can be updated, while keeping parameters The parameters can be updated by gradient descent method, as shown in the following equations (7) and (8):

[0181] (7);

[0182] (8);

[0183] Here is another example of data description in dictionary form:

[0184] ===========;

[0185] "Row 1": The content of various PM2.5 pollutants in Beijing.

[0186] "Row 2": The content of various PM2.5 pollutants in Shanghai.

[0187] "Column 1": Sulfide content,

[0188] "Column 2": black carbon content;

[0189] ===========;

[0190] For each number in a table in the original document of the document or in a table obtained from the original document, the description of its row entry (e.g., row attributes) and the description of its column entry (e.g., column attributes) can be obtained according to the row and column in which it is located (e.g., based on the row index and column index of the table) as shown in the above example. For example, for the number in the first row and second column of the table, it can be known from the dictionary-style data description in the above example that its row attribute is "the content of various PM2.5 pollutants in the Beijing area" and its column attribute is "black carbon content". This structured data description in the form of a dictionary can be used by downstream agents (e.g., the fourth agent 124) to check whether errors occur in the downstream data integration stage (e.g., performed by the third agent 123). For example, if the first column of the integrated table is "black carbon content", it is only necessary to verify whether the column attribute of each number in the first column is "black carbon content".

[0191] The following will combine Figure 2 The following describes an example in which the second agent 122 calls the multimodal macro model to convert a chart in a document into multiple sets of data (eg, multiple tables) in chart form.

[0192] In some embodiments, for images in a document that are difficult to convert into a table, the second agent 122 may be configured to call the multimodal large model to output the description content of the image point by point. In some embodiments, the multimodal large model may be instructed to output the description content of the image point by point through the design of prompt words. Figure 3 An example is described in which the second agent 122 calls upon the multimodal large model to output a description of the main contents of a picture point by point.

[0193] With respect to the third agent 123, its implementation details are further described.

[0194] In some embodiments, the user can provide items of interest to the second language model called by the third agent 123 by providing a table template. The following Table 2 is an example table template.

[0195]

[0196] Through the above table template, the second largest language model can understand that the items that the user cares about include "country", "province", "city", "including record number", "sulfate", "nitrate", "ammonium salt", "organic carbon", and "black carbon". By providing an existing table as a template, the user does not need to input the items that the user cares about.

[0197] In some embodiments, a table template in markdown format may be provided, but the scope of the present invention is not limited in this regard.

[0198] In some embodiments, the user may also provide the items that the user is interested in through other methods. For example, the items that the user is interested in may be provided to the second largest language model through text input.

[0199] In some embodiments, after the second language model receives the second user input including the item that the user cares about, the third agent 123 may be configured to determine whether each set of data (in the form of a table) in the multiple sets of data (in the form of a table converted by the second agent 122) contains the target data associated with the item that the user cares about. The third agent 123 may be configured to integrate the sets of data including the target data in the multiple sets of data into a single set of data (in the form of a table) based on the result of the determination.

[0200] In some embodiments, the second largest language model is enabled to determine whether each set of data includes target data associated with an item that the user is concerned about through the prompt word. The following prompt word example 3 shows an example of the prompt word.

[0201] ===========;

[0202] Prompt word example 3:

[0203] Who are you <input1>Experts in the field, skilled in analyzing whether the tables in the paper contain data of interest.

[0204] Determine whether multiple tables in the paper contain data on the topic of interest.

[0205] When replying, please strictly follow the following rules:

[0206] 1. The user will provide multiple tables (first level), each of which can contain sub-tables (second level).

[0207] You just need to decide whether the first-level table is relevant to the topic.

[0208] In other words, the number of answers you give should be equal to the number of first-level tables.

[0209] 2. For each first-level table, indicate using 1 or 0 whether it contains data on the topic of interest, with 1 for yes and 0 for no.

[0210] Please do not give any other responses.

[0211] 3. Please express your final answer in list form and do not respond with other text.

[0212] Example:

[0213] [1, 0, 1, 0, 1, ...];

[0214] ===========;

[0215] By inputting the above prompt word example 3, the second largest language model will give a judgment of 0 or 1 for each set of data in the form of a table (which may correspond to each table), where 0 indicates that the table does not include the item that the user is interested in, and 1 indicates that the table includes the item that the user is interested in. The table marked as 1 can be input into the second largest language model for data integration.

[0216] In some embodiments, the third agent 123 may be configured to output the source of each data in a single set of data, wherein the source indicates which set of data from multiple sets of data and which document from multiple documents. By outputting the source of each data in a single set of data while outputting the integrated single set of data, the second largest language model can be prevented from "hallucination" and outputting non-existent data, and the source is input to the downstream agent for inspection. At the same time, in order to verify the accuracy of data extraction and integration, a portion of the forms will be manually sampled. During the sampling process, the source can be used to help inspectors quickly verify the authenticity of the data.

[0217] The following Table 3 shows a single set of data in a tabular form with data sources output by the third agent 123. The entries in Table 3 are associated with the entries in Table 2 above.

[0218]

[0219] As described above, in some embodiments, the second agent 122 may generate multiple sets of data based on the hierarchical data description. In some embodiments, the hierarchical data description may include a data description in the form of a dictionary, which may include a description of row items and / or a description of column items associated with a table form. For these embodiments, in some embodiments, semantic matching may be used, and the second largest language model called by the third agent 123 may be fine-tuned so that the second largest language model can better utilize the hierarchical data description provided by the upstream second agent 122 for data integration.

[0220] In some embodiments, the third agent 123 may be configured to analyze the semantic similarity of the description of the row item and / or the description of the column item to perform a consistency check on the multiple sets of data. In some embodiments, the description of the row item and / or the description of the column item may include the full name and the abbreviation of the variable. The multiple sets of data are integrated based on the results of the consistency check.

[0221] The semantic matching and instruction fine-tuning process of the second largest language model as an example is described as follows.

[0222] First, the hierarchical data description output by the second agent 122 is converted into a semantic vector representation according to the following formula (9):

[0223] (9);

[0224] Among them, E() represents the pre-trained language model, T represents the hierarchical data description (which is in text form), and v T Represents a semantic vector.

[0225] Secondly, for the keywords "table title" and "table footnote", the same processing is performed according to the following formulas (10) and (11) to obtain the corresponding semantic vector representation:

[0226] (10);

[0227] (11);

[0228] Then, cosine similarity is used to calculate the similarity between the hierarchical data description and the keywords "table title" and "table footnote" , , as shown in the following formula (12) and formula (13):

[0229] (12);

[0230] (13);

[0231] Then, the hierarchical data description with the highest similarity is selected as the part corresponding to the keywords "table title" and "table footnote", as shown in the following formulas (14) and (15):

[0232] (14);

[0233] (15);

[0234] Finally, through instruction fine-tuning technology, the second largest language model focuses on hierarchical data description, thus avoiding disagreements in the data integration stage.

[0235] The following prompt word example 4 is a prompt word example that can implement the above process.

[0236] ===========;

[0237] Prompt word example 4:

[0238] Please extract data relevant to your topic from multiple tables in your paper and organize them into a new table.

[0239] Please strictly follow the following rules when replying:

[0240] 1. Ensure that the converted data matches the original data in value and units, and clearly indicate the units.

[0241] 2. Please only provide one integrated table in markdown format, discarding unnecessary and unmergeable data.

[0242] 3. The format of the output table after integration should follow the template provided by the user. However, do not output the data in the template.

[0243] 4. If only one piece of data can be integrated, please output a single-row table.

[0244] If multiple pieces of data can be combined, output a multi-row table.

[0245] If there is a missing row in the integrated data, please use "NaN" instead.

[0246] 5. After integrating the tables, indicate which table each data comes from, with the coordinates being the i-th row and the j-th column.

[0247] 6. When integrating tables, please pay attention to the primary and secondary descriptions of each table. The primary description reflects the main content of the table. The secondary description reflects the specific meaning of each row and column in the table. When integrating data from different tables, please ensure that the meaning of the rows and columns is consistent and the units are the same.

[0248] ===========;

[0249] By using semantic matching instead of string matching in the data merging stage through the second language model, the data of items with the same semantics but different string expressions can be effectively integrated, thereby increasing the amount of effective and accurate integrated data. For example, "Beijing area" and "Beijing" are semantically the same, but the strings are different. Semantic matching can effectively integrate the data corresponding to these two items.

[0250] With respect to the fourth agent 124, its implementation details are further described.

[0251] In some embodiments, the fourth agent 124 may be configured to determine the overall score based on at least one of the user interest relevance and accuracy. The fourth agent 124 may be configured to compare the overall score with a preset threshold. The fourth agent 124 may be configured to iteratively provide instructions to update the single set of data based on the comparison result until the overall score reaches the threshold.

[0252] In some embodiments, the fourth agent 124 may be configured to not provide an instruction to update a single set of data in response to the result of the comparison indicating that the overall score is lower than a threshold. As an example, the fourth agent 124 may provide an instruction to the third agent 123 in response to the result of the comparison indicating that the overall score is lower than a threshold. Based on the instruction, the third agent 123 may reject certain data in the data integration stage. In some embodiments, the instruction may include detailed information of one or more data that need to be updated so that the third agent 123 can locate these data more quickly to reintegrate the data. As an example, the fourth agent 124 may check that one or more data in the single set of data in the form of a table output by the third agent 123 is not from the multiple sets of data in the form of a table output by the second agent 122, thereby determining that the one or more data are inaccurate. Accordingly, the fourth agent 124 provides an instruction to the third agent 123. The instruction may cause the third agent 123 to continue to reject the one or more data in the data integration stage and reintegrate the data. Thereafter, the fourth agent 124 may check the single set of data generated after the third agent 123 reintegrates. This process can be iterated until the overall score of the inspection of the single set of data integrated by the third agent 123 reaches a threshold.

[0253] In some embodiments, the fourth agent 124 may be configured to respond to the comparison result indicating that the overall score is greater than or equal to the threshold value, and not provide an instruction to update the single set of data. This indicates that the current single set of data has passed the inspection and can be used in subsequent processes to generate a report of literature meta-analysis based thereon.

[0254] The following shows the process in which the fourth agent 124 iteratively improves the single set of data output by the third agent 123 through inspection, and the fourth agent 124 scores the inspection results and provides outputs of suggestions and decisions.

[0255] ===========;

[0256] {'Topic relevance': 6, 'Accuracy': 4, 'Overall score': 5, 'Recommendation': 'Incorporate geographic data, avoid data mismatches, enhance interpretation details, and ensure clear data source references.', 'Decision': 'Reject'};

[0257] ===========;

[0258] {'Topic relevance': 7, 'Accuracy': 4, 'Overall score': 5, 'Recommendation': "Ensure that all necessary details are included, such as exact years and geographic coordinates. Improve clarity and justification of data use, and ensure high accuracy when interpreting and combining data from multiple sources", 'Decision':'Reject'};

[0259] ===========;

[0260] {'Topic Relevance': 7, 'Accuracy': 8, 'Overall Score': 7, 'Recommendation': "Students should explore potential ways to infer or approximate the missing geographic and temporal data, even if speculative, to enhance the completeness of the analysis. In addition, a discussion of how this limitation affects the applicability of the data provides deeper insights into data utilization and reliability.", 'Decision': 'Accept'};

[0261] ===========;

[0262] Table 4 below shows an example table after checking by the fourth agent 124.

[0263]

[0264] As described above, in some embodiments, the third agent 123 is configured to output the source of each data in the single set of data. For such embodiments, in some embodiments, the fourth agent 124 may be configured to compare each data in the single set of data with corresponding data in the multiple data. The fourth agent 124 may be configured to determine accuracy based on the result of the comparison.

[0265] The following shows an example process in which the third agent 123 updates a single set of data based on the inspection results included in the instructions provided by the fourth agent 124 to improve the overall score.

[0266] The third agent 123 may output an example integrated table with data source explanations, as shown in Table 5 below:

[0267]

[0268] The output result of the fourth agent 124 checking the above Table 5 is as follows:

[0269] ===========;

[0270] 'Topic relevance': 8, 'Accuracy': 4, 'Overall score': 5, 'Recommendation': 'There is an error in the source of number 15. The data recorded in Table 4 is nitrogen dioxide, not carbon dioxide', 'Decision': 'Reject';

[0271] ===========;

[0272] The third agent 123 updates Table 5 based on the above output of the fourth agent 124 to output an example updated integrated table, as shown in the following Table 6:

[0273]

[0274] The output result of the fourth agent 124 checking the above Table 6 is as follows:

[0275] ===========;

[0276] 'Topic relevance': 8, 'Accuracy': 8, 'Overall score': 8, 'Recommendation': 'No significant issues', 'Decision': 'Accept';

[0277] ===========;

[0278] The above output result indicates that Table 6 has passed the inspection of the fourth agent 124, and the data in the table can be used to generate a report on literature meta-analysis.

[0279] In some embodiments, the plurality of agents may further include an optional fifth agent 125. The fifth agent 125 may be located upstream of the first agent 121.

[0280] In some embodiments, the fifth agent 125 may be configured to receive multiple original documents including documents. The multiple original documents may include documents downloaded from a database or obtained in any other way. The fifth agent 125 may be configured to perform page parsing on each of the multiple original documents to determine multiple parsed elements. The fifth agent 125 may be configured to merge multiple parsed elements to form each parsed document, wherein each parsed document formed constitutes a plurality of documents received by the first agent 121. Some embodiments utilize the fifth agent 125 to perform page parsing on the original documents containing documents, and the parsed documents formed can be better analyzed and processed by downstream agents.

[0281] In some embodiments, the plurality of original documents may include a diagram. The fifth agent 125 may be configured to invoke the multimodal language model to convert the diagram into corresponding parsed elements of the plurality of parsed elements.

[0282] In some embodiments, the chart may include a statistical chart and a table, and the multimodal language model may include a first multimodal language model and a second multimodal language model. The fifth agent 125 may be configured to call the first multimodal language model to convert the statistical chart into a corresponding parsed element in a plurality of parsed elements. The fifth agent 125 may be configured to call the second multimodal language model to convert the table into a corresponding parsed element in a plurality of parsed elements. As an example, the first multimodal language model may be a vision-language model (VLM). As an example, the second multimodal language model may be a vision language model. Some embodiments further subdivide the chart into a statistical chart and a table, and according to the different characteristics of the two, use separate multimodal language models to parse the statistical chart and the table into corresponding parsed elements, thereby improving the accuracy and effect of parsing the original document.

[0283] During the document parsing process, OCR technology often makes recognition errors for complex tables, especially those containing multiple sub-tables, thus affecting the accuracy of the parsed data.

[0284] In some embodiments, by designing prompt words, the first multimodal language model can convert each table (or sub-table) separately when parsing a complex table, thereby improving the accuracy of parsing. The following prompt word example 5 is an example of a prompt word input to the first multimodal language model to parse a table.

[0285] ===========;

[0286] Prompt word example 5:

[0287] Who are you <input1>Expert in the field, and is good at converting tables in papers from image format to markdown text.

[0288] Please convert the following jpg table into markdown text.

[0289] When replying, please strictly follow the following rules:

[0290] 1. Ensure that the converted data matches the original data in value and units, and clearly indicate the units.

[0291] 2. Keep the original format of the table.

[0292] 3. If there are multiple tables in the image, convert each table separately.

[0293] 4. For each table, give it a title that reflects the content of the table.

[0294] 5. For each table, provide a footnote describing the full name of each row and column.

[0295] ===========;

[0296] In some embodiments, for statistical graphs, by designing prompt words, the second multimodal language model can convert multiple statistical graphs into texts respectively, and for each of the multiple data in each statistical graph, each data is listed in the corresponding text. The following prompt word example 6 is an example of prompt words input to the second multimodal language model to enable it to parse the statistical graph.

[0297] ===========;

[0298] Prompt word example 6:

[0299] Who are you <input1>Expert in the field, good at converting statistical graphs in papers from image format to text.

[0300] Please convert the following jpg statistical chart into text.

[0301] Please strictly follow the following rules when replying:

[0302] 1. Ensure that the converted data is consistent with the original data in value and unit, and clearly indicate the unit.

[0303] 2. If there are multiple statistical charts in the image, please convert them into multiple texts respectively.

[0304] 3. If there are multiple data in a statistical chart, please list them separately in the corresponding text.

[0305] 4. Please analyze the data in each statistical chart and draw conclusions. For example: Figure 1 This shows that plant carbon dioxide emissions are positively correlated with soil moisture content.

[0306] ===========;

[0307] In some embodiments, the plurality of original documents may include text. The fifth agent 125 may be configured to convert the text into corresponding parsed elements of the plurality of parsed elements using OCR technology.

[0308] In some embodiments, the plurality of original documents may include formulas. The fifth agent 125 may be configured to call the fourth language model to convert the formulas into corresponding parsed elements among the plurality of parsed elements.

[0309] In some embodiments, a large amount of formula data sets may be used to perform fine-tuning training on the fourth language model, thereby significantly improving the formula conversion capability of the fourth language model.

[0310] In addition to the accurate parsing of each element of the document, the layout of each parsed element in the parsed document is also very important for subsequent analysis and processing.

[0311] In some embodiments, the fifth agent 125 is configured to input multiple parsed elements into the pages of the document in multiple candidate layouts for each document to be formed. The fifth agent 125 is configured to record the size and position of each parsed element in the page for each document to be formed and for each candidate layout, and calculate the adaptability under the current candidate layout based on the area of ​​the area occupied by the multiple parsed elements in the page and the total area of ​​the page. The fifth agent 125 is configured to merge multiple parsed elements in the candidate layout with the greatest adaptability for each document to be formed.

[0312] As an example, DocLayout-YOLO can be used to parse the layout of a document. DocLayout-YOLO uses YOLO-v10 as its infrastructure, and its efficient real-time detection capability is suitable for a variety of document layout detection.

[0313] For this model, in the pre-training stage, the model introduces the Mesh-candidate BestFit method to treat document synthesis as a two-dimensional box packing problem. Through this innovative perspective, the model can better understand and identify the layout characteristics of documents.

[0314] The Mesh-candidate BestFit algorithm aims to achieve and exploit layout effects by efficiently organizing and laying out document elements. It operates according to the following process.

[0315] First, document elements are input. Specifically, document elements to be synthesized, such as text boxes, images, tables, etc., are collected, and their sizes and positions are recorded.

[0316] Next, create a grid structure. Specifically, divide the document area into two-dimensional mesh units, where each mesh unit represents a possible layout position. Set the size of the mesh unit. Usually, the layout can be made based on the overall size of the document and the size of the elements.

[0317] Subsequently, candidate layouts are generated. Specifically, for each document element to be synthesized, multiple candidate layouts are generated. These candidate layouts are arrangements of different positions and combinations in the grid cells.

[0318] Subsequently, fitness evaluation is performed. Specifically, the fitness of each subsequent layout is calculated to evaluate its space utilization and layout rationality. As an example, the fitness can be calculated according to the following formula (16):

[0319] (16);

[0320] Among them, "used area" refers to the area occupied by document elements, and "total area" refers to the total area of ​​the document area.

[0321] Then, the best candidate layout is selected. Specifically, the layout with the highest fitness is selected from the generated candidate layouts as the best layout.

[0322] Subsequently, the layout is implemented. Specifically, according to the selected optimal layout, the document elements are placed in the corresponding grid cells to complete the synthesis of the document.

[0323] Finally, the synthesized document is output. Specifically, the synthesized document is output to ensure that all elements are arranged according to the optimal layout.

[0324] In some embodiments, a large-scale and diverse synthetic document dataset may be created to train the above-mentioned model for document layout. The synthetic document dataset may cover the structure of various documents, thereby enhancing the generalization ability of the model.

[0325] For multiple pages of a document, when the corresponding intelligent agent calls the model for analysis and processing, it is necessary to consider the overall space utilization and reduce page overlap.

[0326] In some embodiments, in order to take into account both space utilization and page overlap, when the fifth agent 125 merges multiple pages of a document, the comprehensive page score Score may be calculated according to the following formula (17):

[0327] Score = used space - overlapped space (17);

[0328] Among them, "used space" means the intersection of page layouts, and "overlapping space" means the union of page layouts.

[0329] By maximizing the comprehensive score of pages when merging pages, the model can make full use of the overall space when merging pages and reduce page number overlap, so that the final page integration effect has a better visual perception and layout rationality.

[0330] In some embodiments, the processing pipeline formed by multiple agents may also extend upstream.

[0331] In some embodiments, the plurality of agents may further include an optional sixth agent 126. The sixth agent 126 may be configured to search for relevant literature based on the research field of interest input by the user to obtain a literature list.

[0332] In some embodiments, the plurality of agents may further include an optional seventh agent 127. The seventh agent 127 may be configured to receive the document list output by the sixth agent 126 and automatically download the documents of each document in the document list. The plurality of documents downloaded by the seventh agent 127 may be used as input to the fifth agent 125, which performs document processing to parse page numbers, or may be directly used as input to the first agent 121, which performs document evaluation.

[0333] In some embodiments, the processing pipeline formed by multiple agents may also extend downstream.

[0334] In some embodiments, the plurality of agents may further include an optional eighth agent 128. The eighth agent 128 may be configured to receive a single set of data in a table form checked by the fourth agent 124 to perform data analysis on the single set of data. For example, regression analysis, cluster analysis, classification analysis, etc. may be performed on the single set of data.

[0335] In some embodiments, the eighth agent 128 may be configured to visualize the results of the analysis. In some embodiments, the visualization may include an automatic mode. In the automatic mode, the eighth agent 128 may call a corresponding model to automatically generate code for visual analysis based on the type and content of the data. Subsequently, by running the code, a visualization result of the data analysis may be directly obtained. In some embodiments, the visualization may include manual visualization. As an example, for more complex data, such as data that needs to be visualized in conjunction with a map, etc., manual visualization may be selected.

[0336] In some embodiments, the plurality of agents may further include an optional ninth agent 129. The ninth agent 129 may be configured to receive the result of the data analysis output by the eighth agent 128 to generate a report of the file meta-analysis.

[0337] Figure 2 A first example graph showing data processed using literature meta-analysis techniques according to some embodiments of the present invention to extract data.

[0338] Figure 2 The following Tables 7, 8 and 9 show the calculation of the computing device based on some embodiments of the present invention. Figure 2 The table generated by the three statistical charts in:

[0339]

[0340]

[0341]

[0342] Figure 3 A second example graph is shown of data processed using literature meta-analysis techniques according to some embodiments of the present invention to extract data.

[0343] Figure 3 The second example chart in cannot be converted into a table. In some embodiments, it can be (for example, by Figure 1 The second agent 122 in the image outputs the description of the image point by point. In some embodiments, the prompt words can be designed to Figure 1 The multimodal large model called by the second agent 122 in the description outputs the content of the picture point by point when the picture cannot be converted into a table. The following prompt word example 6 is an example of such a prompt word.

[0344] ===========;

[0345] Prompt word example 6:

[0346] If possible, please convert the image into a markdown table. If the image cannot be converted into a markdown table, please describe the image content point by point.

[0347] Simply start with "1. xxx" without any other text.

[0348] Example 2:

[0349] 1. The Nainital area is marked on the map and may be an important geological sampling site.

[0350] 2. There is a small inset on the top right corner of the map showing the larger geographical location relative to Nainital.

[0351] 3. xxx;

[0352] ===========;

[0353] Figure 4 is a flow chart of a processing pipeline for performing literature meta-analysis according to some embodiments of the present invention.

[0354] The processing pipeline may include a literature evaluation stage 410, a data extraction stage 420, a data integration stage 430, and a data check stage 440. The data integration stage 430 and the data check stage 440 may be iteratively executed until the integrated data meets the requirements. Figure 1 The data extraction phase 420 may be performed in conjunction with Figure 1 The data integration phase 430 may correspond to the operation performed by the second agent 122 in Figure 1 The data checking phase 440 may correspond to the operation performed by the third agent 123 in Figure 1 Corresponding to the operation performed by the fourth agent 124 in.

[0355] In some embodiments, the processing pipeline may also include an optional document processing stage 450. The document processing stage 450 may be associated with Figure 1 Corresponding to the operation performed by the fifth agent 125 in.

[0356] In some embodiments, the processing pipeline may also include an optional literature search stage 460. The literature search stage 460 may be combined with Figure 1 Corresponding to the operation performed by the sixth agent 126 in .

[0357] In some embodiments, the processing pipeline may also include an optional file download stage 470. The file download stage 470 may be associated with Figure 1 Corresponding to the operation performed by the seventh agent 127 in.

[0358] In some embodiments, the processing pipeline may also include an optional data analysis stage 480. The data analysis stage 480 may be associated with Figure 1 Corresponding to the operation performed by the eighth agent 128 in.

[0359] In some embodiments, the processing pipeline may also include an optional report generation stage 490. The report generation stage 490 may be associated with Figure 1 Corresponding to the operation performed by the ninth agent 129 in.

[0360] According to another aspect of the present invention, a method for literature meta-analysis is provided.

[0361] Figure 5 is a flow chart of a method for literature meta-analysis according to some embodiments of the present invention. Figure 1 Executed by the computing device 100 in .

[0362] The method may include step S1: the first agent calls the first language model to perform document evaluation. Step S1 may be performed by Figure 1 The first agent 121 in executes.

[0363] The method may include step S2: the second agent calls the multimodal large model to perform data extraction. Step S2 may be performed by Figure 1 The second agent 122 in executes.

[0364] The method may include step S3: the third agent calls the second language model to perform data integration. Step S3 may be performed by Figure 1 The third agent 123 in executes.

[0365] The method may include step S4: the fourth agent calls the third language model to perform data checking. Step S4 may be performed by Figure 1 The fourth agent 124 in executes.

[0366] Optionally, the method may further include step S5: a fifth agent performs document processing. Step S5 may be performed by Figure 1 The fifth agent 125 in executes.

[0367] Optionally, the method may further include step S6: a sixth agent performs a literature search. Step S6 may be performed by Figure 1 The sixth agent 126 in is executed.

[0368] Optionally, the method may further include step S7: downloading the document by a seventh agent. Step S7 may be performed by Figure 1 The seventh agent 127 in is executed.

[0369] Optionally, the method may further include step S8: performing data analysis by an eighth agent. Step S8 may be performed by Figure 1 The eighth agent 128 in is executed.

[0370] Optionally, the method may further include step S9: generating a report by a ninth agent. Step S9 may be performed by Figure 1 The ninth agent 129 in is executed.

[0371] Figure 6 is a flow chart of a first process associated with a method for literature meta-analysis according to some embodiments of the present invention. The first process may be Figure 1 The first agent 121 in the embodiment is executed and may be Figure 5 The specific implementation manner of step S1 in the method described above is described below, but the scope of the present invention is not limited thereto.

[0372] The first process may include step S11: receiving a plurality of documents and a first user input, wherein the plurality of documents include documents and the first user input includes a user requirement.

[0373] The first process may include step S12: scoring the literature corresponding to each document based on user requirements.

[0374] The first process may include step S13: screening the plurality of documents based on the result of the scoring to determine a subset of the documents.

[0375] Figure 7 is a flow chart of a second process associated with a method for literature meta-analysis according to some embodiments of the present invention. The second process may be Figure 1 The second agent 122 in the Figure 5 The specific implementation manner of step S2 in the method in the embodiment of the present invention is described below, but the scope of the present invention is not limited thereto.

[0376] The second process may include step S21: receiving a subset of documents.

[0377] The second process may include step S22: converting the multimodal data of each document in the subset of documents into a plurality of sets of data in a table form.

[0378] Figure 8 is a flow chart of a third process associated with a method for literature meta-analysis according to some embodiments of the present invention. The third process may be Figure 1 The third agent 123 in is executed and can be Figure 5 The specific implementation manner of step S3 in the method in the embodiment of the present invention is described below, but the scope of the present invention is not limited thereto.

[0379] The third process may include step S31: receiving multiple groups of data and second user input, wherein the second user input includes items that the user is concerned about.

[0380] The third process may include step S32: integrating multiple sets of data into a single set of data in a table format based on items that the user is concerned about.

[0381] Fig. 9 is a flow chart of a fourth process associated with a method for literature meta-analysis according to some embodiments of the present invention. The fourth process may be Figure 1 The fourth agent 124 in the embodiment is executed and may be Figure 5 The specific implementation manner of step S4 in the method in the embodiment of the present invention is described below, but the scope of the present invention is not limited thereto.

[0382] The fourth process may include step S41: receiving multiple sets of data and a single set of data.

[0383] The fourth process may include step S42 of checking a single set of data against multiple sets of data based on at least one of user interest relevance and accuracy.

[0384] The fourth process may include step S43: iteratively providing instructions to update the single set of data based on the result of the check until the updated single set of data passes the check, wherein a report of the literature meta-analysis is generated based on the single set of data that passes the check.

[0385] Fig.10 is a flow chart of a fifth process associated with a method for literature meta-analysis according to some embodiments of the present invention. The fifth process may be Figure 6 The specific implementation of step S12 in the first process is described above, but the scope of the present invention is not limited thereto.

[0386] The fifth process may include step S121: determining an independent score for each document in the first number of documents based on user requirements, and determining a relative score for each document in the first number of documents as a whole.

[0387] The fifth process may include step S122: determining a comprehensive score for each document based on the individual score and the pair score of each document in the first number of documents.

[0388] The fifth process may include step S123: reading a portion at a predetermined position of each document to score each document, wherein the portion at the predetermined position includes an abstract of the document and a portion of the body of the document, and the amount of text in the portion at the predetermined position meets the context window length limit of the first language model.

[0389] In some embodiments, step S123 may be performed independently of step S121 and step S122. In some embodiments, step S123 may be performed in combination with step S121 and step S122.

[0390] Fig.11 is a flow chart of a sixth process associated with a method for literature meta-analysis according to some embodiments of the present invention. The sixth process may be Figure 7 The specific implementation of step S22 in the second process in FIG. 1 is described below, but the scope of the present invention is not limited thereto.

[0391] The sixth process may include step S221: generating a hierarchical data description based on the multimodal data in each document, wherein data descriptions at different levels describe the multimodal data at different levels of structure.

[0392] The sixth process may include step S222: generating multiple groups of data based on the hierarchical data description.

[0393] The sixth process may include step S223: outputting the description content of the picture point by point.

[0394] In some embodiments, step S223 may be performed independently of step S221 and step S222. In some embodiments, step S223 may be performed in combination with step S221 and step S222.

[0395] Fig.12 is a flow chart of a seventh process associated with a method for literature meta-analysis according to some embodiments of the present invention. The seventh process may be Figure 8 The specific implementation of step S32 in the third process in FIG. 1 is described below, but the scope of the present invention is not limited thereto.

[0396] The seventh process may include step S321: determining whether each set of data in the plurality of sets of data includes target data associated with an item of interest to the user.

[0397] The seventh process may include step S322: based on the result of the determination, integrating the groups of data including the target data in the multiple groups of data into a single group of data.

[0398] The seventh process may include step S323: outputting the source of each data in the single set of data, wherein the source indicates which set of data among the multiple sets of data and which document among the multiple documents the data comes from.

[0399] In some embodiments, step S323 may be performed independently of step S321 and step S322. In some embodiments, step S323 may be performed in combination with step S321 and step S322.

[0400] Fig.13 is a flow chart of an eighth process associated with a method for literature meta-analysis according to some embodiments of the present invention. The eighth process may be Fig. 9 The specific implementation of step S42 in the fourth process in FIG. 1 is described below, but the scope of the present invention is not limited thereto.

[0401] The eighth process may include step S421: determining an overall score based on at least one of user interest relevance and accuracy.

[0402] The eighth process may include step S422: comparing the overall score with a preset threshold.

[0403] The eighth process may include step S423: iteratively providing an indication based on the result of the comparison until the overall score reaches a threshold. The indication may be Fig. 9 The instruction to update the single set of data in step S43 in .

[0404] Fig.14 is a flow chart of a ninth process associated with a method for literature meta-analysis according to some embodiments of the present invention. The ninth process may be Fig.13 The specific implementation of step S423 in the eighth process is described above, but the scope of the present invention is not limited to this.

[0405] The ninth process may include step S4321: determining whether the overall score is greater than or equal to a threshold.

[0406] The ninth process may include step S4322: in response to the result of the above comparison indicating that the overall score is greater than or equal to the threshold, no indication is provided. The indication may be Fig. 9 The instruction to update the single set of data in step S43 in .

[0407] The ninth process may include step S4323: in response to the result of the above comparison indicating that the overall score is less than the threshold, providing an indication. The indication may be Fig. 9 The instruction to update the single set of data in step S43 in .

[0408] Fig.15 is a flow chart of a tenth process associated with a method for literature meta-analysis according to some embodiments of the present invention. The tenth process may be Figure 1 The fifth agent 125 in the embodiment is executed and may be Figure 5 The specific implementation manner of step S5 in the method in the embodiment of the present invention is described below, but the scope of the present invention is not limited thereto.

[0409] The tenth process may include step S51: receiving a plurality of original documents including documents.

[0410] The tenth process may include step S52: performing page parsing on each of the plurality of original documents to determine a plurality of parsed elements.

[0411] The tenth process may include step S53: merging multiple parsed elements to form each parsed document, wherein each parsed document formed constitutes multiple documents.

[0412] Fig.16 is a flow chart of an eleventh process associated with a method for literature meta-analysis according to some embodiments of the present invention. The eleventh process may be Fig.15 The specific implementation of step S52 in the tenth process in FIG. 1 is described below, but the scope of the present invention is not limited thereto.

[0413] The eleventh process may include step S521: calling a multimodal language model to convert the graph into a corresponding parsed element among a plurality of parsed elements.

[0414] In some embodiments, the chart may include a statistical chart. Step S521 may include step S5211: calling a first multimodal language model to convert the statistical chart into a corresponding parsed element in a plurality of parsed elements.

[0415] In some embodiments, the diagram may include a table. Step S521 may include step S5212: calling a second multimodal language model to convert the table into a corresponding parsed element of the plurality of parsed elements.

[0416] The eleventh process may include step S522: using optical character recognition to convert the text into a corresponding parsed element among a plurality of parsed elements.

[0417] The eleventh process may include step S523: using the fourth language model to convert the formula into a corresponding parsed element among the plurality of parsed elements.

[0418] Step S521 , step S522 , and step S523 may be performed independently of each other or in combination with each other.

[0419] Fig.17 is a flow chart of a twelfth process associated with a method for literature meta-analysis according to some embodiments of the present invention. The twelfth process may be Fig.15 The specific implementation of step S53 in the tenth process in FIG. 1 is described below, but the scope of the present invention is not limited thereto.

[0420] The twelfth process may include step S531: inputting a plurality of parsed elements into a page of the document in a plurality of candidate layouts respectively.

[0421] The twelfth process may include step S532: recording the size and position of each parsed element in the page for each candidate layout, and calculating the fitness under the current candidate layout based on the area of ​​the region occupied by multiple parsed elements in the page and the total area of ​​the page.

[0422] The twelfth process may include step S533: merging the multiple parsed elements with a candidate layout having the greatest fitness.

[0423] According to another aspect of the present invention, a computer-readable storage medium is provided.

[0424] Fig.18 is a block diagram of a computer-readable storage medium 1800 according to some embodiments of the present invention.

[0425] The computer readable storage medium 1800 stores a computer program 1850. When the computer program 1850 is executed by the processor, the computer program 1850 implements the above-mentioned Figure 5-Figure 17 The steps of each method or process are described.

[0426] According to another aspect of the present invention, a computer program product is provided.

[0427] Fig.19 is a block diagram of a computer program product 1900 according to some embodiments of the invention.

[0428] The computer program product 1900 may include a computer program 1850. When the computer program 1850 is executed by a processor, the computer program 1850 implements the above-mentioned Figure 5-Figure 17 The steps of each method or process are described.

Claims

1. A computing device for literature meta-analysis, characterized in that: include: Computing resources; as well as A plurality of agents, the plurality of agents being executed by the computing resources, the plurality of agents comprising: The first agent is configured to call the first language model to: receiving a plurality of documents and a first user input, the plurality of documents comprising documents, the first user input comprising a user requirement; Scoring the literature corresponding to each document based on the user's requirements; Filtering the plurality of documents based on the results of the scoring to determine a subset of documents; The second agent is configured to call the multimodal large model to: receiving a subset of the documents; converting the multimodal data in each document in the subset of documents into a plurality of sets of data in a tabular form, each set of data including an entry and data for the entry; The third agent is configured to call the second largest language model to: receiving the plurality of sets of data and a second user input, wherein the second user input includes items of interest to the user; Integrating the multiple sets of data into a single set of data in a table format based on the items that the user is concerned about; and The fourth agent is configured to call the third language model to: receiving the multiple sets of data and the single set of data; checking the single set of data against the multiple sets of data based on at least one of relevance to user interests and accuracy; and Based on the result of the check, an instruction to update the single set of data is iteratively provided until the updated single set of data passes the check, wherein a report of the literature meta-analysis is generated based on the single set of data that passes the check.

2. The computing device of claim 1, wherein: the plurality of documents comprises a first number of documents, The first agent is configured to call the first large language model to: Based on the user requirement, determining an independent score for each document in the first number of documents, and determining a relative score for each document in the first number of documents as a whole; determining a composite score for each document based on the individual scores and the relative scores of each document in the first number of documents; as well as The plurality of documents are filtered based on a composite score of each document in the first number of documents.

3. The computing device of claim 1, wherein: The user requirements include at least one of subject relevance, innovation, feasibility, and / or It is characterized in that the first user inputs the reason for requiring the first language model to output a score.

4. The computing device of claim 1, wherein: The first agent is configured to call the first large language model to: A portion at a predetermined position of each document is read to score each document, wherein the portion at the predetermined position includes an abstract of the document and a portion of the main text of the document, and the amount of text in the portion at the predetermined position complies with a context window length limit of the first largest language model.

5. The computing device of claim 1, wherein: The second agent is configured to call the multimodal large model to: Based on the multimodal data in each document, generating hierarchical data descriptions, wherein different levels of data descriptions describe the multimodal data with different levels of structure; and Based on the hierarchical data description, the multiple groups of data are generated.

6. The computing device of claim 5, wherein: The hierarchical data description includes a description of the overall content of the multimodal data.

7. The computing device of claim 5, wherein: The hierarchical data description includes a data description in a dictionary form, and the data description in the dictionary form includes a description of row items and / or a description of column items associated with a table form.

8. The computing device of claim 7, wherein: The third agent is configured to call the second language model to: Analyze the semantic similarity of the description of the row item and / or the description of the column item to perform a consistency check on the multiple sets of data, wherein the description of the row item and / or the description of the column item includes the full name and the abbreviation of the variable, and the multiple sets of data are integrated based on the result of the consistency check.

9. The computing device of claim 1, wherein: The multimodal data includes pictures, The second agent is configured to call the multimodal large model to: Output the description content of the picture point by point.

10. The computing device of claim 1, wherein: The third agent is configured to call the second language model to: Determining whether each of the plurality of data sets includes target data associated with the item of interest to the user; and Based on the result of the determination, each set of data including the target data in the plurality of sets of data is integrated into the single set of data.

11. The computing device of claim 1, wherein: The third agent is configured to call the second language model to: The source of each data in the single set of data is output, where the source is used to indicate which set of data in the multiple sets of data and which document in the multiple documents each data comes from.

12. The computing device of claim 11, wherein: The fourth agent is configured to call the third language model to: Comparing each data in the single set of data with corresponding data in the multiple sets of data; as well as Based on the results of the comparison, the accuracy of the data is determined.

13. The computing device of claim 1, wherein: The fourth agent is configured to call the third language model to: determining an overall score based on at least one of the user interest relevance and accuracy; comparing the overall score with a preset threshold; as well as Based on a result of the comparing, the indication is iteratively provided until the overall score reaches the threshold.

14. The computing device of claim 13, wherein: The fourth agent is configured to call the third language model to: In response to a result of the comparing indicating that the overall score is below the threshold, providing the indication; In response to a result of the comparing indicating that the overall score is greater than or equal to the threshold, not providing the indication.

15. The computing device of claim 1, wherein: The plurality of agents also include: The fifth agent is configured as: receiving a plurality of original documents including documents; performing page parsing on each of the plurality of original documents to determine a plurality of parsed elements; The plurality of parsed elements are merged to form each parsed document, wherein each parsed document formed constitutes the plurality of documents.

16. The computing device of claim 15, wherein: The plurality of original documents include diagrams, The fifth agent is configured to: The multimodal language model is invoked to convert the graph into corresponding parsed elements of the plurality of parsed elements.

17. The computing device of claim 16, wherein: The chart includes a statistical chart and a table, the multimodal language model includes a first multimodal language model and a second multimodal language model, The fifth agent is configured to: calling the first multimodal language model to convert the statistical graph into corresponding parsed elements among the plurality of parsed elements; and The second multimodal language model is called to convert the table into corresponding parsed elements among the plurality of parsed elements.

18. The computing device of claim 16, wherein: The multiple original documents include text and formulas, The fifth agent is configured to: converting the text into corresponding parsed elements of the plurality of parsed elements using optical character recognition technology; A fourth language model is called to convert the formula into corresponding parsed elements among the plurality of parsed elements.

19. The computing device of claim 15, wherein: The fifth agent is configured to, for each document to be formed: Inputting the plurality of parsed elements into pages of the document in a plurality of candidate layouts respectively; Recording the size and position of each parsed element in the page for each candidate layout, and calculating the fitness under the current candidate layout based on the area of ​​the region occupied by the multiple parsed elements in the page and the total area of ​​the page; as well as The multiple parsed elements are merged using the candidate layout with the maximum fitness.

20. A method for literature meta-analysis, characterized in that: The following steps are involved: S1: The first agent calls the first language model to perform literature review, and the step S1 includes: S11: receiving a plurality of documents and a first user input, wherein the plurality of documents include documents, and the first user input includes a user requirement; S12: Score the literature corresponding to each document based on the user's request; S13: Filtering the plurality of documents based on the scoring result to determine a subset of documents; S2: The second agent calls the multimodal large model to perform data extraction, and the step S2 includes: S21: receiving a subset of the document; S22: converting the multimodal data in each document in the subset of documents into multiple sets of data in a table form, each set of data including an entry and data for the entry; S3: The third agent calls the second largest language model to perform data integration, and the step S3 includes: S31: receiving the multiple groups of data and a second user input, where the second user input includes items that the user is concerned about; S32: integrating the multiple sets of data into a single set of data in a table format based on the items that the user is concerned about; S4: The fourth agent calls the third language model to perform data checking, and the step S4 includes: S41: receiving the multiple groups of data and the single group of data; S42: Checking the single set of data against the multiple sets of data based on at least one of user interest relevance and accuracy; S43: Based on the result of the check, iteratively provide an instruction to update the single set of data until the updated single set of data passes the check, wherein a report of the literature meta-analysis is generated based on the single set of data that passes the check.

21. The method of claim 20, wherein: The step S22 comprises the following steps: S221: generating hierarchical data descriptions based on the multimodal data in each document, wherein each hierarchical data description describes the multimodal data at a different structural level; and S222: Generate the multiple groups of data based on the hierarchical data description.

22. The method of claim 20, wherein: The step S32 comprises the following steps: S321: Determine whether each of the plurality of data groups includes target data associated with the item that the user is concerned about; and S322: Integrate the groups of data including the target data in the multiple groups of data into the single group of data.

23. The method of claim 20, wherein: The step S42 comprises the following steps: S421: Determine an overall score based on at least one of the user interest relevance and accuracy; S422: Compare the overall score with a preset threshold; and S423: Based on the result of the comparison, iteratively provide the indication until the overall score reaches the threshold.

24. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 20 to 23 are implemented.

25. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 20 to 23 are implemented.

Images