An academic paper abstract generation method based on image-text fusion index

Abstract

The application relates to an academic paper abstract generation method based on a picture-text fusion index and belongs to the technical field of artificial intelligence and multi-modal information processing. The method comprises the following steps: obtaining academic paper data from a public data source; according to the academic papers in the academic paper data, logically cutting the paper content, establishing the association between the segments and images, and constructing a picture-text fusion index; performing semantic retrieval on four dimensions in the academic paper to obtain the final retrieval content set of all dimensions; and generating a picture-text fusion abstract based on the retrieval content set of the four dimensions. The application strengthens the picture-text association strength and the chapter context coherence by constructing the picture-text fusion index, realizes the deep coupling of the image and the text at the semantic level, and improves the picture-text collaborative expression capability of the abstract; the structured dimension template and the content retrieval mechanism driven by the standard question are designed to ensure the structured coverage of the key information and significantly improve the logical coherence and readability of the abstract.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and multimodal information processing technology, and specifically relates to a method for generating academic paper abstracts based on image-text fusion indexing. Background Technology

[0002] With the rapid increase in the number of academic documents, researchers face the challenge of information overload, making automatic summarization technology an important tool for improving reading efficiency. Traditional summarization methods mainly focus on the plain text modality, extracting or generating the core content of papers to provide researchers with a quick way to understand the literature. However, with the rise of multimodal technologies, incorporating images (such as charts and diagrams) into the summarization process has gradually gained attention to enhance the understanding and expression of complex scientific literature.

[0003] Existing image-text fusion summarization methods suffer from three key shortcomings: First, there is a lack of semantic alignment between images and text. Most methods employ a piecemeal strategy of "generating a text summary first, then attaching key images," relying on heuristic rules or global similarity matching for image selection. This fails to establish fine-grained semantic connections between images and specific text fragments, resulting in a disconnect between image content and context. Second, image citation relationships are not explicitly modeled. The citation positions and functional roles of images in the original text (such as demonstrating methodological processes or supporting experimental conclusions) are not structurally utilized, leading to a lack of logical basis for image insertion in the summary and a degree of arbitrariness. Third, image-text fusion remains at the level of formal splicing. The generated text lacks natural language representation of the images, and there is no collaborative expression mechanism between images and text. Readers must infer the relationship between the two themselves, severely impacting information comprehension efficiency and summary usability. These problems collectively prevent existing methods from achieving true "image-text fusion," limiting their application value in demanding scenarios such as intelligent summarization of academic literature.

[0004] To address the aforementioned issues, this invention proposes an academic paper abstract generation method based on a text-image fusion index. Through steps such as text-image fusion index construction, dimension-based retrieval, and text-image fusion abstract generation, it achieves deep semantic integration of text and images, thereby generating a text-image fusion abstract that is complete in information, clear in structure, and highly coupled with images, providing efficient technical support for academic research and document processing. Summary of the Invention

[0005] To achieve the above objectives, the present invention employs the following technical solution: This invention provides a method for generating academic paper abstracts based on image-text fusion indexing, comprising the following steps: Step 1: Obtain academic paper data from publicly available data sources; Step 2: Based on the academic papers in the academic paper data, logically segment the paper content and establish the association between fragments and images to construct an image-text fusion index; Step 3: Perform semantic search on the four dimensions of the academic paper to obtain the final search content set for all dimensions; Step 4: Generate a fusion summary of images and text based on the search content set from four dimensions.

[0006] Furthermore, existing technologies suffer from problems such as a lack of semantic alignment between images and text, the absence of explicit modeling of image referencing relationships, and a lack of contextual support for image insertion. To address these issues, this invention divides the paper content into semantically coherent logical segments and establishes structured associations between images and specific text segments, providing a traceable and locatable indexing foundation for deep image-text fusion. Step two specifically involves: Document parsing: Using a PDF document parsing tool to parse the academic paper PDF file D, obtaining text files and charts / images; Image analysis: Based on each chart image and its title, generate detailed semantic descriptions using multimodal LLM; Hierarchical construction: By recognizing and processing text files, we obtain an initial set of titles, the paper title, the reconstructed set of chapter titles, the set of chapter content, and the set of fragments; Semantic summarization generation: For each segment of content in each chapter of the chapter content set, based on the paper title, the hierarchical path of the chapter to which the segment belongs, the content of the segment, and the summary of the previous segment, a semantic summary of the corresponding segment is generated using multimodal LLM. Image-text association mapping: Traverse all segments in all chapters, identify chart references, and insert the corresponding chart title and its semantic description at the end of the segment to obtain the image-text fusion index.

[0007] Furthermore, the hierarchical structure is specifically as follows: Chapter title extraction: Identify title lines in a text file that begin with the "#" symbol using a regular expression matching algorithm to obtain an initial title set; Hierarchical Restructuring: Multimodal LLM analysis is used to analyze the logical and semantic relationships between headings in the initial heading set, reconstruct the nested structure, and obtain the paper title and the reconstructed set of chapter headings; Segmentation by Chapter: Based on the rows of chapter titles reconstructed from the chapter title set, the chapter content is segmented to obtain a set of chapter content; Sentence-based segmentation: The content of each chapter in the chapter content set is divided into continuous segments according to a fixed number of sentences using a sentence segmentation method, resulting in a set of segments.

[0008] Furthermore, the chapter-based segmentation is specifically as follows: A regular expression matching algorithm is used to locate the specific line number position of each chapter title in the text file; the line numbers corresponding to each chapter title are arranged in order of appearance to form an ordered sequence of segmentation markers; the text lines between two adjacent segmentation markers are taken as the chapter content corresponding to the previous chapter title; for the last chapter title, all text from its corresponding line number to the end of the file is taken as its chapter content; thus, a set of chapter content is obtained.

[0009] Furthermore, existing methods suffer from a lack of logical flow between text and images and arbitrary image insertion due to their singular information organization dimensions and lack of thematic guidance in image retrieval. This invention retrieves highly relevant text fragments and associated images from four semantic dimensions: research background, methodology, experimental results, and conclusion analysis. This forms a structured, thematically clear multimodal content set, ensuring the semantic consistency and logical rationality of text and image retrieval. Step three specifically involves: Dimension definition: Define the dimensions used for retrieval, including research motivation, research contribution, methods, and experiments; Relevant chapter acquisition: For each dimension, use multimodal LLM to identify the most relevant chapter titles in the chapter title set; extract all fragments and summaries from the most relevant chapter titles in the image-text fusion index to obtain the content most relevant to the target dimension; Search content set acquisition: Preset standard questions for each dimension, calculate the similarity between the preset standard questions and all fragments and summaries to obtain search results, merge and deduplicate the search results to obtain the final search content set for all dimensions.

[0010] Furthermore, the specific steps for obtaining the search content set are as follows: Pre-set standard question set: Pre-set standard questions for each dimension, each question describing the core information requirement of that dimension in a different way, resulting in a standard question set for that dimension; Pre-set standard questions with multiple expressions for each dimension to transform abstract semantic dimensions into concrete, computable query intents, improving the semantic coverage and matching accuracy of retrieval; Through question-and-answer semantic matching, enhance the recall and identification of key content, avoid omissions due to differences in expression, thereby achieving comprehensive and accurate extraction of multi-dimensional information, supporting the logic and completeness of subsequent image-text fusion summaries.

[0011] Semantic relevance calculation: For each standard question, the similarity between each standard question and all fragments and summaries is calculated using a vector similarity algorithm. The top K most relevant fragments and summaries are selected, merged, and deduplicated to obtain the search results. Merging search results: The search results for all questions under the same dimension are combined and deduplicated to obtain the final search content set for that dimension; the final search content sets for each dimension are combined to obtain the final search content set for all dimensions.

[0012] Furthermore, existing technologies suffer from problems such as simply splicing text and images, lacking a collaborative generation mechanism, and failing to provide linguistic references to images in the text. To address this issue, this invention employs joint modeling of text and images during the generation process to automatically determine the timing of image insertion in the output summary and generate natural language references such as "as shown in Figure X," forming a semantically coherent, logically clear, and integrated text-image summary, significantly improving information delivery efficiency and user comprehension experience. Step four specifically involves: Summary text generation: Input the prompt template and the final search content set of all dimensions into the multimodal LLM to obtain summary text with image placeholders; Image placeholder replacement: By identifying image placeholders in the summary text, the corresponding image file links are retrieved from the text file using a regular expression matching algorithm based on the image placeholders. The corresponding image file links are then used to replace the image placeholders, resulting in the final image-text fusion index summary.

[0013] The advantages of this invention are: This invention strengthens the connection between images and text and enhances the coherence of chapter context by constructing an image-text fusion index, achieving deep semantic coupling between images and text and improving the image-text collaborative expression capability of the abstract. By designing a structured dimensional template and a content retrieval mechanism driven by standard questions, it ensures the structured coverage of key information, significantly improving the logical coherence and readability of the abstract. The method of this invention breaks through the technical limitations of traditional single-modal or simple concatenation of abstracts, providing efficient support for academic search engines, intelligent literature recommendation systems, and scientific research auxiliary tools, and significantly improving the efficiency of researchers in obtaining core information from papers. Attached Figure Description

[0014] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0015] Figure 1 This is a flowchart of the steps of the method of the present invention; Figure 2 This is the file directory parsed by the method of this invention; Figure 3 The flowchart for constructing the image-text fusion index for this invention is shown. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1 In this embodiment, as Figure 1 As shown, this invention provides a method for generating academic paper abstracts based on image-text fusion indexing, the specific steps of which include: S1. Obtain academic paper data from publicly available data sources; In one embodiment, papers from the ArXiv website or other open-source academic conference and journal websites are used. This invention downloads the academic papers for which fused abstracts need to be generated in PDF format as input data for subsequent processing.

[0018] S2. Based on the academic papers in the academic paper data, logically segment the paper content and establish the association between fragments and images to construct an image-text fusion index, such as... Figure 3 As shown, this invention divides the content of a paper into semantically coherent logical segments and establishes a structured association between images and specific text segments, providing a traceable and locatable indexing foundation for deep image-text fusion. Specifically: S21. Document parsing: Use the PDF document parsing tool MinerU to parse the academic paper PDF file D, thereby obtaining the Markdown format text file and all the charts saved as images; In one embodiment, an academic paper is randomly selected from the data downloaded in step S1. The paper's title is: Time-Series Representation Learning via Dual Reference Contrasting. MinerU is then used to parse the paper's PDF file. The parsing command is: `magic-pdf -p . / TA.pdf -o output_dir -mauto`, where ". / TA.pdf" is the path and name of the PDF file, "output_dir" is the folder to save the parsing results, and the rest are default parameters. After executing the above parsing command, seven files and one folder are generated in the `output_dir` folder. The seven files contain a Markdown-formatted text file (TA.md) required for subsequent operations of this invention. The folder named "images" stores all charts saved as images, such as... Figure 2 As shown.

[0019] S22, Image Analysis: Based on each chart image and its title Using multimodal LLM to generate detailed semantic descriptions ; In one embodiment, multimodal LLM is used to generate a detailed semantic description from the chart title and chart content obtained in step S21.

[0020] S23, Hierarchical Construction: S231, Chapter Title Extraction: Using a regular expression matching algorithm, identify heading lines in Markdown formatted text files that begin with the "#" symbol to obtain an initial heading set. ,in For document title, This indicates the title of the nth section; In one embodiment, the execution code for the regular expression matching algorithm to identify header lines in the Markdown-formatted text file TA.md that begin with the "#" symbol is: with open('. / TA / TA.md', 'r', encoding='utf-8') as f: md_content = f.read() md_content.split('\n') heading_lines = [(i, line.strip()) for i, line in enumerate(lines) ifline.lstrip().startswith('#')] Where “. / TA / TA.md” is the path and name of the Markdown file, and “heading_lines” are all the identified header lines; the header lines extracted using the above code are: # Time-Series Representation Learning via Dual Reference Contrasting, # Authors, # Abstract, # CCS Concepts, # Keywords, # ACM Reference Format, # 1 Introduction, # 2 Related Work, # 2.1 Data Augmentation of Time Series, # 2.2 Contrastive learning for time series, ## 3 Method, # 3.1 Problem Definition, # 3.2 Model Architecture, # 3.3 Generating and Encoding Dual References, # 3.4 Dual Reference Contrastive Learning, # 3.5 Few-shot Fine-tuning, # 4 Experiments, # 4.1 Datasets, # 4.2 Experimental Settings, # 4.3 Experimental Results, # 5 Conclusions, # Acknowledgments, # References, where # Time-Series Representation Learning via Dual Reference Contrasting is the document title. .

[0021] S232. Hierarchical Reconstruction: Using multimodal LLM analysis, the logical and semantic relationships between headings in the initial heading set are reconstructed to reconstruct the nested structure and obtain the paper titles. and chapter title set ,in, This represents the title of the m-th reconstructed chapter; In one embodiment, the header row content obtained in step S231 is passed to the multimodal LLM, which is required to analyze the logical and semantic relationships between headers in the initial header set, reconstruct the nested structure, and obtain the reconstructed headers as follows: # Time-Series Representation Learning via Dual Reference Contrasting, ## Authors, ## Abstract, ## CCS Concepts, ## Keywords, ## ACM Reference Format, ## 1 Introduction, ## 2 Related Work, ### 2.1 Data Augmentation of Time Series, ### 2.2 Contrastive learning for time series, ## 3 Method, ### 3.1 Problem Definition, ### 3.2 Model Architecture, ### 3.3 Generating and Encoding Dual References, ### 3.4 Dual Reference Contrastive Learning, ### 3.5 Few-shot Fine-tuning, ## 4 Experiments, ### 4.1 Datasets, ### 4.2 Experimental Settings, ###4.3 Experimental Results, ##5 Conclusions, ## Acknowledgments, ## References; where # Time-Series Representation Learning via Dual Reference Contrasting is the paper title. .

[0022] S233. Chapter Segmentation: Based on the lines containing the reconstructed chapter titles, a regular expression matching algorithm is used to locate the specific line number of each chapter title in the text file; the line numbers corresponding to each chapter title are arranged in order of appearance, forming an ordered sequence of segmentation markers; the text lines between two adjacent segmentation markers are taken as the chapter content corresponding to the previous chapter title; for the last chapter title, all text from its corresponding line number to the end of the file is taken as its chapter content; thus, the entire text file content is divided into several continuous text segments, resulting in a set of chapter content. ,in, Indicates chapter title and The content between, Indicates chapter title and The content between them, and so on, Indicates chapter title Content up to the end of the text file; S234. Segment by sentence: Divide the content of each chapter... The sentence segmentation method is used to divide the sentence into segments according to a fixed number of sentences. A set of segments is obtained by analyzing consecutive segments. , ,in, Represents the segmented first... A segment, This represents the number of segments.

[0023] S24. Semantic summarization generation: For each segment Based on the paper title The hierarchical path of the chapter to which the segment belongs, the content of the segment, and a summary of the previous segment are used to generate a semantic summary containing contextual information using multimodal LLM. The formula is as follows: , in, This indicates an LLM operation. Indicates the title of the paper. Indicates from chapter To the title of the paper The path, This is the original text of the fragment. This is a summary of the previous segment, if This indicates that the segment is the first segment of the corresponding chapter. An empty string.

[0024] S25. Image-Text Association Mapping: Traverse all segments in all chapters, identify chart references such as "Fig. 1" or "Table 1", and map the corresponding chart titles. and its semantic description Insert it at the end of the segment to create a combined text and image index.

[0025] S3. This invention retrieves highly relevant text fragments and associated images from four semantic dimensions: research background, methodology, experimental results, and conclusion analysis. This forms a structured, thematically clear multimodal content set, ensuring semantic consistency and logical rationality in text and image calls. Semantic retrieval is performed on four key dimensions of academic papers to obtain optimal relevant content. Specifically: S31. Dimension Definition: Define four core dimensions most relevant to academic papers as dimensions for retrieval, including research motivation. Research contributions ,method and experiment , , This represents the set of all retrieved dimensions.

[0026] S32. Relevant Chapter Acquisition: For each dimension, use multimodal LLM to identify the chapter title set. The most relevant chapter titles in this dimension The formula is as follows: , in, This indicates an LLM-based similarity recognition operation. This indicates an operation to retrieve the most relevant chapter titles; Subsequently, all fragments and summaries in the most relevant chapter titles are extracted from the image-text fusion index to obtain the content most relevant to the target dimension.

[0027] S33. Retrieval of content sets: S331, Preset Standard Question Set: For each dimension, i.e. We predefine standard questions for this dimension, and each question describes the core information requirement of that dimension in a different way, thus obtaining a set of standard questions for that dimension. ,in, Indicates the first The first dimension One question; In one embodiment, the four dimensions in S31, namely research motivation, are addressed. Research contributions ,method and experiment The standard questions preset for each dimension are as follows: Research Motivation: What scientific or technological problem does this paper aim to solve? What is the purpose of this research? What shortcomings of existing methods does this paper address? Research Contributions: What new perspectives, models, or findings does this paper present? What are the main innovative points of this study? What are the contributions of this paper? Methodology: What is the workflow of the method presented in this paper? What techniques, models, or algorithms are used or proposed in this paper? How are the key steps in the method presented in this paper implemented? Experiments: What datasets and evaluation metrics were used in the experimental section of this paper? What are the main experimental results of this paper? How does the performance of our method compare to baseline methods? The semantic dimensions of academic papers (such as "research motivation" and "research contribution") are themselves high-level abstract concepts, making them difficult to use directly for precise retrieval. By designing a set of standard questions for each dimension (such as "What is the purpose of this study?" and "What shortcomings of existing methods does this paper address?"), these abstract dimensions can be concretized into clear information needs, making the retrieval targets clearer and more operational.

[0028] Pre-setting multiple standard questions for each dimension is intended to cover a wider range of language variations, avoid missing key content due to differences in expression, and improve the recall rate of the search. For example, in this embodiment, the questions on research motivation, such as "the purpose of conducting the research" and "the shortcomings to be addressed," both fall under the category of research motivation. However, by using different questions for retrieval, it can be ensured that the retrieved information is more comprehensive.

[0029] S332. Semantic Relevance Calculation: For each standard problem Using a vector similarity algorithm based on XLM-RoBERTa, the similarity between each standard question and all fragments and their summaries is calculated. The top K most relevant fragments and summaries are selected, merged, and deduplicated to form the search results. The formula is as follows: , in, Represents the set of all fragments; This refers to any segment among all segments; This indicates that semantic similarity is calculated based on vectors. This indicates selecting the top two with the highest similarity. Each segment and summary is processed and deduplicated. S333. Merging Search Results: The search results for all questions under the same dimension are combined and deduplicated to obtain the final set of search content for that dimension. , in, This represents the final set of retrieved content within the same dimension; Subsequently, the final search content set across all dimensions was retrieved. Perform a union operation to obtain the final set of retrieved content across all dimensions: , in, This represents the final set of retrieved content across all dimensions.

[0030] S4. This invention, through joint modeling of text and images during the generation process, automatically determines the timing of image insertion in the output summary and generates natural language citations such as "as shown in Figure X," forming a semantically coherent, logically clear, and integrated text-image summary, significantly improving information delivery efficiency and user comprehension experience; it generates integrated text-image summaries based on a four-dimensional search content set. Specifically: S41. Summary Text Generation: This generates the final search content set across all dimensions. The input to the multimodal LLM is required to generate summary text that includes image placeholders. ; S42. Image placeholder replacement: By recognizing summary text Image placeholders in the text are used to retrieve corresponding image file links from the original paper using a regular expression matching algorithm. These image placeholders are then replaced to generate the final image-text fusion abstract of the image-text fusion index. The formula is as follows: in, This indicates a placeholder replacement operation; This represents an image placeholder; This is the actual link to the corresponding image.

[0031] Example 2 To better demonstrate the technical effectiveness of this invention, this embodiment applies the method of this invention to the task of generating summaries through graph-text fusion in academic papers, and conducts experimental verification on three authoritative academic datasets: QASPER, QASA, and M3SciQA. Table 1 shows the performance comparison results between this invention and baseline methods. As can be seen from Table 1, this invention significantly outperforms advanced baseline methods such as Graph-based Retrieval-Augmented Generation (GraphRAG) and GPT-4o-mini (a lightweight version of GPT-4o) in key indicators such as semantic alignment, logical citation, information integrity, cross-modal coherence, and overall quality. On the QASPER dataset, this invention achieved a high win rate in the "logical citation" metric, far exceeding GraphRAG and GPT-4o-mini; it also performed excellently in "cross-modal coherence," significantly higher than the baseline method (GraphRAG). This indicates that by constructing an image-text fusion index and a four-dimensional semantic retrieval mechanism, this invention effectively achieved semantic alignment and logical binding between images and text, solving the problems of arbitrary image insertion and lack of natural language reference in existing technologies, thus achieving a deep fusion of reasonable image citation placement and coherent text expression. On the QASA and M3SciQA datasets, this invention demonstrated outstanding win rates in the "information completeness" metric, significantly higher than GraphRAG. This indicates that by pre-setting a standard question set encompassing four dimensions—research motivation, research contribution, methods, and experiments—this invention can systematically retrieve and organize the core content of the paper, comprehensively covering the key elements of academic discourse, and avoiding the incomplete abstract problem caused by information fragmentation in traditional methods. Considering the results of the three datasets, this invention significantly outperformed GraphRAG in terms of average win rate in "overall quality," while GraphRAG performed relatively poorly on average, and GPT-4o-mini also performed at a low level. This demonstrates that the present invention, through its end-to-end design of "logical segmentation - image and text indexing - multi-dimensional retrieval - collaborative generation," not only improves the information integrity and image-text consistency of the summary but also significantly enhances users' understanding efficiency and trust in the summary content. In particular, the standard question-driven semantic retrieval mechanism provides clear thematic basis and contextual support for image retrieval, achieving a fundamental leap from "image and text splicing" to "image and text fusion." This effectively overcomes key shortcomings in existing technologies, such as image-text disconnect and heavy comprehension burden, resulting in more professional, readable, and practical summaries.

[0032] Table 1. Performance comparison of the present invention and baseline methods on three academic datasets. Table 1 illustrates the performance comparison between the present invention and baseline methods on three academic datasets: QASPER, QASA, and M3SciQA. All values ​​represent the win rate (%) of the present invention relative to baseline methods (GraphRAG, GPT-4o-mini) on each metric. Higher values ​​indicate that the image-text fusion summary generated by the present invention outperforms the baseline methods in the corresponding dimension. The metrics are explained below: Semantic alignment: This measures the degree of semantic matching between images and text descriptions in a summary. This invention significantly outperforms others in this metric, demonstrating that by constructing an image-text fusion index, it effectively achieves fine-grained semantic alignment between images and text fragments, solving the problems of image isolation and image-text disconnect in existing technologies.

[0033] Logical referencing: This measures whether the text generates natural language references such as "as shown in Figure X" at appropriate locations. This invention excels in this metric, indicating that its generation process can accurately pinpoint the timing of image insertion and achieve logical linkage between text and images.

[0034] Information completeness: This measures whether the abstract comprehensively covers the four core dimensions of research motivation, methods, experiments, and conclusions. This invention scored the highest, verifying that the four-dimensional retrieval mechanism can systematically extract key content and avoid information omissions.

[0035] Cross-modal coherence: Measures whether the overall text and image are semantically coherent without requiring user inference. This invention significantly outperforms the baseline, demonstrating that its generated summaries are more readable and easier to understand.

[0036] Overall Quality: The overall usability and professionalism of the summaries are comprehensively evaluated. The overall win rate of this invention on all datasets is significantly higher than GraphRAG and GPT-4o-mini, fully demonstrating its significant technical advantages in academic image and text summarization tasks.

[0037] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating academic paper abstracts based on image-text fusion indexing, characterized in that, Includes the following steps: Obtain academic paper data from publicly available data sources; Based on academic papers in the academic paper data, the paper content is logically segmented and the association between fragments and images is established to construct an image-text fusion index; including: Image analysis: Based on each chart image and its title, generate detailed semantic descriptions using multimodal LLM; Semantic summarization generation: For each segment of content in each chapter of the chapter content set, based on the paper title, the hierarchical path of the chapter to which the segment belongs, the content of the segment, and the summary of the previous segment, a semantic summary of the corresponding segment is generated using multimodal LLM. Image-text association mapping: Traverse all segments in all chapters, identify chart references, and insert the corresponding chart title and its semantic description at the end of the segment to obtain the image-text fusion index; Semantic retrieval is performed on four dimensions of academic papers to obtain the final retrieval content set for all dimensions; the specific process is as follows: Dimension definition: Define the dimensions used for retrieval, including research motivation, research contribution, methods, and experiments; Relevant chapter acquisition: For each dimension, use multimodal LLM to identify the most relevant chapter titles in the chapter title set; extract all fragments and semantic summaries from the most relevant chapter titles in the image-text fusion index to obtain the content most relevant to the target dimension; Search content set acquisition: Preset standard questions for each dimension, calculate the similarity between the preset standard questions and all fragments and semantic summaries to obtain search results, merge and deduplicate the search results to obtain the final search content set for all dimensions; The process of generating a fusion summary of images and text based on the search content set from four dimensions is as follows: Summary text generation: Input the final retrieval content set of all dimensions into the multimodal LLM to obtain summary text with image placeholders; Image placeholder replacement: By identifying image placeholders in the summary text, the corresponding image file links are retrieved from the text file using a regular expression matching algorithm based on the image placeholders. The corresponding image file links are then used to replace the image placeholders, resulting in the final image-text fusion index summary.

2. The method for generating academic paper abstracts based on image-text fusion indexing according to claim 1, characterized in that, Based on academic papers in the academic paper data, the paper content is logically segmented and the association between fragments and images is established to construct an image-text fusion index; it also includes: Document parsing: Using PDF document parsing tools to parse academic paper PDF files to obtain text files and charts / images; Hierarchical construction: By recognizing and processing text files, we obtain an initial set of titles, the paper title, the reconstructed set of chapter titles, the set of chapter content, and the set of fragments.

3. The method for generating academic paper abstracts based on image-text fusion indexing according to claim 2, characterized in that, By recognizing and processing the text file, we obtain the initial title set, the paper title, the reconstructed chapter title set, the chapter content set, and the fragment set: Chapter title extraction: Identify title lines in a text file that begin with the "#" symbol using a regular expression matching algorithm to obtain an initial title set; Hierarchical Restructuring: Multimodal LLM analysis is used to analyze the logical and semantic relationships between headings in the initial heading set, reconstruct the nested structure, and obtain the paper title and the reconstructed set of chapter headings; Segmentation by Chapter: Based on the rows of chapter titles reconstructed from the chapter title set, the chapter content is segmented to obtain a set of chapter content; Sentence-based segmentation: For each chapter in the content set, the sentence segmentation method is used to divide it into continuous segments according to a fixed number of sentences, resulting in a set of segments.

4. The method for generating academic paper abstracts based on image-text fusion indexing according to claim 3, characterized in that, Based on the rows containing the reconstructed chapter titles from the chapter title set, the chapter content is segmented to obtain the chapter content set: The regular expression matching algorithm is used to locate the specific line number position of each chapter title in the text file; the line numbers corresponding to each chapter title are arranged in order of appearance to form an ordered sequence of segmentation markers; the text lines between two adjacent segmentation markers are taken as the chapter content corresponding to the previous chapter title; for the last chapter title, all text from its corresponding line number to the end of the file is taken as its chapter content; thus, the chapter content set is obtained.

5. The method for generating academic paper abstracts based on image-text fusion indexing according to claim 4, characterized in that, For each dimension, a predefined standard question is set. The search results are obtained by calculating the similarity between the predefined standard question and all fragments and semantic summaries. The search results are then merged and deduplicated to obtain the final search content set for all dimensions. Pre-set standard question set: Pre-set standard questions for each dimension. Each question describes the core information requirements of that dimension in a different way, thus obtaining the standard question set for that dimension. Semantic relevance calculation: For each standard question, the similarity between each standard question and all fragments and semantic summaries is calculated using a vector similarity algorithm. The top K most relevant fragments and semantic summaries are selected, merged, and deduplicated to obtain the search results. Merging search results: The search results for all questions under the same dimension are combined and deduplicated to obtain the final search content set for that dimension; the final search content sets for each dimension are combined to obtain the final search content set for all dimensions.

Images