Paper sorting method and system based on multi-modal information, electronic equipment and medium

Abstract

The invention discloses a paper sorting method and system based on multi-modal information, electronic equipment and a medium, and the method comprises the steps: splitting a to-be-processed paper PDF document according to pages, and storing each page as image data; processing the image data corresponding to each page to obtain each layout area category and corresponding bounding box coordinates and character fields; carrying out normalization and offset processing on the bounding box coordinates in each page, and sorting the bounding box coordinates after normalization and offset processing according to a sequence from top to bottom and from left to right; setting invalid layout area categories, and deleting the invalid layout area categories to obtain a full-text rough sorting result; and setting a cue word template, filling the cue word template with the full-text coarse sorting result, and inputting the full-text coarse sorting result into the large language model to obtain a full-text fine sorting result.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a paper sorting method, electronic equipment, and medium based on multimodal information. Background Art

[0002] PDF is the main circulation format for scientific papers. Currently, a scientific paper in PDF format containing multiple pages can be parsed into several paragraphs through optical character recognition (OCR) technology.

[0003] Existing methods for sorting paper content infer the reading order by comparing the spatial position relationship of each paragraph. For example, the paragraphs are arranged from top to bottom and from left to right. Due to the high complexity of the layout structure of scientific papers, the sorting schemes based on spatial position relationships generally have the problem of poor universality, so there is currently no technical solution that can fully automatically extract the content of scientific papers in PDF format and display the content in reading order.

[0004] Therefore, it is a technical problem to be solved urgently to propose a parsing technology for scientific papers in PDF format so as to sort the text paragraphs into the reading order of the paper. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a paper sorting method, electronic device, and medium based on multimodal information.

[0006] In a first aspect, an embodiment of the present invention provides a method for ranking papers based on multimodal information, the method comprising:

[0007] Split the paper PDF document to be processed into pages and save each page as image data; process the image data corresponding to each page to obtain each layout area category and the corresponding bounding box coordinates and text fields;

[0008] Normalize and offset the bounding box coordinates in each page, and sort the normalized and offset bounding box coordinates in order from top to bottom and from left to right; set invalid layout area categories and delete them to obtain a rough sorting result of the entire text;

[0009] Set a prompt word template, fill the prompt word template with the rough sorting result of the full text, and then input it into the large language model to obtain the fine sorting result of the full text.

[0010] Furthermore, the process of processing the image data corresponding to each page to obtain each layout area category and the corresponding bounding box coordinates and text fields includes:

[0011] Input the image data corresponding to each page into the pre-trained recognition model, and output the page number, layout area category and the corresponding bounding box coordinates;

[0012] Processing the image data corresponding to each layout area category to obtain a line bounding box for each line of text in the layout area;

[0013] The text recognition is performed row by row on the area corresponding to each layout area category, so as to obtain each layout area category and the corresponding bounding box coordinates and text field.

[0014] Furthermore, performing text recognition on a line-by-line basis on an area corresponding to each layout area category includes:

[0015] When the area corresponding to a certain layout area category contains multiple lines of text, the recognition results of each line of text are concatenated in sequence to obtain the text field corresponding to the layout area category.

[0016] Furthermore, the layout area categories include: chapter title, body text, header, footer, page number, figure, figure title, table, table title, and formula.

[0017] Furthermore, the process of normalizing and offsetting the bounding box coordinates in each page includes:

[0018] Normalize and offset the bounding box coordinates in each page so that the bounding box coordinates of different pages are unified in the same coordinate system. The expression is as follows:

[0019] x = x / page_width;

[0020] y=y / page_height+page_number;

[0021] Among them, page_width is the page width, page_height is the page height, page_number is the page number, page_number = 0, 1, 2..., N, N is the total number of pages.

[0022] Furthermore, a prompt word template is set, and the rough sorting result of the full text is filled into the prompt word template and then input into the large language model. The process of obtaining the fine sorting result of the full text includes:

[0023] Extract the text fields in the rough sorting results of the full text, and splice them in order to obtain the rough sorting articles;

[0024] Set the prompt word template;

[0025] The roughly ranked articles are filled with the prompt word template and then input into the large language model to obtain the refined ranking results of the full text.

[0026] In a second aspect, an embodiment of the present invention provides a paper ranking system based on multimodal information, which is used to implement the above-mentioned paper ranking method based on multimodal information, and the system includes:

[0027] The layout analysis and text recognition module is used to split the PDF document to be processed into pages and save each page as image data; process the image data corresponding to each page to obtain each layout area category and the corresponding bounding box coordinates and text fields;

[0028] The coordinate sorting module is used to normalize and offset the bounding box coordinates in each page, and sort the bounding box coordinates after normalization and offset processing from top to bottom and from left to right; set the invalid layout area category and delete it to obtain the rough sorting result of the whole text;

[0029] The semantic sorting module is used to set a prompt word template, fill the prompt word template with the rough sorting result of the full text, and then input it into the large language model to obtain the precise sorting result of the full text.

[0030] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned paper sorting method based on multimodal information.

[0031] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for sorting papers based on multimodal information.

[0032] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instructions, which, when executed by a processor, implements the above-mentioned paper ranking method based on multimodal information.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention provides a method for sorting papers based on multimodal information, which combines computer vision and a large language model, utilizes multiple modal information such as images and texts, splits the paper PDF document to be processed by page, and saves each page as image data; processes the image data corresponding to each page, obtains each layout area category and the corresponding bounding box coordinates and text fields; sorts the bounding box coordinates in a top-to-bottom and left-to-right order; deletes invalid layout area categories, obtains a rough sorting result of the full text; sets a prompt word template, fills the rough sorting result of the full text into the prompt word template, and then inputs it into the large language model to obtain a precise sorting result of the full text. The present invention realizes high-precision automatic analysis of scientific literature, which is of great significance to improving the research efficiency of scientific researchers. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0036] Figure 1 A schematic diagram of a paper ranking method based on multimodal information provided by an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the flow of layout analysis and text recognition provided by an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of bounding box coordinates provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of semantic sorting provided by an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0042] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0043] like Figure 1 As shown, an embodiment of the present invention provides a method for sorting papers based on multimodal information, and the method comprises the following steps:

[0044] Step S1, split the paper PDF document to be processed into pages, and save each page as image data; process the image data corresponding to each page to obtain each layout area category and the corresponding bounding box coordinates and text fields.

[0045] like Figure 2 As shown, the step S1 specifically includes the following steps:

[0046] Step S101, split the paper PDF document to be processed into pages, and save each page in the form of image data.

[0047] Step S102, inputting the image data corresponding to each page into a pre-trained recognition model, and outputting the page number, layout area category and corresponding bounding box coordinates.

[0048] Furthermore, the layout area categories include: chapter title, body text, header, footer, page number, figure, figure title, table, table title, formula. Figure 3 As shown, the coordinates of the bounding box are marked as: [x0, y0, x1, y1], where (x0, y0) represents the coordinates of the upper left corner of the bounding box, and (x1, y1) represents the coordinates of the lower right corner of the bounding box.

[0049] Furthermore, the recognition results output by the recognition model are as follows: [

[0051] {

[0052] 'Page number': positive integer,

[0053] 'Category': layout area category name,

[0054] 'Bounding box coordinates': bounding box coordinate values,

[0055] },

[0056] … ]

[0058] The recognition model may be a document understanding model (LayoutLM), a document converter model (DocFormer) or a document understanding transformation model (Donut, OCR-free Document Understanding Transformer).

[0059] In this example, the document understanding model (LayoutLM) is selected, and the PublayNet dataset is used to annotate the PublayNet dataset according to the layout area categories including chapter titles, body text, headers, footers, page numbers, figures, figure titles, tables, table titles, and formulas. The LayoutLM model is trained using the annotated PublayNet dataset through the mean squared error loss (MSE). During the inference process, the image data to be processed is input into the pre-trained LayoutLM model to obtain the page number, layout area category, and the corresponding bounding box coordinates.

[0060] Step S103, processing the image data corresponding to each layout area category to obtain a line bounding box for each line of text in the layout area, and obtaining the following recognition results: [

[0062] {

[0063] 'Page number': positive integer,

[0064] 'Category': layout area category name,

[0065] 'Bounding box coordinates': bounding box coordinate values,

[0066] 'Row bounding box coordinates': row bounding box coordinate values,

[0067] },

[0068] … ]

[0070] Furthermore, in this example, the image data corresponding to each layout area category is processed by edge detection-based methods, connected component-based methods, pixel projection-based methods, and Hough transform-based methods to obtain a line bounding box for each line of text in the layout area. Exemplarily, the image corresponding to the layout area category of body text (i.e., a certain paragraph) is processed by an edge detection algorithm to obtain a bounding box for each line of text in the area.

[0071] Step S104, perform text recognition on the area corresponding to the layout area category line by line; if the area corresponding to a layout area category contains multiple lines of text, the recognition results of each line of text are sequentially spliced ​​to obtain the text field corresponding to the layout area category. Specifically, the recognition results are as follows: [

[0073] {

[0074] 'Page number': positive integer,

[0075] 'Category': layout area category name,

[0076] 'Bounding box coordinates': bounding box coordinate values,

[0077] 'Text': text recognition result in the bounding box

[0078] },

[0079] … ]

[0081] Furthermore, in this example, the text recognition algorithm may be Connectionist Temporal Classification, Sequence-to-Sequence Models, Convolutional Recurrent Neural Network or Spatial Transformer Networks.

[0082] Step S2, normalize and offset the bounding box coordinates in each page, and sort the normalized and offset bounding box coordinates in order from top to bottom and from left to right; set invalid layout area categories and delete them to obtain a rough sorting result of the entire text.

[0083] Furthermore, the bounding box coordinates in each page are normalized and offset so that the bounding box coordinates of different pages are unified in the same coordinate system. The conversion formula is:

[0084] x=x / page_width

[0085] y=y / page_height+page_number

[0086] Among them, page_width is the page width, page_height is the page height, page_number is the page number, page_number = 0, 1, 2..., N, N is the total number of pages.

[0087] Step S3, setting a prompt word template, filling the prompt word template with the rough sorting result of the full text, and then inputting it into the large language model to obtain the precise sorting result of the full text.

[0088] Specifically, Figure 4 As shown, the text fields in the rough sorting results of the full text are extracted and spliced ​​in order to obtain the rough sorting articles; the spliced ​​rough sorting articles are filled with the prompt word template and then input into the large language model.

[0089] In this example, a large language model that can support an input length of 128K tokens is used, such as Llama3.1, Claude3, Qwen2, Mistral, etc.

[0090] Example 1

[0091] The following takes a certain document as an example to illustrate the specific implementation process of the method of the present invention.

[0092] Step S1, split the paper PDF document to be processed into pages, and save each page as image data; process the image data corresponding to each page to obtain each layout area category and the corresponding bounding box coordinates and text fields.

[0093] Specifically, the steps include:

[0094] Step S101, split the paper PDF document to be processed into pages, and save each page in the form of image data.

[0095] Step S102, inputting the image data corresponding to each page into a pre-trained recognition model, and outputting the page number, layout area category and corresponding bounding box coordinates.

[0096] Specifically, the recognition model selects a document understanding model (LayoutLM), and inputs each page image into the previously trained LayoutLM.

[0097] Among them, the recognition results of page 0 are as follows: [

[0099] {'Page Number': 0, 'Category': 'Chapter Title', 'Bounding Box Coordinates': [x0, y0, x1, y1]}

[0100] {'Page number': 0, 'Category': 'Page number', 'Bounding box coordinates': [x0, y0, x1, y1]}

[0101] {'Page number': 0, 'Category': 'Footer', 'Bounding box coordinates': [x0, y0, x1, y1]} ]

[0103] Furthermore, the recognition results of page 1 are as follows: [

[0105] {'Page number': 1, 'Category': 'Normal text', 'Bounding box coordinates': [x0, y0, x1, y1]}

[0106] {'Page number': 1, 'Category': 'Normal text', 'Bounding box coordinates': [x0, y0, x1, y1]} ]

[0108] Furthermore, the recognition result of page 2 is as follows: [

[0110] {'Page number': 2, 'Category': 'Table', 'Bounding box coordinates': [x0, y0, x1, y1]}

[0111] {'Page number': 2, 'Category': 'Table title', 'Bounding box coordinates': [x0, y0, x1, y1]}

[0112] {'Page number': 2, 'Category': 'Normal text', 'Bounding box coordinates': [x0, y0, x1, y1]} ]

[0114] Furthermore, the recognition results of page 3 are as follows: [

[0116] {'Page number': 3, 'Category': 'Image', 'Bounding box coordinates': [x0, y0, x1, y1]}

[0117] {'Page number': 3, 'Category': 'Normal text', 'Bounding box coordinates': [x0, y0, x1, y1]} ]

[0119] Step S103, processing the image data corresponding to each layout area category to obtain a line bounding box for each line of text in the layout area.

[0120] Among them, the recognition results of page 0 are as follows: [

[0122] {'Page Number': 0, 'Category': 'Chapter Title', 'Bounding Box Coordinates': [x0, y0, x1, y1], 'Row Bounding Box': [x0, y0, x1, y1], ...]}

[0123] {'Page number': 0, 'Category': 'Page number', 'Bounding box coordinates': [x0, y0, x1, y1], 'Row bounding box': [x0, y0, x1, y1], ...]}

[0124] {'Page Number': 0, 'Category': 'Footer', 'Bounding Box Coordinates': [x0, y0, x1, y1], 'Row Bounding Box': [x0, y0, x1, y1], ...]} ]

[0126] Furthermore, the recognition results of page 1 are as follows: [

[0128] {'Page number': 1, 'Category': 'Normal text', 'Bounding box coordinates': [x0, y0, x1, y1], 'Line bounding box': [x0, y0, x1, y1], ...]}

[0129] {'Page number': 1, 'Category': 'Normal text', 'Bounding box coordinates': [x0, y0, x1, y1], 'Line bounding box': [x0, y0, x1, y1], ...]} ]

[0131] Furthermore, the recognition result of page 2 is as follows: [

[0133] {'Page number': 2, 'Category': 'Table', 'Bounding box coordinates': [x0, y0, x1, y1], 'Row bounding box': [x0, y0, x1, y1], ...]}

[0134] {'Page Number': 2, 'Category': 'Table Title', 'Bounding Box Coordinates': [x0, y0, x1, y1], 'Row Bounding Box': [x0, y0, x1, y1], ...]}

[0135] {'Page number': 2, 'Category': 'Normal text', 'Bounding box coordinates': [x0, y0, x1, y1], 'Line bounding box': [x0, y0, x1, y1], ...]} ]

[0137] Furthermore, the recognition results of page 3 are as follows: [

[0139] {'Page Number': 3, 'Category': 'Image', 'Bounding Box Coordinates': [x0, y0, x1, y1], 'Row Bounding Box': [x0, y0, x1, y1], ...]}

[0140] {'Page number': 3, 'Category': 'Normal text', 'Bounding box coordinates': [x0, y0, x1, y1], 'Line bounding box': [x0, y0, x1, y1], ...]} ]

[0142] Step S104, performing text recognition on the area corresponding to the layout area category line by line; if the area corresponding to a certain layout area category contains multiple lines of text, the recognition results of each line of text are sequentially spliced ​​to obtain the text field corresponding to the layout area category.

[0143] The recognition results are as follows: [

[0145] {'Page Number': 0, 'Category': 'Chapter Title', 'Bounding Box Coordinates': [x0, y0, x1, y1], 'Text': 'ABC'}

[0146] {'Page Number': 0, 'Category': 'Page Number', 'Bounding Box Coordinates': [x0, y0, x1, y1], 'Text': '0'}

[0147] {'Page number': 0, 'Category': 'Footer', 'Bounding box coordinates': [x0, y0, x1, y1], 'Text': '0'}

[0148] {'Page Number': 1, 'Category': 'Normal Text', 'Bounding Box Coordinates': [x0, y0, x1, y1], 'Text': 'HIJK'}

[0149] {'Page number': 1, 'Category': 'Normal text', 'Bounding box coordinates': [x0, y0, x1, y1], 'Text': 'DEFG'}

[0150] {'Page Number': 2, 'Category': 'Table', 'Bounding Box Coordinates': [x0, y0, x1, y1], 'Text': 'LMN'}

[0151] {'Page Number': 2, 'Category': 'Table Title', 'Bounding Box Coordinates': [x0, y0, x1, y1], 'Text': 'OPQ'}

[0152] {'Page Number': 2, 'Category': 'Normal Text', 'Bounding Box Coordinates': [x0, y0, x1, y1], 'Text': 'RSTU'}

[0153] {'Page Number': 3, 'Category': 'Image', 'Bounding Box Coordinates': [x0, y0, x1, y1], 'Text': '()%￥%&'}

[0154] {'Page Number': 3, 'Category': 'Normal Text', 'Bounding Box Coordinates': [x0, y0, x1, y1], 'Text': 'VWXYZ'} ]

[0156] Step S2, normalize and offset the bounding box coordinates in each page, and sort the normalized and offset bounding box coordinates in order from top to bottom and from left to right; set invalid layout area categories and delete them to obtain a rough sorting result of the entire text.

[0157] Specifically, in this example, the invalid layout area categories are set to include: header, footer, page number, figure, figure title, formula. After deleting the invalid layout area, the rough sorting result of the full text is obtained as follows: [

[0159] {'Page Number': 0, 'Category': 'Chapter Title', 'Normalized Bounding Box Coordinates': [x0, y0, x1, y1], 'Text': 'ABC'}

[0160] {'Page number': 1, 'Category': 'Normal text', 'Normalized bounding box coordinates': [x0, y0, x1, y1], 'Text': 'HIJK'}

[0161] {'Page number': 1, 'Category': 'Normal text', 'Normalized bounding box coordinates': [x0, y0, x1, y1], 'Text': 'DEFG'}

[0162] {'Page Number': 2, 'Category': 'Table', 'Normalized Bounding Box Coordinates': [x0, y0, x1, y1], 'Text': 'LMN'}

[0163] {'Page Number': 2, 'Category': 'Table Title', 'Normalized Bounding Box Coordinates': [x0, y0, x1, y1], 'Text': 'OPQ'}

[0164] {'Page Number': 2, 'Category': 'Normal Text', 'Normalized Bounding Box Coordinates': [x0, y0, x1, y1], 'Text': 'RSTU'}

[0165] {'Page number': 3, 'Category': 'Normal text', 'Normalized bounding box coordinates': [x0, y0, x1, y1], 'Text': 'VWXYZ'} ]

[0167] Step S3, setting a prompt word template, filling the prompt word template with the rough sorting result of the full text, and then inputting it into the large language model to obtain the precise sorting result of the full text.

[0168] First, the text fields in the rough sorting result of the full text are extracted and concatenated in order. In this example, [ABCHIJKDEFGLMNOPQRSTUVWXYZ] is concatenated.

[0169] In this example, the prompt word template is set as:

[0170] “

[0171] You are a researcher with extensive experience in writing scientific papers. The following is the content of a scientific paper, including the title of the text, the content of the text, the title of the table, and the content of the table, but not including page numbers, images, image titles, and formulas. The order of the paragraphs in this scientific paper is wrong, and irrelevant content may be mixed in. Please reorder the content of this paper, correct the order errors, and delete irrelevant content. Note that you should not modify the content of the article, only sort and delete it.

[0172] The content of the article that needs to be modified is as follows:

[0173] [Full text rough sorting results]

[0174] ”

[0175] Furthermore, in this example, the large language model uses Llama3.1 which can support an input length of 128K tokens, and fills the rough sorting result of the full text into the prompt word template:

[0176] “

[0177] You are a researcher with extensive experience in writing scientific papers. The following is the content of a scientific paper, including the title of the text, the content of the text, the title of the table, and the content of the table, but not including page numbers, images, image titles, and formulas. The order of the paragraphs in this scientific paper is wrong, and irrelevant content may be mixed in. Please reorder the content of this paper, correct the order errors, and delete irrelevant content. Note that you should not modify the content of the article, only sort and delete it.

[0178] The content of the article that needs to be modified is as follows:

[0179] [ABCHIJKDEFGLMNOPQRSTUVWXYZ]

[0180] ”

[0181] Finally, the full-text precise ranking result output by the large language model is: ADCBEFGHIJKLMNOPQRSTUVWXYZ.

[0182] On the other hand, an embodiment of the present invention further provides a paper ranking system based on multimodal information, the system comprising:

[0183] The layout analysis and text recognition module is used to split the PDF document to be processed into pages and save each page as image data; process the image data corresponding to each page to obtain each layout area category and the corresponding bounding box coordinates and text fields;

[0184] The coordinate sorting module is used to normalize and offset the bounding box coordinates in each page, and sort the bounding box coordinates after normalization and offset processing from top to bottom and from left to right; set the invalid layout area category and delete it to obtain the rough sorting result of the whole text;

[0185] The semantic sorting module is used to set a prompt word template, fill the prompt word template with the rough sorting result of the full text, and then input it into the large language model to obtain the precise sorting result of the full text.

[0186] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0187] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. A person of ordinary skill in the art can understand and implement it without creative work.

[0188] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned paper ranking method based on multimodal information. Figure 5 As shown, it is a hardware structure diagram of any device with data processing capability in which the paper ranking method based on multimodal information provided by the embodiment of the present invention is located, except Figure 5 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0189] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the paper sorting method based on multimodal information as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0190] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only.

[0191] It will be appreciated that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A paper ranking method based on multimodal information, It is characterized in that The method comprises: Split the paper PDF document to be processed into pages and save each page as image data; process the image data corresponding to each page to obtain each layout area category and the corresponding bounding box coordinates and text fields; Normalize and offset the bounding box coordinates in each page, and sort the normalized and offset bounding box coordinates in order from top to bottom and from left to right; set invalid layout area categories and delete them to obtain a rough sorting result of the entire text; Set a prompt word template, fill the prompt word template with the rough sorting result of the full text, and then input it into the large language model to obtain the fine sorting result of the full text.

2. According to the method for ranking papers based on multimodal information according to claim 1, It is characterized in that The process of processing the image data corresponding to each page to obtain each layout area category and the corresponding bounding box coordinates and text fields includes: Input the image data corresponding to each page into the pre-trained recognition model, and output the page number, layout area category and the corresponding bounding box coordinates; Processing the image data corresponding to each layout area category to obtain a line bounding box for each line of text in the layout area; The text recognition is performed row by row on the area corresponding to each layout area category, so as to obtain each layout area category and the corresponding bounding box coordinates and text field.

3. According to the method for ranking papers based on multimodal information as described in claim 2, It is characterized in that The text recognition is performed on a line-by-line basis on the area corresponding to each layout area category, including: When the area corresponding to a certain layout area category contains multiple lines of text, the recognition results of each line of text are concatenated in sequence to obtain the text field corresponding to the layout area category.

4. A method for ranking papers based on multimodal information according to claim 1 or 2, It is characterized in that The layout area categories include: chapter title, body text, header, footer, page number, figure, figure title, table, table title, and formula.

5. According to the method for ranking papers based on multimodal information as claimed in claim 1, It is characterized in that The process of normalizing and offsetting the bounding box coordinates in each page includes: Normalize and offset the bounding box coordinates in each page so that the bounding box coordinates of different pages are unified in the same coordinate system. The expression is as follows: x = x / page_width; y=y / page_height+page_number; Among them, page_width is the page width, page_height is the page height, page_number is the page number, page_number = 0, 1, 2..., N, N is the total number of pages.

6. According to claim 1, a method for ranking papers based on multimodal information, It is characterized in that The process of setting a prompt word template, filling the rough sorting result of the full text into the prompt word template and then inputting it into the large language model to obtain the fine sorting result of the full text includes: Extract the text fields in the rough sorting results of the full text, and splice them in order to obtain the rough sorting articles; Set the prompt word template; The roughly ranked articles are filled with the prompt word template and then input into the large language model to obtain the refined ranking results of the full text.

7. A paper ranking system based on multimodal information, It is characterized in that For implementing the paper ranking method based on multimodal information as described in any one of claims 1 to 6 above, the system comprises: The layout analysis and text recognition module is used to split the PDF document to be processed into pages and save each page as image data; process the image data corresponding to each page to obtain each layout area category and the corresponding bounding box coordinates and text fields; The coordinate sorting module is used to normalize and offset the bounding box coordinates in each page, and sort the bounding box coordinates after normalization and offset processing from top to bottom and from left to right; set the invalid layout area category and delete it to obtain the rough sorting result of the whole text; The semantic sorting module is used to set a prompt word template, fill the prompt word template with the rough sorting result of the full text, and then input it into the large language model to obtain the precise sorting result of the full text.

8. An electronic device comprising a memory and a processor, It is characterized in that The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the paper sorting method based on multimodal information as described in any one of claims 1-6 above.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method for sorting papers based on multimodal information as described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, It is characterized in that When the computer program / instruction is executed by a processor, the method for sorting papers based on multimodal information described in any one of claims 1 to 6 is implemented.

Images