Processing method and device for mining patent information in combination with large language model

Abstract

The embodiment of the invention relates to a processing method and device for mining patent information in combination with a large language model, and the method comprises the steps: selecting the large language model as a first model, assigning five target tasks for the first model, constructing five task data sets, and carrying out the fine adjustment of the five task data sets; receiving a PDF file of the small molecule medicine patent after fine tuning is finished, and extracting an image, a text block and a table of the PDF file; molecular image / molecular label / irrelevant table screening and molecular activity table splicing are carried out based on the model; constructing a label-image corresponding relation table based on the molecular image / molecular label / spliced table information sequence and the first model, and constructing a label-activity information corresponding relation table based on the first model and the spliced table information sequence; combining the two corresponding relation tables to obtain a multi-modal comprehensive information table; and finally, forming corresponding mining report feedback by the patent file and the multi-mode comprehensive information table. According to the invention, the data mining efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a processing method and device for mining patent information in combination with a large language model. Background Art

[0002] In the development process of small molecule drugs, collecting and analyzing patent information is an essential step. The large amount of compound molecular structure, chemical name and biological activity data contained in patents are important sources of information for drug design and optimization. However, these data usually exist in non-standardized formats, scattered in the text, tables and images of patent documents, requiring researchers to collect information through manual operations. Obviously, this will inevitably reduce R&D efficiency. Summary of the invention

[0003] The purpose of the present invention is to provide a processing method, device, electronic device and computer-readable storage medium for mining patent information in combination with a large language model in view of the defects of the prior art. The present invention selects a type of pre-trained multimodal large language model as the first model, and specifies five target tasks for it (molecular image recognition of input images, molecular label recognition of input text, classification of input tables, similarity recognition of two input data rows, and molecular label column recognition of input tables), customizes five instruction text templates, constructs five data sets, and fine-tunes the first model for five types of tasks based on the five data sets and the corresponding five instruction text templates; after the model fine-tuning is completed, the PDF file of the small molecule drug patent input by the user is received, and the image, text block and table of the patent document are extracted based on the PDF content extraction tool; then, the first image information sequence is subjected to molecular image screening based on the first instruction template and the first model, and the first image information sequence is subjected to molecular image screening based on the first instruction template and the first model. The first text block information sequence is screened for molecular labels based on the second instruction template and the first model, the first table information sequence is filtered for other tables based on the third instruction template and the first model, and the molecular activity table and the headerless table are spliced ​​based on the fourth instruction template, the first model and the second table information sequence; then, a label-image correspondence table is constructed based on the first molecular image information sequence, the first molecular label information sequence, the first spliced ​​table information sequence, the fifth instruction template and the first model, and a label-activity information correspondence table is constructed based on the fifth instruction template, the first model and the first spliced ​​table information sequence; finally, a multimodal table is merged based on the label-image correspondence table and the label-activity information correspondence table to obtain a corresponding multimodal comprehensive information table to feedback to the current user. The patent information mining mechanism provided by the present invention can perform molecular activity data mining for small molecule drug patents, and based on the present invention, data mining efficiency can be improved, data mining costs can be reduced, and R&D efficiency can be improved.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present invention provides a processing method for mining patent information in combination with a large language model, the method comprising:

[0005] A type of pre-trained multimodal large language model is selected as the corresponding first model; and five target tasks are specified for the first model, recorded as the corresponding first, second, third, fourth and fifth tasks; and the instruction text templates are customized for the first, second, third, fourth and fifth tasks, recorded as the corresponding first, second, third, fourth and fifth instruction templates; and five task data sets are constructed by collecting public small molecule drug patent literature information, recorded as the corresponding first, second, third, fourth and fifth data sets; and the first model is fine-tuned based on the first, second, third, fourth and fifth data sets and the corresponding first, second, third, fourth and fifth instruction templates; the first task is to perform molecular image recognition on the input image; the second task is to perform molecular label recognition on the input text; the third task is to classify the input table; the fourth task is to perform similarity recognition on two input data rows, and the total number of data fields in the two data rows is consistent; the fifth task is to perform molecular label column recognition on the input table;

[0006] After the fine-tuning of the first model is completed, a PDF file of a small molecule drug patent input by a user is received as a corresponding first patent file; and images, text blocks and tables are extracted from the first patent file based on a preset PDF content extraction tool to obtain a corresponding first image information sequence, a first text block information sequence and a first table information sequence;

[0007] Based on the first instruction template and the first model, the first image information sequence is subjected to molecular image screening processing to obtain a corresponding first molecular image information sequence; and based on the second instruction template and the first model, the first text block information sequence is subjected to molecular label screening processing to obtain a corresponding first molecular label information sequence; and based on the third instruction template and the first model, the first table information sequence is subjected to other table filtering processing to obtain a corresponding second table information sequence; and based on the fourth instruction template, the first model and the second table information sequence, the molecular activity table and the headerless table are spliced ​​to obtain a corresponding first spliced ​​table information sequence;

[0008] Based on the first molecular image information sequence, the first molecular label information sequence, the first splicing table information sequence, the fifth instruction template and the first model, a label-image correspondence table is constructed; and based on the fifth instruction template, the first model and the first splicing table information sequence, a label-activity information correspondence table is constructed; and based on the label-image correspondence table and the label-activity information correspondence table, a multimodal table is merged to obtain a corresponding multimodal comprehensive information table;

[0009] A first mining report corresponding to the first patent document and the multimodal comprehensive information table is fed back to the current user.

[0010] Preferably, the first model includes at least a GPT series model, an LLaMA series model, a Qwen series model, and a Wenxin series model;

[0011] The first, second, third, fourth and fifth instruction templates are each a formatted instruction text module;

[0012] The first instruction template is used to take the input image as the corresponding current image; and to instruct the first model to identify whether the current image is a molecular structure diagram through a natural language description text to obtain a corresponding first recognition result, and output the first recognition result as the corresponding first instruction feedback; the first recognition result includes yes and no;

[0013] The second instruction template is used to take the input text as the corresponding current context; and instruct the first model to identify whether the current context contains molecular label information through a natural language description text to obtain a corresponding second recognition result, and further perform molecular label information extraction processing on the current context to obtain corresponding first molecular label extraction information when the second recognition result is yes, and set the corresponding first molecular label extraction information to be empty when the second recognition result is no, and the obtained second recognition result and the first molecular label extraction information constitute the corresponding second instruction feedback output; the second recognition result includes yes and no;

[0014] The third instruction template is used to take the input table as the corresponding current table; and to indicate the first model through a natural language description text to identify which category the current table belongs to, namely, a molecular activity table, a table without a header, and other tables, and output the third recognition result as the corresponding third instruction feedback; the third recognition result includes a molecular activity table, a table without a header, and other tables;

[0015] The fourth instruction template is used to take the two input data rows as the corresponding current data row A and current data row B; and instruct the first model through a natural language description text to perform field-by-field data similarity estimation on the current data rows A and B according to the data field correspondence to obtain the corresponding field similarity, and calculate the average similarity based on all the obtained field similarities, and identify whether each of the field similarities and the average similarity are higher than a preset similarity threshold, if both are higher than the similarity threshold, set the corresponding fourth recognition result as similar, if at least one of the field similarities or the average similarity is lower than the similarity threshold, set the corresponding fourth recognition result as dissimilar, and output the fourth recognition result as the corresponding fourth instruction feedback; the fourth recognition result includes similarity and dissimilarity;

[0016] The fifth instruction template is used to take the input table as the corresponding current table; and instruct the first model to identify whether there is a molecular label column in the current table through a natural language description text to obtain a corresponding fifth recognition result, and further perform molecular label column information extraction processing on the current table to obtain corresponding first column extraction information when the fifth recognition result is yes, and set the corresponding first column extraction information to be empty when the fifth recognition result is no, and the corresponding fifth instruction feedback output is composed of the obtained fifth recognition result and the first column extraction information; the fifth recognition result includes yes and no; the first column extraction information is composed of one or more first column unit information when it is not empty; the first column unit information includes row index information and molecular label information;

[0017] The first image information sequence includes a plurality of first image information; the first image information includes a first identifier, a first page number, a first upper left corner row and column coordinate, a first upper right corner row and column coordinate, a first lower right corner row and column coordinate, a first lower left corner row and column coordinate and a first image; the first image is a visible image of a molecular structure;

[0018] The first text block information sequence includes a plurality of first text block information; the first text block information includes a second identifier, a second page number, a second upper left corner row and column coordinate, a second upper right corner row and column coordinate, a second lower right corner row and column coordinate, a second lower left corner row and column coordinate and a first block of text;

[0019] The first table information sequence includes a plurality of first table information; the first table information includes a third identifier, a third page number, a third upper left corner row and column coordinate, a third upper right corner row and column coordinate, a third lower right corner row and column coordinate, a third lower left corner row and column coordinate and a first table; the first table includes one or more first table rows; the first table row includes a plurality of first row unit texts;

[0020] The first spliced ​​table information sequence includes a plurality of first spliced ​​table information; the first spliced ​​table information includes a fourth identifier, a leading page number, a trailing page number, a leading page upper left corner row and column coordinates, a leading page upper right corner row and column coordinates, a leading page lower right corner row and column coordinates, a leading page lower left corner row and column coordinates, a trailing page upper left corner row and column coordinates, a trailing page upper right corner row and column coordinates, a trailing page lower right corner row and column coordinates, a trailing page lower left corner row and column coordinates, and a first spliced ​​table; the first spliced ​​table includes one or more second table rows; the second table rows include a plurality of second row unit texts;

[0021] The label-image correspondence table includes a plurality of first correspondence records; the first correspondence record includes a first molecule label field and a first molecule image field;

[0022] The label-activity information correspondence table includes a plurality of second correspondence records; the second correspondence records include a second molecule label field and a plurality of first molecule activity fields;

[0023] The multimodal comprehensive information table includes a plurality of third correspondence records; the third correspondence records include the third molecule label field, the second molecule image field and multiple types of second molecule activity fields.

[0024] Preferably, performing molecular image screening processing on the first image information sequence based on the first instruction template and the first model to obtain a corresponding first molecular image information sequence specifically includes:

[0025] A round of traversal is performed on all the first images of the first image information sequence; and during this round of traversal, the first image currently traversed is used as the corresponding current image and input into the first instruction template for corresponding instruction text assembly to obtain the corresponding current task instruction; and the current task instruction is input into the first model for molecular image recognition task processing to obtain the corresponding first instruction feedback; and the first recognition result in the first instruction feedback is used as the corresponding current recognition result; and when the current recognition result is no, the first image information corresponding to the current image is deleted from the first image information sequence; and at the end of this round of traversal, the latest first image information sequence is used as the corresponding first molecular image information sequence.

[0026] Preferably, the performing molecular label screening processing on the first text block information sequence based on the second instruction template and the first model to obtain the corresponding first molecular label information sequence specifically includes:

[0027] A round of traversal is performed on all the first blocks of text in the first text block information sequence; and in this round of traversal, the first block of text currently traversed is used as the corresponding current text to input the second instruction template for corresponding instruction text assembly to obtain the corresponding current task instruction; and the current task instruction is input into the first model for molecular label recognition task processing to obtain the corresponding second instruction feedback; and the second recognition result and the first molecular label extraction information fed back by the second instruction are used as the corresponding current recognition result and current molecular label; and the current recognition result is recognized; if the current recognition result is, the text content of the first block of text corresponding to the current text in the first text block information sequence is reset to the corresponding current molecular label; if the current recognition result is no, the first text block information corresponding to the current text is deleted from the first text block information sequence; and at the end of this round of traversal, the latest first text block information sequence is used as the corresponding first molecular label information sequence.

[0028] Preferably, performing other table filtering processing on the first table information sequence based on the third instruction template and the first model to obtain the corresponding second table information sequence specifically includes:

[0029] A round of traversal is performed on all the first tables in the first table information sequence; and in this round of traversal, the first table currently traversed is used as the corresponding current table and input into the third instruction template to assemble the corresponding instruction text to obtain the corresponding current task instruction; and the current task instruction is input into the first model to perform table classification task processing to obtain the corresponding third instruction feedback; and the third recognition result of the third instruction is used as the corresponding current recognition result; and the current recognition result is recognized; if the current recognition result is a molecular activity table, the first table information corresponding to the current table is marked as the corresponding first type of table information; if the current recognition result is a table without a header, the first table information corresponding to the current table is marked as the corresponding second type of table information; if the current recognition result is other tables, the first table information corresponding to the current table is deleted from the first table information sequence; and at the end of this round of traversal, the latest first table information sequence is used as the corresponding second table information sequence.

[0030] Preferably, the step of performing splicing of the molecular activity table and the headerless table based on the fourth instruction template, the first model and the second table information sequence to obtain a corresponding first spliced ​​table information sequence specifically includes:

[0031] Step 61, initializing an empty sequence as the corresponding third table information sequence; and taking the first of the first table information in the second table information sequence as the corresponding current table information;

[0032] Step 62, identifying whether the current table information is marked as the type of table information; if the current table information is marked as the type of table information, identifying whether the row coordinates of the third lower left corner row and column coordinates of the current table information are the bottom row of the current page, if so, go to step 63, if not, go to step 64; if the current table information is not marked as the type of table information, go to step 65;

[0033] Step 63, add 1 to the third page number of the current table information to obtain the corresponding next page number; and use the second type of table information in which each of the third page numbers in the second table information sequence matches the next page number and the row coordinates of the third upper left corner row and column coordinates are the top row of the current page as the corresponding candidate table information; and identify whether the candidate table information is empty; if the candidate table information is empty, go to step 64; if the candidate table information is not empty, record the first table of the candidate table information as the corresponding next table, and record the first table of the current table information as the corresponding previous table, and count the total number of the first row of unit texts of any one of the first table rows of the previous table to obtain the corresponding first unit total number, and count the The total number of the first row of unit text is counted to obtain the corresponding total number of the second unit, and whether the total number of the first and second units are equal is identified; if the total number of the first and second units are not equal, go to step 64; if the total number of the first and second units are equal, the second first table row of the front table and the first first table row of the back table are used as the corresponding first and second data rows to input the fourth instruction template to perform corresponding instruction text assembly to obtain the corresponding current task instruction, and the current task instruction is input into the first model to perform data row similarity recognition task processing to obtain the corresponding fourth instruction feedback, and the fourth recognition result in the fourth instruction feedback is used as the corresponding current recognition result, and the current recognition result is identified; if the current recognition result is not similar, go to step 64;If the current recognition result is similar, then a new first spliced ​​table information with empty content is added to the third table information sequence as the corresponding current spliced ​​table information, and a unique identifier is assigned to the current spliced ​​table information as the corresponding current spliced ​​table identifier, and the post-table is added to the end of the pre-table to obtain the corresponding current spliced ​​table, and the fourth identifier of the current spliced ​​table information is set to the corresponding current spliced ​​table identifier, and the pre- and post-page numbers of the current spliced ​​table information are set to the third page numbers corresponding to the pre- and post-tables, and the row and column coordinates of the upper left corner of the pre-page, the row and column coordinates of the upper right corner of the pre-page, and the row and column coordinates of the lower right corner of the pre-page of the current spliced ​​table information are set to the corresponding current spliced ​​table identifier. and the lower left corner row and column coordinates of the leading page are set to the third upper left corner row and column coordinates, the third upper right corner row and column coordinates, the third lower right corner row and column coordinates and the third lower left corner row and column coordinates corresponding to the leading table, and the upper left corner row and column coordinates, the upper right corner row and column coordinates, the lower right corner row and column coordinates and the lower left corner row and column coordinates of the trailing page of the current splicing table information are set to the third upper left corner row and column coordinates, the third upper right corner row and column coordinates, the third lower right corner row and column coordinates and the third lower left corner row and column coordinates corresponding to the trailing table, and the first splicing table of the current splicing table information is set to the corresponding current splicing table, and after the setting of the current splicing table information is completed, go to step 65;

[0034] Step 64, record the first table of the current table information as the corresponding current table; add a new first spliced ​​table information with empty content in the third table information sequence as the corresponding current spliced ​​table information; assign a unique identifier to the current spliced ​​table information as the corresponding current spliced ​​table identifier; set the fourth identifier of the current spliced ​​table information to the corresponding current spliced ​​table identifier; set the leading page number of the current spliced ​​table information to the third page number corresponding to the current table; set the following page number of the current spliced ​​table information to empty; and The upper left corner row and column coordinates, the upper right corner row and column coordinates, the lower right corner row and column coordinates and the lower left corner row and column coordinates of the preceding page of the grid information are set to the third upper left corner row and column coordinates, the third upper right corner row and column coordinates, the third lower right corner row and column coordinates and the third lower left corner row and column coordinates corresponding to the current table; and the upper left corner row and column coordinates, the upper right corner row and column coordinates, the lower right corner row and column coordinates and the lower left corner row and column coordinates of the following page of the current spliced ​​table information are all set to null; and the first spliced ​​table of the current spliced ​​table information is set to the corresponding current table;

[0035] Step 65, identifying whether the current table information is the last one of the first table information in the second table information sequence; if not, extracting the next one of the first table information in the second table information sequence as the new current table information and returning to step 62; if so, outputting the latest first splicing table information sequence as the result of this splicing processing.

[0036] Preferably, the constructing of the label-image correspondence table based on the first molecular image information sequence, the first molecular label information sequence, the first splicing table information sequence, the fifth instruction template and the first model specifically includes:

[0037] Step 71, taking the first of the first molecular image information sequences as the corresponding current molecular image;

[0038] Step 72, taking the first page number corresponding to the current molecular image as the corresponding current image page number, and recording the area covered by the first upper left corner row and column coordinates, the first upper right corner row and column coordinates, the first lower right corner row and column coordinates and the first lower left corner row and column coordinates corresponding to the current molecular image as the corresponding current image area;

[0039] Step 73, record the first spliced ​​table information in which the preceding page number or the succeeding page number in the first spliced ​​table information sequence matches the current image page number as the corresponding first same-page table information; and based on the row and column coordinates of the upper left corner of the preceding / successing page, the row and column coordinates of the upper right corner of the preceding / successing page, the row and column coordinates of the lower right corner of the preceding / successing page, and the row and column coordinates of the lower left corner of the preceding / successing page of each of the first same-page table information, confirm the coverage of the current first same-page table information on the current image page number to obtain the corresponding first same-page table area; and when the number of the first same-page table areas is not zero, confirm whether there is one of the first same-page table areas that intersects with the current image area to obtain the corresponding first confirmation result; and identify the first confirmation result; if the first confirmation result is not present, go to step 74; if the first confirmation result is present, take the first same-page table information corresponding to the first same-page table area that currently intersects with the current image area as the corresponding current same-page table information, and go to step 75;

[0040] Step 74, record the first text block information whose second page number in the first molecule label information sequence matches the current image page number as the corresponding first same-page text block information; and confirm the coverage of the current first same-page text block information on the current image page number based on the second upper left corner row and column coordinates, the second upper right corner row and column coordinates, the second lower right corner row and column coordinates and the second lower left corner row and column coordinates of each of the first same-page text block information to obtain the corresponding first same-page text block area; and take the first block of text of the first text block information corresponding to the first same-page text block area that is closest to the current image area as the corresponding current molecule label; and go to step 76;

[0041] Step 75, input the first spliced ​​table of the current same-page table information into the fifth instruction template for corresponding instruction text assembly to obtain the corresponding current task instruction; and input the current task instruction into the first model for molecular label column identification task processing to obtain the corresponding fifth instruction feedback; and extract the corresponding first column extraction information from the fifth instruction feedback; and record the full table coverage area of ​​the single page or double pages of the current same-page table information as the corresponding first table area; and record the first upper left corner row and column coordinates, the first upper right corner row and column coordinates, the first lower right corner row and column coordinates and the first molecular image information sequence of each of the first molecular images. The area covered by the first lower left corner row and column coordinates is recorded as the corresponding first image area; the first image area that intersects with the first table area is recorded as the corresponding first in-table image area, and all the first in-table image areas are sorted in order from front to back in the intersection area to form a corresponding first in-table image area sequence; the sequence index corresponding to the first in-table image area that matches the current image area in the first in-table image area sequence is used as the corresponding current row index; and the molecular label information of the first column unit information whose row index information in the first column extraction information matches the current row index is extracted as the corresponding current molecular label;

[0042] Step 76, taking the current molecular image as a corresponding first molecular image field; taking the current molecular label as a corresponding first molecular label field; and forming a corresponding first corresponding relationship record by the obtained first molecular label field and the first molecular image field;

[0043] Step 77, identifying whether the current molecular image is the last first molecular image of the first molecular image information sequence; if not, extracting the next first molecular image of the first molecular image information sequence as the new current molecular image, and returning to step 72; if so, forming the corresponding label-image correspondence table by all the first correspondence records obtained.

[0044] Preferably, the step of constructing a label-activity information correspondence table based on the fifth instruction template, the first model and the first concatenated table information sequence specifically includes:

[0045] Step 81, extract the first second table row of all the first concatenated tables of the first concatenated table information sequence to form a corresponding second table row set; extract all the second row unit texts of the second table row set to form a corresponding second row unit text set; perform unit text deduplication processing on the second row unit text set; perform a round of traversal on all the second row unit texts of the second row unit text set that have completed deduplication; and in this round of traversal, use the second row unit text currently traversed as the corresponding current field name; and based on the preset molecular activity keyword dictionary, identify whether the current field name is a type of molecular activity name to obtain a corresponding activity name recognition result; and when the activity name recognition result is yes, mark the current field name as a corresponding molecular activity field name; and at the end of this round of traversal, all the obtained molecular activity field names form a corresponding molecular activity field name set;

[0046] Step 82, perform a round of traversal of all the first splicing tables of the first splicing table information sequence; and in this round of traversal, use the first splicing table currently traversed as the corresponding current splicing table; and input the current splicing table into the fifth instruction template to assemble the corresponding instruction text to obtain the corresponding current task instruction; and input the current task instruction into the first model to perform molecular label column identification task processing to obtain the corresponding fifth instruction feedback; and extract the corresponding first column extraction information from the fifth instruction feedback; and use the first second table row of the current splicing table as the corresponding current header row; and use the second row unit text corresponding to the first column extraction information in the current header row as a corresponding Molecular label field name; and perform a poll on all the second-row unit texts except the molecular label field name in the current header row; and during this polling process, record the second-row unit text currently polled as the corresponding current field name, and identify whether the current field name is included in the molecular activity field name set, if so, take the second-row unit text currently polled as a corresponding molecular activity field name, if not, record the arrangement index of the second-row unit text corresponding to the current field name in the current header row as the corresponding column index to be deleted; and at the end of this polling, delete the second-row unit texts corresponding to all the column indexes to be deleted in all the second table rows of the current spliced ​​table;

[0047] Step 83, after the current round of traversal of all the first splicing tables in the first splicing table information sequence is completed, all the field contents of all the first splicing tables are fully connected and merged using the molecular label field name as the connection key field name to obtain a new table with a molecular label field and all molecular activity fields as the corresponding label-activity information correspondence table.

[0048] The second aspect of the embodiment of the present invention provides a device for implementing the processing method of mining patent information in combination with a large language model as described in the first aspect, the device comprising: a large language model fine-tuning module, a patent information receiving module, a patent information denoising module, a patent information mining module, and a mining report feedback module;

[0049] The large language model fine-tuning module is used to select a type of pre-trained multimodal large language model as the corresponding first model; and specify five target tasks for the first model, recorded as the corresponding first, second, third, fourth and fifth tasks; and customize instruction text templates for the first, second, third, fourth and fifth tasks, recorded as the corresponding first, second, third, fourth and fifth instruction templates; and construct five task data sets by collecting public small molecule drug patent literature information, recorded as the corresponding first, second, third, fourth and fifth data sets; and fine-tune the first model based on the first, second, third, fourth and fifth data sets and the corresponding first, second, third, fourth and fifth instruction templates; the first task is to perform molecular image recognition on the input image; the second task is to perform molecular label recognition on the input text; the third task is to classify the input table; the fourth task is to perform similarity recognition on two input data rows, and the total number of data fields in the two data rows is consistent; the fifth task is to perform molecular label column recognition on the input table;

[0050] The patent information receiving module is used to receive the PDF file of the small molecule drug patent input by the user as the corresponding first patent file after the first model fine-tuning is completed; and extract the image, text block and table of the first patent file based on the preset PDF content extraction tool to obtain the corresponding first image information sequence, first text block information sequence and first table information sequence;

[0051] The patent information denoising module is used to perform molecular image screening processing on the first image information sequence based on the first instruction template and the first model to obtain a corresponding first molecular image information sequence; and to perform molecular label screening processing on the first text block information sequence based on the second instruction template and the first model to obtain a corresponding first molecular label information sequence; and to perform other table filtering processing on the first table information sequence based on the third instruction template and the first model to obtain a corresponding second table information sequence; and to perform molecular activity table and headerless table splicing processing based on the fourth instruction template, the first model and the second table information sequence to obtain a corresponding first spliced ​​table information sequence;

[0052] The patent information mining module is used to construct a label-image correspondence table based on the first molecular image information sequence, the first molecular label information sequence, the first splicing table information sequence, the fifth instruction template and the first model; and to construct a label-activity information correspondence table based on the fifth instruction template, the first model and the first splicing table information sequence; and to merge multimodal tables based on the label-image correspondence table and the label-activity information correspondence table to obtain a corresponding multimodal comprehensive information table;

[0053] The mining report feedback module is used to feed back a first mining report corresponding to the first patent document and the multimodal comprehensive information table to the current user.

[0054] A third aspect of an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0055] The processor is used to be coupled to the memory, read and execute instructions in the memory, so as to implement the method steps described in the first aspect above;

[0056] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0057] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed by a computer, the computer executes the instructions of the method described in the first aspect above.

[0058] The embodiment of the present invention provides a processing method, device, electronic device and computer-readable storage medium for mining patent information in combination with a large language model. As can be seen from the above content, the embodiment of the present invention selects a type of pre-trained multimodal large language model as the first model, and specifies five target tasks for it (molecular image recognition of input images, molecular label recognition of input text, classification of input tables, similarity recognition of two input data rows, and molecular label column recognition of input tables), customizes five instruction text templates, constructs five data sets, and fine-tunes the first model for five types of tasks based on the five data sets and the corresponding five instruction text templates; after the model fine-tuning is completed, the PDF file of the small molecule drug patent input by the user is received, and the patent document is extracted based on the PDF content extraction tool. Image, text block and table extraction; then, the first image information sequence is molecularly imaged based on the first instruction template and the first model. Image screening, molecular label screening of the first text block information sequence based on the second instruction template and the first model, other table filtering of the first table information sequence based on the third instruction template and the first model, molecular activity table and headerless table splicing based on the fourth instruction template, the first model and the second table information sequence; then, a label-image correspondence table is constructed based on the first molecular image information sequence, the first molecular label information sequence, the first spliced ​​table information sequence, the fifth instruction template and the first model, and a label-activity information correspondence table is constructed based on the fifth instruction template, the first model and the first spliced ​​table information sequence; finally, multimodal tables are merged based on the label-image correspondence table and the label-activity information correspondence table to obtain the corresponding multimodal comprehensive information table and feedback to the current user. The patent information mining mechanism provided in the embodiment of the present invention can perform molecular activity data mining for small molecule drug patents. Based on the embodiment of the present invention, not only the data mining efficiency is improved, the data mining cost is reduced, but also the research and development efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of a processing method for mining patent information by combining a large language model provided in the first embodiment of the present invention;

[0060] Figure 2 A module structure diagram of a processing device for mining patent information in combination with a large language model provided in the second embodiment of the present invention;

[0061] Figure 3 A schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Embodiment 1 of the present invention provides a processing method for mining patent information by combining a large language model, such as Figure 1 As shown in the schematic diagram of a processing method for mining patent information by combining a large language model provided in the first embodiment of the present invention, the method mainly includes the following steps:

[0064] Step 1, select a type of pre-trained multimodal large language model as the corresponding first model; and specify five target tasks for the first model, recorded as the corresponding first, second, third, fourth and fifth tasks; and customize instruction text templates for the first, second, third, fourth and fifth tasks, recorded as the corresponding first, second, third, fourth and fifth instruction templates; and construct five task data sets by collecting public small molecule drug patent literature information, recorded as the corresponding first, second, third, fourth and fifth data sets; and fine-tune the first model based on the first, second, third, fourth and fifth data sets and the corresponding first, second, third, fourth and fifth instruction templates;

[0065] Specifically comprising: step 11, selecting a type of pre-trained multimodal large language model as the corresponding first model;

[0066] Here, the first model of the embodiment of the present invention includes at least a GPT series model, an LLaMA series model, a Qwen series model, and a Wenxin series model;

[0067] Step 12, and specify five target tasks for the first model as the corresponding first, second, third, fourth and fifth tasks;

[0068] Here, the five target tasks specified by the embodiment of the present invention for the first model are specifically: the first task is to perform molecular image recognition on the input image; the second task is to perform molecular label recognition on the input text; the third task is to classify the input table; the fourth task is to perform similarity recognition on two input data rows, and the total number of data fields in the two data rows is consistent; the fifth task is to perform molecular label column recognition on the input table;

[0069] Step 13, and the customized instruction text templates for the first, second, third, fourth and fifth tasks are recorded as the corresponding first, second, third, fourth and fifth instruction templates;

[0070] Here, the first, second, third, fourth and fifth instruction templates provided in the embodiment of the present invention are each a formatted instruction text module; wherein:

[0071] 1) The first instruction template is used to take the input image as the corresponding current image; and to instruct the first model to identify whether the current image is a molecular structure diagram through a natural language description text to obtain a corresponding first recognition result, and output the first recognition result as the corresponding first instruction feedback; the first recognition result includes yes and no;

[0072] 2) The second instruction template is used to take the input text as the corresponding current context; and instruct the first model to identify whether the current context contains molecular label information through a natural language description text to obtain a corresponding second recognition result, and further perform molecular label information extraction processing on the current context to obtain corresponding first molecular label extraction information when the second recognition result is yes, and set the corresponding first molecular label extraction information to be empty when the second recognition result is no, and the obtained second recognition result and the first molecular label extraction information constitute the corresponding second instruction feedback output; the second recognition result includes yes and no;

[0073] 3) The third instruction template is used to take the input table as the corresponding current table; and to instruct the first model through a natural language description text to identify which category the current table belongs to, namely, a molecular activity table, a table without a header, and other tables, and to output the third recognition result as the corresponding third instruction feedback; the third recognition result includes a molecular activity table, a table without a header, and other tables;

[0074] 4) The fourth instruction template is used to take the two input data rows as the corresponding current data row A and current data row B; and instruct the first model through a natural language description text to perform field-by-field data similarity estimation on the current data rows A and B according to the data field correspondence to obtain the corresponding field similarity, and calculate the average similarity based on all the obtained field similarities, and identify whether the similarities of each field and the average similarity are both higher than the preset similarity threshold, if both are higher than the similarity threshold, then set the corresponding fourth recognition result as similar, if at least one field similarity or the average similarity is lower than the similarity threshold, then set the corresponding fourth recognition result as dissimilar, and output the fourth recognition result as the corresponding fourth instruction feedback; the fourth recognition result includes similarity and dissimilarity;

[0075] 5) The fifth instruction template is used to use the input table as the corresponding current table; and to instruct the first model through a natural language description text to identify whether there is a molecular label column in the current table to obtain a corresponding fifth recognition result, and when the fifth recognition result is yes, further perform molecular label column information extraction processing on the current table to obtain corresponding first column extraction information, and when the fifth recognition result is no, set the corresponding first column extraction information to be empty, and the obtained fifth recognition result and the first column extraction information constitute the corresponding fifth instruction feedback output; the fifth recognition result includes yes and no; when the first column extraction information is not empty, it is composed of one or more first column unit information; the first column unit information includes row index information and molecular label information;

[0076] Step 14, and construct five task data sets by collecting public small molecule drug patent literature information and record them as the corresponding first, second, third, fourth and fifth data sets;

[0077] Here, the data contents of the first, second, third, fourth and fifth data sets constructed by big data collection in the embodiment of the present invention are as follows:

[0078] 1) The first data set includes a plurality of first data records; the first data record includes a first training image and a first label recognition result; the first training image includes a positive sample image and a negative sample image, the positive sample image is a visual image of a molecular structure, and the negative sample image is a visual image unrelated to the molecular structure; the first label recognition result includes yes and no, when the first training image is a positive sample image, the first label recognition result is yes, and when the first training image is a negative sample image, the first label recognition result is no;

[0079] 2) The second data set includes a plurality of second data records; the second data record includes a first training text, a second label recognition result and a first label molecule label; the first training text includes a positive sample text and a negative sample text, the positive sample text is a text segment with molecule label information, and the negative sample text is a text segment without molecule label information; the second label recognition result includes yes and no, when the first training text is a positive sample text, the second label recognition result is yes, and the first label molecule label is the molecule label information in the current first training text, and when the first training text is a negative sample text, the second label recognition result is no, and the first label molecule label is empty;

[0080] 3) The third data set includes multiple third data records; the third data record includes the first training table and the third label recognition result; the table type of the first training table includes a molecular activity data table type, a headerless data table type, and any other data table type that is neither a molecular activity data table type nor a headerless data table type; the field set of the molecular activity data table type should include a molecular label field and at least one type of molecular activity field, and the types of molecular activity fields include at least catalytic activity, binding activity, signal transduction activity, toxicity activity, antioxidant activity, drug activity, photosensitivity, conductivity, and thermosensitivity; the data table corresponding to the headerless data table type has no field set; the third label recognition result includes a molecular activity table, a headerless table, and other tables;

[0081] 4) The fourth data set includes a plurality of fourth data records; the fourth data record includes a first training data row, a second training data row and a fourth label recognition result; the number of data fields in the first and second training data rows is the same; the fourth label recognition result includes similarity and dissimilarity; when the data formats of two field data corresponding to each data field in the first and second training data rows meet the same format requirement, the fourth label recognition result is similar; when the data formats of two field data corresponding to at least one data field in the first and second training data rows do not meet the same format requirement, the fourth label recognition result is dissimilar;

[0082] 5) The fifth data set includes a plurality of fifth data records; the fifth data record includes a second training table, a fifth label recognition result and first label column information; the second training table includes a positive sample table and a negative sample table, the positive sample table is a data table with a molecular label field, and the negative sample table is a data table without a molecular label field; the fifth label recognition result includes yes and no; when the second training table is a positive sample table, the fifth label recognition result is yes, the first label column information is composed of one or more first label column unit information, and the first label column unit information includes row index information and molecular label information; when the second training table is a negative sample table, the fifth label recognition result is no, and the first label column information is empty;

[0083] Step 15, fine-tuning the first model based on the first, second, third, fourth and fifth data sets and the corresponding first, second, third, fourth and fifth instruction templates;

[0084] Specifically, it includes: step 151, fine-tuning the first model for the first task based on the first data set and the corresponding first instruction template;

[0085] Specifically, it includes: step 1511, dividing the first data set into two sub-data sets based on a preset first division ratio and recording them as a corresponding first training set and a first evaluation set;

[0086] Here, the first segmentation ratio is a preset ratio parameter, such as 8:2; the first training set and the first evaluation set are both composed of a plurality of first data records; the ratio of the total number of records of the first training set to the first evaluation set satisfies the first segmentation ratio;

[0087] Step 1512, perform a round of traversal on all first data records of the first training set; and in this round of traversal, use the first data record currently traversed as the corresponding current training record; and extract the first training image of the current training record as the corresponding current image; and input the current image into the first instruction template to assemble the corresponding instruction text to obtain the corresponding first task instruction; and input the first task instruction into the first model to process the molecular image recognition task to obtain the corresponding first instruction feedback; and extract the corresponding first recognition result from the first instruction feedback; and form a corresponding first prediction-label pair from the first recognition result and the first label recognition result of the current training record; and at the end of this round of traversal, form all the first prediction-label pairs obtained in this round of traversal into a corresponding first prediction-label pair set;

[0088] Step 1513: Substitute the first prediction-label pair set into the preset first model loss function L M1 ; and based on the preset first model optimizer, move towards making the first model loss function L M1 A round of fine-tuning is performed on the first model in the direction of reaching the minimum value;

[0089] Among them, the first model loss function L M1 At least including L1 loss function, L2 loss function, log likelihood loss function, cross entropy loss function and intersection loss function; the first model optimizer at least includes SGD optimizer, Adam optimizer and AdamW optimizer;

[0090] Step 1514, perform a round of traversal on all first data records of the first evaluation set; and in this round of traversal, use the first data record currently traversed as the corresponding current evaluation record; and extract the first evaluation image of the current evaluation record as the corresponding current image; and input the current image into the first instruction template to assemble the corresponding instruction text to obtain the corresponding first task instruction; and input the first task instruction into the first model to process the molecular image recognition task to obtain the corresponding first instruction feedback; and extract the corresponding first recognition result from the first instruction feedback; and form a corresponding second prediction-label pair from the first recognition result and the first label recognition result of the current evaluation record; and at the end of this round of traversal, form all the second prediction-label pairs obtained in this round of traversal into a corresponding second prediction-label pair set;

[0091] Step 1515, calculating the accuracy, precision, recall and F1 score of the set based on the second prediction-label pair to obtain the corresponding first accuracy, first precision, first recall and first F1 score;

[0092] Step 1516, identifying the first accuracy, the first precision, the first recall and the first F1 score; if the first accuracy does not meet the preset first accuracy range or the first accuracy does not meet the preset first accuracy range or the first recall does not meet the preset first recall range or the first F1 score does not meet the preset first F1 score range, returning to step 1512 to continue training; if the first accuracy meets the first accuracy range and the first accuracy meets the first accuracy range and the first recall meets the first recall range and the first F1 score meets the first F1 score range, stopping the current round of fine-tuning and confirming the end of the first task fine-tuning;

[0093] Here, the first accuracy range, the first precision range, the first recall range, and the first F1 score range are four set value ranges;

[0094] Step 152, after the first task fine-tuning is completed, the first model is fine-tuned for the second task based on the second data set and the corresponding second instruction template;

[0095] Specifically, it includes: step 1521, dividing the second data set into two sub-data sets based on a preset second division ratio and recording them as corresponding second training sets and second evaluation sets;

[0096] Here, the second segmentation ratio is a preset ratio parameter, such as 8:2; the second training set and the second evaluation set are both composed of a plurality of second data records; the ratio of the total number of records of the second training set to the second evaluation set satisfies the second segmentation ratio;

[0097] Step 1522, taking the first second data record of the second training set as the corresponding current training record;

[0098] Step 1523, inputting the first training text of the current training record into the second instruction template to perform corresponding instruction text assembly to obtain the corresponding second task instruction; inputting the second task instruction into the first model to perform molecular label recognition task processing to obtain the corresponding second instruction feedback; and extracting the corresponding second recognition result and the first molecular label extraction information from the second instruction feedback;

[0099] Step 1524: perform binary classification vector conversion on the second recognition result and the second label recognition result of the current training record to obtain the corresponding predicted classification vector y pre and label classification vector y tag; and perform word embedding encoding on the first molecular label extraction information and the first label molecular label of the current training record to obtain the corresponding predicted text vector z pre and label text vector z tag ; Predict the classification vector y pre , label classification vector y tag , predicted text vector z pre and label text vector z tag Bring in the preset second model loss function L M2 ; and based on the preset second model optimizer, move towards making the second model loss function L M2 A round of fine-tuning is performed on the first model in the direction of reaching the minimum value;

[0100] Here, the second model loss function L M2 for:

[0101] L M2 =L M2-a (y pre ,y tag )+L M2-n (z pre ,z tag );

[0102] Among them, the sub-loss function L M2-a At least including log-likelihood loss function, cross entropy loss function and intersection loss function, sub-loss function L M2-b At least including L1 loss function and L2 loss function;

[0103] In addition, the second model optimizer includes at least an SGD optimizer, an Adam optimizer, and an AdamW optimizer;

[0104] Step 1525, identifying whether the current training record is the last second data record of the second training set; if so, going to step 1526; if not, extracting the next second data record of the second training set as the new current training record and returning to step 1523;

[0105] Step 1526, perform a round of traversal on all second data records of the second evaluation set; and in this round of traversal, use the currently traversed second data record as the corresponding current evaluation record; input the first training text of the current evaluation record into the second instruction template to perform corresponding instruction text assembly to obtain the corresponding second task instruction; input the second task instruction into the first model to perform molecular label recognition task processing to obtain the corresponding second instruction feedback; extract the corresponding second recognition result and the first molecular label extraction information from the second instruction feedback; and form a corresponding third prediction-label pair with the first molecular label extraction information and the first label molecular label of the current evaluation record; and at the end of this round of traversal, form all the third prediction-label pairs obtained in this round of traversal into a corresponding third prediction-label pair set;

[0106] Step 1527, calculating the accuracy, precision, recall and F1 score of the third prediction-label pair set to obtain a corresponding second accuracy, second precision, second recall and second F1 score;

[0107] Step 1528, identifying the second accuracy, the second precision, the second recall, and the second F1 score; if the second accuracy does not meet the preset second accuracy range or the second accuracy does not meet the preset second accuracy range or the second recall does not meet the preset second recall range or the second F1 score does not meet the preset second F1 score range, returning to step 1522 to continue training; if the second accuracy meets the second accuracy range and the second accuracy meets the second accuracy range and the second recall meets the second recall range and the second F1 score meets the second F1 score range, then stop the current round of fine-tuning and confirm that the second task fine-tuning is finished;

[0108] Here, the second accuracy range, the second precision range, the second recall range, and the second F1 score range are four set value ranges;

[0109] Step 153: after the second task fine-tuning is completed, the first model is fine-tuned for the third task based on the third data set and the corresponding third instruction template;

[0110] Specifically, it includes: step 1531, dividing the third data set into two sub-data sets based on a preset third division ratio and recording them as a corresponding third training set and a third evaluation set;

[0111] Here, the third segmentation ratio is a preset ratio parameter, such as 8:2; the third training set and the third evaluation set are both composed of a plurality of third data records; the ratio of the total number of records of the third training set to the third evaluation set satisfies the third segmentation ratio;

[0112] Step 1532, perform a round of traversal on all third data records of the third training set; and in this round of traversal, use the third data record currently traversed as the corresponding current training record; and input the first training table of the current training record into the third instruction template to assemble the corresponding instruction text to obtain the corresponding third task instruction; and input the third task instruction into the first model to perform table classification task processing to obtain the corresponding third instruction feedback; and extract the corresponding third recognition result from the third instruction feedback; and form a corresponding fourth prediction-label pair from the third recognition result and the third label recognition result of the current training record; and at the end of this round of traversal, form all the fourth prediction-label pairs obtained in this round of traversal into a corresponding fourth prediction-label pair set;

[0113] Step 1533: Substitute the fourth prediction-label pair set into the preset third model loss function L M3 ; and based on the preset third model optimizer, move towards making the third model loss function L M3 A round of fine-tuning is performed on the first model in the direction of reaching the minimum value;

[0114] Among them, the third model loss function L M3 At least including L1 loss function, L2 loss function, cross entropy loss function and intersection loss function; the third model optimizer at least includes SGD optimizer, Adam optimizer and AdamW optimizer;

[0115] Step 1534, perform a round of traversal on all third data records of the third evaluation set; and in this round of traversal, use the third data record currently traversed as the corresponding current evaluation record; and input the first training table of the current evaluation record into the third instruction template to assemble the corresponding instruction text to obtain the corresponding third task instruction; and input the third task instruction into the first model to perform table classification task processing to obtain the corresponding third instruction feedback; and extract the corresponding third recognition result from the third instruction feedback; and form a corresponding fifth prediction-label pair from the third recognition result and the third label recognition result of the current evaluation record; and at the end of this round of traversal, form all the fifth prediction-label pairs obtained in this round of traversal into a corresponding fifth prediction-label pair set;

[0116] Step 1535, calculating the accuracy, precision, recall and F1 score of the fifth prediction-label pair set to obtain a corresponding third accuracy, third precision, third recall and third F1 score;

[0117] Step 1536, identifying the third accuracy, the third precision, the third recall and the third F1 score; if the third accuracy does not meet the preset third accuracy range or the third accuracy does not meet the preset third accuracy range or the third recall does not meet the preset third recall range or the third F1 score does not meet the preset third F1 score range, returning to step 1532 to continue training; if the third accuracy meets the third accuracy range and the third accuracy meets the third accuracy range and the third recall meets the third recall range and the third F1 score meets the third F1 score range, stopping the current round of fine-tuning and confirming the end of the third task fine-tuning;

[0118] Here, the third accuracy range, the third precision range, the third recall range, and the third F1 score range are four set value ranges;

[0119] Step 154, after the third task fine-tuning is completed, the first model is fine-tuned for the fourth task based on the fourth data set and the corresponding fourth instruction template;

[0120] Specifically, it includes: step 1541, dividing the fourth data set into two sub-data sets based on a preset fourth division ratio and recording them as a corresponding fourth training set and a fourth evaluation set;

[0121] Here, the fourth segmentation ratio is a preset ratio parameter, such as 8:2; the fourth training set and the fourth evaluation set are both composed of a plurality of fourth data records; the ratio of the total number of records of the fourth training set to the fourth evaluation set satisfies the fourth segmentation ratio;

[0122] Step 1542, perform a round of traversal on all fourth data records of the fourth training set; and in this round of traversal, use the fourth data record currently traversed as the corresponding current training record; and input the first and second training data rows of the current training record into the fourth instruction template to assemble the corresponding instruction text to obtain the corresponding fourth task instruction; and input the fourth task instruction into the first model to perform data row similarity recognition task processing to obtain the corresponding fourth instruction feedback; and extract the corresponding fourth recognition result from the fourth instruction feedback; and form a corresponding sixth prediction-label pair from the fourth recognition result and the fourth label recognition result of the current training record; and at the end of this round of traversal, form all the sixth prediction-label pairs obtained in this round of traversal into a corresponding sixth prediction-label pair set;

[0123] Step 1543: Substitute the sixth prediction-label pair set into the preset fourth model loss function L M4 ; and based on the preset fourth model optimizer, move towards making the fourth model loss function L M4 A round of fine-tuning is performed on the first model in the direction of reaching the minimum value;

[0124] Among them, the fourth model loss function LM4 At least including L1 loss function, L2 loss function, log likelihood loss function, cross entropy loss function and intersection loss function; the fourth model optimizer at least includes SGD optimizer, Adam optimizer and AdamW optimizer;

[0125] Step 1544, perform a round of traversal on all fourth data records of the fourth evaluation set; and in this round of traversal, use the fourth data record currently traversed as the corresponding current evaluation record; and input the first and second training data rows of the current evaluation record into the fourth instruction template to assemble the corresponding instruction text to obtain the corresponding fourth task instruction; and input the fourth task instruction into the first model to perform data row similarity recognition task processing to obtain the corresponding fourth instruction feedback; and extract the corresponding fourth recognition result from the fourth instruction feedback; and form a corresponding seventh prediction-label pair from the fourth recognition result and the fourth label recognition result of the current evaluation record; and at the end of this round of traversal, form all the seventh prediction-label pairs obtained in this round of traversal into a corresponding seventh prediction-label pair set;

[0126] Step 1545, calculating the accuracy, precision, recall and F1 score of the seventh prediction-label pair set to obtain a corresponding fourth accuracy, fourth precision, fourth recall and fourth F1 score;

[0127] Step 1546, identifying the fourth accuracy, the fourth precision, the fourth recall and the fourth F1 score; if the fourth accuracy does not meet the preset fourth accuracy range or the fourth accuracy does not meet the preset fourth accuracy range or the fourth recall does not meet the preset fourth recall range or the fourth F1 score does not meet the preset fourth F1 score range, returning to step 1542 to continue training; if the fourth accuracy meets the fourth accuracy range and the fourth accuracy meets the fourth accuracy range and the fourth recall meets the fourth recall range and the fourth F1 score meets the fourth F1 score range, stopping the current round of fine-tuning and confirming that the fourth task fine-tuning is finished;

[0128] Here, the fourth accuracy range, the fourth precision range, the fourth recall range, and the fourth F1 score range are four set value ranges;

[0129] Step 155, after the fourth task fine-tuning is completed, the first model is fine-tuned for the fifth task based on the fifth data set and the corresponding fifth instruction template;

[0130] Specifically, it includes: step 1551, dividing the fifth data set into two sub-data sets based on a preset fifth division ratio and recording them as a corresponding fifth training set and a fifth evaluation set;

[0131] Here, the fifth segmentation ratio is a preset ratio parameter, such as 8:2; the fifth training set and the fifth evaluation set are both composed of a plurality of fifth data records; the ratio of the total number of records of the fifth training set to the fifth evaluation set satisfies the fifth segmentation ratio;

[0132] Step 1552, taking the first fifth data record of the fifth training set as the corresponding current training record;

[0133] Step 1553, input the second training table of the current training record into the fifth instruction template to assemble the corresponding instruction text to obtain the corresponding fifth task instruction; input the fifth task instruction into the first model to perform molecular label column recognition task processing to obtain the corresponding fifth instruction feedback; and extract the corresponding fifth recognition result and the first column extraction information from the fifth instruction feedback;

[0134] Step 1554: perform binary classification vector conversion on the fifth recognition result and the fifth label recognition result of the current training record to obtain the corresponding predicted classification vector x pre and label classification vector x tag ;

[0135] Step 1555, and count the total number of the first column unit information of the first column extraction information to obtain the corresponding first total number, and count the total number of the first label column unit information of the first label column information of the current training record to the corresponding second total number; and use the maximum value of the first and second total numbers as the corresponding current maximum value N; and when there is a total number less than the current maximum value N in the first and second total numbers, use the first column extraction information or the first label column information corresponding to the first or second total number that is currently less than the current maximum value N as the corresponding current column information, and supplement the total number of column unit information of the current column information to N by supplementing the current column information with all empty column unit information; and perform one-hot encoding on the row index information of each first column unit information of the current first column extraction information to obtain the corresponding row index encoding vector y pre,i , and perform word embedding encoding on the molecular label information of each first column unit information to obtain the corresponding label encoding vector z pre,i , 1≤unit index i≤N; and perform unique hot encoding on the row index information of each first label column unit information of the current first label column information to obtain the corresponding row index encoding vector y tag,i , and perform word embedding encoding on the molecular label information of each first label column unit information to obtain the corresponding label encoding vector z tag,i ;

[0136] Step 1556: predict the classification vector x pre , label classification vector x tag , N row index encoding vectors y pre,i , N row index encoding vectors ytag,i , N labeled encoding vectors z pre,i and N labeled encoding vectors z tag,i Bring in the preset fifth model loss function L M5 ; and based on the preset fifth model optimizer, move towards making the fifth model loss function L M5 A round of fine-tuning is performed on the first model in the direction of reaching the minimum value;

[0137] Here, the fifth model loss function L M5 for:

[0138]

[0139] Among them, the sub-loss function L M5-a At least including log-likelihood loss function, cross entropy loss function and intersection loss function, sub-loss function L M5-b At least including L1 loss function, L2 loss function, sub-loss function L M5-c At least including L1 loss function and L2 loss function;

[0140] In addition, the fifth model optimizer includes at least an SGD optimizer, an Adam optimizer, and an AdamW optimizer;

[0141] Step 1557, identifying whether the current training record is the last fifth data record of the fifth training set; if so, going to step 1558; if not, extracting the next fifth data record of the fifth training set as the new current training record and returning to step 1553;

[0142] Step 1558, perform a round of traversal on all fifth data records of the fifth evaluation set; and in this round of traversal, use the fifth data record currently traversed as the corresponding current evaluation record; input the second training table of the current evaluation record into the fifth instruction template to assemble the corresponding instruction text to obtain the corresponding fifth task instruction; input the fifth task instruction into the first model to perform molecular label column recognition task processing to obtain the corresponding fifth instruction feedback; extract the corresponding fifth recognition result and the first column extraction information from the fifth instruction feedback; and form a corresponding eighth prediction-label pair from the first column extraction information and the first label column information of the current evaluation record; and at the end of this round of traversal, form all the eighth prediction-label pairs obtained in this round of traversal into a corresponding eighth prediction-label pair set;

[0143] Step 1559, based on the eighth prediction-label pair, the accuracy, precision, recall and F1 score of the set are calculated to obtain the corresponding fifth accuracy, fifth precision, fifth recall and fifth F1 score; and the fifth accuracy, fifth precision, fifth recall and fifth F1 score are identified; if the fifth accuracy does not meet the preset fifth accuracy range or the fifth precision does not meet the preset fifth precision range or the fifth recall does not meet the preset fifth recall range or the fifth F1 score does not meet the preset fifth F1 score range, then return to step 1552 to continue training; if the fifth accuracy meets the fifth accuracy range and the fifth precision meets the fifth precision range and the fifth recall meets the fifth recall range and the fifth F1 score meets the fifth F1 score range, then stop this round of fine-tuning and confirm that the fifth task fine-tuning is completed;

[0144] Here, the fifth accuracy range, the fifth precision range, the fifth recall range, and the fifth F1 score range are four set value ranges;

[0145] Step 156: After the fifth task fine-tuning is completed, confirm that the first model fine-tuning is completed.

[0146] Step 2: After the fine-tuning of the first model is completed, the PDF file of the small molecule drug patent input by the user is received as the corresponding first patent file; and based on the preset PDF content extraction tool, the image, text block and table of the first patent file are extracted to obtain the corresponding first image information sequence, first text block information sequence and first table information sequence.

[0147] Here, PDF content extraction tools are a type of mature PDF file parsing tools / libraries, commonly used ones include PyPDF, pdfminer.six, PyMuPDF, pdfplumber, pypdfium2, Unstructured, Tabula, Camelot, etc.

[0148] The data of the first image information sequence, the first text block information sequence and the first table information sequence extracted by the PDF content extraction tool in the embodiment of the present invention are as follows: 1) The first image information sequence includes multiple first image information; the first image information includes a first identifier, a first page number, a first upper left corner row and column coordinate, a first upper right corner row and column coordinate, a first lower right corner row and column coordinate, a first lower left corner row and column coordinate and a first image; the first image is a visual diagram of a molecular structure. 2) The first text block information sequence includes multiple first text block information; the first text block information includes a second identifier, a second page number, a second upper left corner row and column coordinate, a second upper right corner row and column coordinate, a second lower right corner row and column coordinate, a second lower left corner row and column coordinate and a first block of text. 3) The first table information sequence includes multiple first table information; the first table information includes a third identifier, a third page number, a third upper left corner row and column coordinate, a third upper right corner row and column coordinate, a third lower right corner row and column coordinate, a third lower left corner row and column coordinate and a first table; the first table includes one or more first table rows; the first table row includes multiple first row unit texts.

[0149] Step 3, based on the first instruction template and the first model, the first image information sequence is subjected to molecular image screening processing to obtain the corresponding first molecular image information sequence; and based on the second instruction template and the first model, the first text block information sequence is subjected to molecular label screening processing to obtain the corresponding first molecular label information sequence; and based on the third instruction template and the first model, the first table information sequence is subjected to other table filtering processing to obtain the corresponding second table information sequence; and based on the fourth instruction template, the first model and the second table information sequence, the molecular activity table and the headerless table are spliced ​​to obtain the corresponding first spliced ​​table information sequence;

[0150] Specifically, it includes: step 31, performing molecular image screening processing on the first image information sequence based on the first instruction template and the first model to obtain a corresponding first molecular image information sequence;

[0151] Specifically, it includes: performing a round of traversal on all first images of the first image information sequence; and in the process of this round of traversal, inputting the currently traversed first image as the corresponding current image into the first instruction template to assemble the corresponding instruction text to obtain the corresponding current task instruction; and inputting the current task instruction into the first model to perform molecular image recognition task processing to obtain the corresponding first instruction feedback; and using the first recognition result in the first instruction feedback as the corresponding current recognition result; and when the current recognition result is no, deleting the first image information corresponding to the current image from the first image information sequence; and at the end of this round of traversal, using the latest first image information sequence as the corresponding first molecular image information sequence;

[0152] Step 32, performing molecular label screening processing on the first text block information sequence based on the second instruction template and the first model to obtain a corresponding first molecular label information sequence;

[0153] Specifically, it includes: performing a round of traversal of all first blocks of text in the first text block information sequence; and in this round of traversal, inputting the first block of text currently traversed as the corresponding current text into the second instruction template to assemble the corresponding instruction text to obtain the corresponding current task instruction; and inputting the current task instruction into the first model to perform molecular label recognition task processing to obtain the corresponding second instruction feedback; and using the second recognition result and the first molecular label extraction information fed back by the second instruction as the corresponding current recognition result and the current molecular label; and recognizing the current recognition result; if the current recognition result is, then resetting the text content of the first block of text corresponding to the current text in the first text block information sequence to the corresponding current molecular label; if the current recognition result is no, then deleting the first text block information corresponding to the current text from the first text block information sequence; and at the end of this round of traversal, using the latest first text block information sequence as the corresponding first molecular label information sequence;

[0154] Step 33, performing other table filtering processing on the first table information sequence based on the third instruction template and the first model to obtain a corresponding second table information sequence;

[0155] Specifically, it includes: performing a round of traversal on all first tables of the first table information sequence; and in the process of this round of traversal, inputting the currently traversed first table as the corresponding current table into the third instruction template to assemble the corresponding instruction text to obtain the corresponding current task instruction; and inputting the current task instruction into the first model to perform table classification task processing to obtain the corresponding third instruction feedback; and using the third recognition result of the third instruction as the corresponding current recognition result; and identifying the current recognition result; if the current recognition result is a molecular activity table, marking the first table information corresponding to the current table as the corresponding first-category table information; if the current recognition result is a table without a header, marking the first table information corresponding to the current table as the corresponding second-category table information; if the current recognition result is other tables, deleting the first table information corresponding to the current table from the first table information sequence; and at the end of this round of traversal, using the latest first table information sequence as the corresponding second table information sequence;

[0156] Step 34, based on the fourth instruction template, the first model and the second table information sequence, the molecular activity table and the headerless table are spliced ​​to obtain a corresponding first spliced ​​table information sequence;

[0157] Wherein, the first spliced ​​table information sequence includes a plurality of first spliced ​​table information; the first spliced ​​table information includes a fourth identifier, a leading page number, a trailing page number, a leading page upper left corner row and column coordinates, a leading page upper right corner row and column coordinates, a leading page lower right corner row and column coordinates, a leading page lower left corner row and column coordinates, a trailing page upper left corner row and column coordinates, a trailing page upper right corner row and column coordinates, a trailing page lower right corner row and column coordinates, a trailing page lower left corner row and column coordinates and a first spliced ​​table; the first spliced ​​table includes one or more second table rows; the second table row includes a plurality of second row unit texts;

[0158] Specifically, it includes: step 341, initializing an empty sequence as the corresponding third table information sequence; and taking the first first table information of the second table information sequence as the corresponding current table information;

[0159] Step 342, whether the current table information is marked as a type of table information is identified; if the current table information is marked as a type of table information, then whether the row coordinates of the third lower left corner row and column coordinates of the current table information are the bottom row of the current page is identified, if so, go to step 343, if not, go to step 344; if the current table information is not marked as a type of table information, go to step 345;

[0160] Step 343, add 1 to the third page number of the current table information to obtain the corresponding next page number; and use the second type of table information in which each third page number in the second table information sequence matches the next page number and the row coordinates of the third upper left corner row and column coordinates are the top row of the current page as the corresponding candidate table information; and identify whether the candidate table information is empty; if the candidate table information is empty, go to step 344; if the candidate table information is not empty, record the first table of the candidate table information as the corresponding next table, and record the first table of the current table information as the corresponding previous table, and count the total number of the first row of unit texts of any first table row of the previous table to obtain the corresponding total number of first units, and count the number of the next table information. The total number of first row unit texts of any first table row is counted to obtain the corresponding second unit total number, and whether the first and second unit total numbers are equal is identified; if the first and second unit total numbers are not equal, go to step 344; if the first and second unit total numbers are equal, the second first table row of the front table and the first first table row of the back table are used as the corresponding first and second data rows and input into the fourth instruction template to perform corresponding instruction text assembly to obtain the corresponding current task instruction, and the current task instruction is input into the first model to perform data row similarity recognition task processing to obtain the corresponding fourth instruction feedback, and the fourth recognition result in the fourth instruction feedback is used as the corresponding current recognition result, and The current recognition result is recognized; if the current recognition result is not similar, go to step 344; if the current recognition result is similar, add a first spliced ​​table information with empty content in the third table information sequence as the corresponding current spliced ​​table information, and assign a unique identifier to the current spliced ​​table information as the corresponding current spliced ​​table identifier, and add the post-table to the end of the pre-table to obtain the corresponding current spliced ​​table, and set the fourth identifier of the current spliced ​​table information to the corresponding current spliced ​​table identifier, and set the pre- and post-page numbers of the current spliced ​​table information to the third page numbers corresponding to the pre- and post-tables, and set the row and column coordinates of the upper left corner of the pre-page of the current spliced ​​table information, the pre-page The upper right corner row and column coordinates, the lower right corner row and column coordinates of the front page, and the lower left corner row and column coordinates of the front page are set to the third upper left corner row and column coordinates, the third upper right corner row and column coordinates, the third lower right corner row and column coordinates, and the third lower left corner row and column coordinates of the rear page upper left corner row and column coordinates, the rear page upper right corner row and column coordinates, the rear page lower right corner row and column coordinates, and the rear page lower left corner row and column coordinates of the current splicing table information are set to the third upper left corner row and column coordinates, the third upper right corner row and column coordinates, the third lower right corner row and column coordinates, and the third lower left corner row and column coordinates of the rear table information, and the first splicing table of the current splicing table information is set to the corresponding current splicing table, and after the setting of the current splicing table information is completed, go to step 345;

[0161] Step 344, record the first table of the current table information as the corresponding current table; add a first spliced ​​table information with empty content in the third table information sequence as the corresponding current spliced ​​table information; assign a unique identifier to the current spliced ​​table information as the corresponding current spliced ​​table identifier; set the fourth identifier of the current spliced ​​table information to the corresponding current spliced ​​table identifier; set the leading page number of the current spliced ​​table information to the third page number corresponding to the current table; set the following page number of the current spliced ​​table information to empty; and The row and column coordinates of the upper left corner, the row and column coordinates of the upper right corner, the row and column coordinates of the lower right corner, and the row and column coordinates of the lower left corner of the preceding page of the spliced ​​table information are set to the third row and column coordinates of the upper left corner, the third row and column coordinates of the upper right corner, the third row and column coordinates of the lower right corner, and the third row and column coordinates of the lower left corner corresponding to the current table; and the row and column coordinates of the upper left corner, the row and column coordinates of the upper right corner, the row and column coordinates of the lower right corner, and the row and column coordinates of the lower left corner of the following page of the current spliced ​​table information are all set to empty; and the first spliced ​​table of the current spliced ​​table information is set to the corresponding current table;

[0162] Step 345, identify whether the current table information is the last first table information of the second table information sequence; if not, extract the next first table information of the second table information sequence as the new current table information and return to step 342; if so, output the latest first splicing table information sequence as the result of this splicing processing.

[0163] Step 4, constructing a label-image correspondence table based on the first molecule image information sequence, the first molecule label information sequence, the first splicing table information sequence, the fifth instruction template and the first model; and constructing a label-activity information correspondence table based on the fifth instruction template, the first model and the first splicing table information sequence; and merging multimodal tables based on the label-image correspondence table and the label-activity information correspondence table to obtain a corresponding multimodal comprehensive information table;

[0164] Specifically comprising: step 41, constructing a label-image correspondence table based on the first molecule image information sequence, the first molecule label information sequence, the first splicing table information sequence, the fifth instruction template and the first model;

[0165] Specifically, it includes: step 411, taking the first first image of the first molecular image information sequence as the corresponding current molecular image;

[0166] Step 412, taking the first page number corresponding to the current molecular image as the corresponding current image page number, and recording the area covered by the first upper left corner row and column coordinates, the first upper right corner row and column coordinates, the first lower right corner row and column coordinates and the first lower left corner row and column coordinates corresponding to the current molecular image as the corresponding current image area;

[0167] Step 413, record the first spliced ​​table information in the first spliced ​​table information sequence whose preceding page number or succeeding page number matches the current image page number as the corresponding first same-page table information; and based on the row and column coordinates of the upper left corner of the preceding / successing page, the row and column coordinates of the upper right corner of the preceding / successing page, the row and column coordinates of the lower right corner of the preceding / successing page, and the row and column coordinates of the lower left corner of the preceding / successing page of each first same-page table information, confirm the coverage of the current first same-page table information on the current image page number to obtain the corresponding first same-page table area; and when the number of first same-page table areas is not zero, confirm whether there is a first same-page table area that intersects with the current image area to obtain the corresponding first confirmation result; and identify the first confirmation result; if the first confirmation result is not present, go to step 414; if the first confirmation result is present, take the first same-page table information corresponding to the first same-page table area that intersects with the current image area as the corresponding current same-page table information, and go to step 415;

[0168] Step 414, record the first text block information whose second page number matches the current image page number in the first molecule label information sequence as the corresponding first same-page text block information; and confirm the coverage of the current first same-page text block information on the current image page number based on the second upper left corner row and column coordinates, the second upper right corner row and column coordinates, the second lower right corner row and column coordinates and the second lower left corner row and column coordinates of each first same-page text block information to obtain the corresponding first same-page text block area; and take the first block of text of the first text block information corresponding to the first same-page text block area closest to the current image area as the corresponding current molecule label; and go to step 416;

[0169] Step 415, input the first spliced ​​table of the current same-page table information into the fifth instruction template to assemble the corresponding instruction text to obtain the corresponding current task instruction; input the current task instruction into the first model to process the molecular label column recognition task to obtain the corresponding fifth instruction feedback; extract the corresponding first column extraction information from the fifth instruction feedback; record the full table coverage area of ​​the single page or double pages of the current same-page table information as the corresponding first table area; record the area covered by the first upper left corner row and column coordinates, the first upper right corner row and column coordinates, the first lower right corner row and column coordinates and the first lower left corner row and column coordinates of each first molecular image in the first molecular image information sequence as the corresponding first image area; record the first image area that has an intersection with the first table area as the corresponding first in-table image area, and sort all the first in-table image areas in the order of the intersection area from front to back to form a corresponding first in-table image area sequence; and use the sequence index corresponding to the first in-table image area that matches the current image area in the first in-table image area sequence as the corresponding current row index; and extract the molecular label information of the first column unit information whose row index information in the first column extraction information matches the current row index as the corresponding current molecular label;

[0170] Step 416, taking the current molecular image as a corresponding first molecular image field; taking the current molecular number as a corresponding first molecular number field; and forming a corresponding first corresponding relationship record by the obtained first molecular number field and the first molecular image field;

[0171] Step 417, identifying whether the current molecular image is the last first molecular image of the first molecular image information sequence; if not, extracting the next first molecular image of the first molecular image information sequence as the new current molecular image, and returning to step 412; if so, composing a corresponding label-image correspondence table with all the obtained first correspondence records;

[0172] Here, the label-image correspondence table of the embodiment of the present invention includes a plurality of first correspondence records; the first correspondence record includes a first molecule label field and a first molecule image field;

[0173] Step 42, constructing a label-activity information correspondence table based on the fifth instruction template, the first model and the first concatenated table information sequence;

[0174] Specifically including: step 421, extracting the first second table rows of all the first spliced ​​tables of the first spliced ​​table information sequence to form a corresponding second table row set; extracting all the second row unit texts of the second table row set to form a corresponding second row unit text set; performing unit text deduplication processing on the second row unit text set; and performing a round of traversal on all the second row unit texts of the second row unit text set that has been deduplicated; and in this round of traversal, taking the currently traversed second row unit text as the corresponding current field name; and based on a preset molecular activity keyword dictionary, identifying whether the current field name is a type of molecular activity name to obtain a corresponding activity name recognition result; and when the activity name recognition result is yes, marking the current field name as a corresponding molecular activity field name; and at the end of this round of traversal, all the obtained molecular activity field names form a corresponding molecular activity field name set;

[0175] Step 422, perform a round of traversal of all the first splicing tables of the first splicing table information sequence; and in this round of traversal, use the currently traversed first splicing table as the corresponding current splicing table; input the current splicing table into the fifth instruction template to assemble the corresponding instruction text to obtain the corresponding current task instruction; input the current task instruction into the first model to perform molecular label column identification task processing to obtain the corresponding fifth instruction feedback; extract the corresponding first column extraction information from the fifth instruction feedback; use the first second table row of the current splicing table as the corresponding current header row; and use the second row of unit text corresponding to the first column extraction information in the current header row as a corresponding The molecular label field name; and poll all the second-row cell texts except the molecular label field name in the current header row; and in this polling process, the second-row cell text currently polled is recorded as the corresponding current field name, and it is identified whether the current field name is included in the molecular activity field name set. If so, the second-row cell text currently polled is used as a corresponding molecular activity field name, if not, the arrangement index of the second-row cell text corresponding to the current field name in the current header row is recorded as the corresponding column index to be deleted; and at the end of this polling, the second-row cell texts corresponding to all the column indexes to be deleted in all the second table rows of the current spliced ​​table are deleted;

[0176] Step 423, after the current round of traversal of all first concatenated tables of the first concatenated table information sequence is completed, all field contents of all first concatenated tables are fully connected and merged using the molecular label field name as the connection key field name to obtain a new table with the molecular label field and all molecular activity fields as the corresponding label-activity information correspondence table;

[0177] Here, the data structure of the label-activity information correspondence table obtained in the embodiment of the present invention is: the label-activity information correspondence table includes multiple second correspondence records; each second correspondence record includes a second molecule label field and multiple types of first molecule activity fields;

[0178] Step 43, merging the multimodal tables based on the label-image correspondence table and the label-activity information correspondence table to obtain a corresponding multimodal comprehensive information table;

[0179] Specifically, it includes: using the molecular label field as the connection key field to fully connect and merge the label-image correspondence table and the label-activity information correspondence table to obtain a new table with a molecular label field, a molecular image field and all molecular activity fields as the corresponding multimodal comprehensive information table.

[0180] Here, the data structure of the multimodal comprehensive information table obtained by the embodiment of the present invention is: the multimodal comprehensive information table includes multiple third correspondence records; each third correspondence record includes a third molecule label field, a second molecule image field and multiple types of second molecule activity fields.

[0181] Step 5: Feedback a first mining report corresponding to the first patent document and the multimodal comprehensive information table to the current user.

[0182] Figure 2 The module structure diagram of a processing device for mining patent information in combination with a large language model provided in the second embodiment of the present invention, the device is a terminal device or server that implements the aforementioned method embodiment, and can also be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiment, for example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 2 As shown, the device includes: a large language model fine-tuning module 201, a patent information receiving module 202, a patent information noise reduction module 203, a patent information mining module 204, and a mining report feedback module 205.

[0183] The large language model fine-tuning module 201 is used to select a type of pre-trained multimodal large language model as the corresponding first model; and specify five target tasks for the first model, recorded as the corresponding first, second, third, fourth and fifth tasks; and customize instruction text templates for the first, second, third, fourth and fifth tasks, recorded as the corresponding first, second, third, fourth and fifth instruction templates; and construct five task data sets by collecting public small molecule drug patent literature information, recorded as the corresponding first, second, third, fourth and fifth data sets; and fine-tune the first model based on the first, second, third, fourth and fifth data sets and the corresponding first, second, third, fourth and fifth instruction templates; the first task is to perform molecular image recognition on the input image; the second task is to perform molecular label recognition on the input text; the third task is to classify the input table; the fourth task is to perform similarity recognition on two input data rows, and the total number of data fields in the two data rows is consistent; the fifth task is to perform molecular label column recognition on the input table.

[0184] The patent information receiving module 202 is used to receive the PDF file of the small molecule drug patent input by the user as the corresponding first patent file after the first model fine-tuning is completed; and based on the preset PDF content extraction tool, extract the image, text block and table of the first patent file to obtain the corresponding first image information sequence, first text block information sequence and first table information sequence.

[0185] The patent information denoising module 203 is used to perform molecular image screening processing on the first image information sequence based on the first instruction template and the first model to obtain the corresponding first molecular image information sequence; and to perform molecular label screening processing on the first text block information sequence based on the second instruction template and the first model to obtain the corresponding first molecular label information sequence; and to perform other table filtering processing on the first table information sequence based on the third instruction template and the first model to obtain the corresponding second table information sequence; and to perform molecular activity table and headerless table splicing processing based on the fourth instruction template, the first model and the second table information sequence to obtain the corresponding first spliced ​​table information sequence.

[0186] The patent information mining module 204 is used to construct a label-image correspondence table based on the first molecular image information sequence, the first molecular label information sequence, the first splicing table information sequence, the fifth instruction template and the first model; and to construct a label-activity information correspondence table based on the fifth instruction template, the first model and the first splicing table information sequence; and to merge multimodal tables based on the label-image correspondence table and the label-activity information correspondence table to obtain a corresponding multimodal comprehensive information table.

[0187] The mining report feedback module 205 is used to feed back the first mining report corresponding to the first patent document and the multimodal comprehensive information table to the current user.

[0188] An embodiment of the present invention provides a processing device for mining patent information in combination with a large language model, which can execute the method steps in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0189] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the large language model fine-tuning module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called by a processing element of the above device and execute the function of the above-determined module. The implementation of other modules is similar. In addition, these modules can be integrated together in whole or in part, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in software form.

[0190] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0191] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the above method embodiments are generated. The above computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) methods. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0192] Figure 3 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. The electronic device may be a terminal device or a server for implementing the method of the aforementioned embodiment, or may be a terminal device or a server for implementing the method of the aforementioned embodiment connected to the aforementioned terminal device or server. Figure 3 As shown, the electronic device may include: a processor 301 (such as a CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiver 303. Various instructions may be stored in the memory 302 to complete various processing functions and implement the processing steps described in the aforementioned embodiment method. Preferably, the electronic device involved in the embodiment of the present invention also includes: a power supply 304, a system bus 305 and a communication port 306. The system bus 305 is used to realize the communication connection between components. The above-mentioned communication port 306 is used for connecting and communicating between the electronic device and other peripherals.

[0193] exist Figure 3The system bus 305 mentioned in the figure can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM), and may also include non-volatile memory (Non-Volatile Memory), such as at least one disk storage.

[0194] The above-mentioned processor can be a general-purpose processor, including a central processing unit CPU, a network processor (Network Processor, NP), a graphics processing unit (Graphics Processing Unit, GPU), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0195] It should be noted that an embodiment of the present invention further provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the method and processing process provided in the above embodiments.

[0196] The embodiment of the present invention provides a processing method, device, electronic device and computer-readable storage medium for mining patent information in combination with a large language model. As can be seen from the above content, the embodiment of the present invention selects a type of pre-trained multimodal large language model as the first model, and specifies five target tasks for it (molecular image recognition of input images, molecular label recognition of input text, classification of input tables, similarity recognition of two input data rows, and molecular label column recognition of input tables), customizes five instruction text templates, constructs five data sets, and fine-tunes the first model for five types of tasks based on the five data sets and the corresponding five instruction text templates; after the model fine-tuning is completed, the PDF file of the small molecule drug patent input by the user is received, and the patent document is extracted based on the PDF content extraction tool. Image, text block and table extraction; then, the first image information sequence is molecularly imaged based on the first instruction template and the first model. Image screening, molecular label screening of the first text block information sequence based on the second instruction template and the first model, other table filtering of the first table information sequence based on the third instruction template and the first model, molecular activity table and headerless table splicing based on the fourth instruction template, the first model and the second table information sequence; then, a label-image correspondence table is constructed based on the first molecular image information sequence, the first molecular label information sequence, the first spliced ​​table information sequence, the fifth instruction template and the first model, and a label-activity information correspondence table is constructed based on the fifth instruction template, the first model and the first spliced ​​table information sequence; finally, multimodal tables are merged based on the label-image correspondence table and the label-activity information correspondence table to obtain the corresponding multimodal comprehensive information table and feedback to the current user. The patent information mining mechanism provided in the embodiment of the present invention can perform molecular activity data mining for small molecule drug patents. Based on the embodiment of the present invention, not only the data mining efficiency is improved, the data mining cost is reduced, but also the research and development efficiency is improved.

[0197] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0198] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for mining patent information in combination with a large language model, characterized in that: The method comprises: A type of pre-trained multimodal large language model is selected as the corresponding first model; and five target tasks are specified for the first model, recorded as the corresponding first, second, third, fourth and fifth tasks; and the instruction text templates are customized for the first, second, third, fourth and fifth tasks, recorded as the corresponding first, second, third, fourth and fifth instruction templates; and five task data sets are constructed by collecting public small molecule drug patent literature information, recorded as the corresponding first, second, third, fourth and fifth data sets; and the first model is fine-tuned based on the first, second, third, fourth and fifth data sets and the corresponding first, second, third, fourth and fifth instruction templates; the first task is to perform molecular image recognition on the input image; the second task is to perform molecular label recognition on the input text; the third task is to classify the input table; the fourth task is to perform similarity recognition on two input data rows, and the total number of data fields in the two data rows is consistent; the fifth task is to perform molecular label column recognition on the input table; After the fine-tuning of the first model is completed, a PDF file of a small molecule drug patent input by a user is received as a corresponding first patent file; and images, text blocks and tables are extracted from the first patent file based on a preset PDF content extraction tool to obtain a corresponding first image information sequence, a first text block information sequence and a first table information sequence; Based on the first instruction template and the first model, the first image information sequence is subjected to molecular image screening processing to obtain a corresponding first molecular image information sequence; and based on the second instruction template and the first model, the first text block information sequence is subjected to molecular label screening processing to obtain a corresponding first molecular label information sequence; and based on the third instruction template and the first model, the first table information sequence is subjected to other table filtering processing to obtain a corresponding second table information sequence; and based on the fourth instruction template, the first model and the second table information sequence, the molecular activity table and the headerless table are spliced ​​to obtain a corresponding first spliced ​​table information sequence; Based on the first molecular image information sequence, the first molecular label information sequence, the first splicing table information sequence, the fifth instruction template and the first model, a label-image correspondence table is constructed; and based on the fifth instruction template, the first model and the first splicing table information sequence, a label-activity information correspondence table is constructed; and based on the label-image correspondence table and the label-activity information correspondence table, a multimodal table is merged to obtain a corresponding multimodal comprehensive information table; A first mining report corresponding to the first patent document and the multimodal comprehensive information table is fed back to the current user.

2. The method for mining patent information in combination with a large language model according to claim 1, characterized in that: The first model at least includes a GPT series model, an LLaMA series model, a Qwen series model, and a Wenxin series model; The first, second, third, fourth and fifth instruction templates are each a formatted instruction text module; The first instruction template is used to use the input image as the corresponding current image; and instructing the first model to identify whether the current image is a molecular structure diagram by a natural language description text to obtain a corresponding first recognition result, and outputting the first recognition result as a corresponding first instruction feedback; the first recognition result includes yes and no; The second instruction template is used to take the input text as the corresponding current context; and instructing the first model to identify whether the current context contains molecular label information by a natural language description text to obtain a corresponding second recognition result, and further extracting the molecular label information from the current context to obtain corresponding first molecular label extraction information when the second recognition result is yes, and setting the corresponding first molecular label extraction information to be empty when the second recognition result is no, and forming a corresponding second instruction feedback output with the obtained second recognition result and the first molecular label extraction information; the second recognition result includes yes and no; The third instruction template is used to take the input table as the corresponding current table; and to indicate the first model through a natural language description text to identify which category the current table belongs to, namely, a molecular activity table, a table without a header, and other tables, and output the third recognition result as the corresponding third instruction feedback; the third recognition result includes a molecular activity table, a table without a header, and other tables; The fourth instruction template is used to take the two input data rows as the corresponding current data row A and current data row B; and instruct the first model through a natural language description text to perform field-by-field data similarity estimation on the current data rows A and B according to the data field correspondence to obtain the corresponding field similarity, and calculate the average similarity based on all the obtained field similarities, and identify whether each of the field similarities and the average similarity are higher than a preset similarity threshold, if both are higher than the similarity threshold, set the corresponding fourth recognition result as similar, if at least one of the field similarities or the average similarity is lower than the similarity threshold, set the corresponding fourth recognition result as dissimilar, and output the fourth recognition result as the corresponding fourth instruction feedback; the fourth recognition result includes similarity and dissimilarity; The fifth instruction template is used to use the input form as the corresponding current form; And through a natural language description text, instruct the first model to identify whether there is a molecular label column in the current table to obtain a corresponding fifth recognition result, and when the fifth recognition result is yes, further perform molecular label column information extraction processing on the current table to obtain corresponding first column extraction information, and when the fifth recognition result is no, set the corresponding first column extraction information to be empty, and the obtained fifth recognition result and the first column extraction information constitute a corresponding fifth instruction feedback output; the fifth recognition result includes yes and no; when the first column extraction information is not empty, it is composed of one or more first column unit information; the first column unit information includes row index information and molecular label information; The first image information sequence includes a plurality of first image information; the first image information includes a first identifier, a first page number, a first upper left corner row and column coordinate, a first upper right corner row and column coordinate, a first lower right corner row and column coordinate, a first lower left corner row and column coordinate and a first image; the first image is a visible image of a molecular structure; The first text block information sequence includes a plurality of first text block information; the first text block information includes a second identifier, a second page number, a second upper left corner row and column coordinate, a second upper right corner row and column coordinate, a second lower right corner row and column coordinate, a second lower left corner row and column coordinate and a first block of text; The first table information sequence includes a plurality of first table information; the first table information includes a third identifier, a third page number, a third upper left corner row and column coordinate, a third upper right corner row and column coordinate, a third lower right corner row and column coordinate, a third lower left corner row and column coordinate and a first table; the first table includes one or more first table rows; the first table row includes a plurality of first row unit texts; The first spliced ​​table information sequence includes a plurality of first spliced ​​table information; the first spliced ​​table information includes a fourth identifier, a leading page number, a trailing page number, a leading page upper left corner row and column coordinates, a leading page upper right corner row and column coordinates, a leading page lower right corner row and column coordinates, a leading page lower left corner row and column coordinates, a trailing page upper left corner row and column coordinates, a trailing page upper right corner row and column coordinates, a trailing page lower right corner row and column coordinates, a trailing page lower left corner row and column coordinates, and a first spliced ​​table; the first spliced ​​table includes one or more second table rows; the second table rows include a plurality of second row unit texts; The label-image correspondence table includes a plurality of first correspondence records; the first correspondence record includes a first molecule label field and a first molecule image field; The label-activity information correspondence table includes a plurality of second correspondence records; the second correspondence records include a second molecule label field and a plurality of first molecule activity fields; The multimodal comprehensive information table includes a plurality of third correspondence records; the third correspondence records include the third molecule label field, the second molecule image field and multiple types of second molecule activity fields.

3. The method for mining patent information in combination with a large language model according to claim 2, characterized in that: The performing molecular image screening processing on the first image information sequence based on the first instruction template and the first model to obtain a corresponding first molecular image information sequence specifically includes: A round of traversal is performed on all the first images of the first image information sequence; and during this round of traversal, the first image currently traversed is used as the corresponding current image and input into the first instruction template for corresponding instruction text assembly to obtain the corresponding current task instruction; and the current task instruction is input into the first model for molecular image recognition task processing to obtain the corresponding first instruction feedback; and the first recognition result in the first instruction feedback is used as the corresponding current recognition result; and when the current recognition result is no, the first image information corresponding to the current image is deleted from the first image information sequence; and at the end of this round of traversal, the latest first image information sequence is used as the corresponding first molecular image information sequence.

4. The method for mining patent information in combination with a large language model according to claim 2, characterized in that: The performing molecular label screening processing on the first text block information sequence based on the second instruction template and the first model to obtain the corresponding first molecular label information sequence specifically includes: A round of traversal is performed on all the first blocks of text in the first text block information sequence; and in this round of traversal, the first block of text currently traversed is used as the corresponding current text to input the second instruction template for corresponding instruction text assembly to obtain the corresponding current task instruction; and the current task instruction is input into the first model for molecular label recognition task processing to obtain the corresponding second instruction feedback; and the second recognition result and the first molecular label extraction information fed back by the second instruction are used as the corresponding current recognition result and current molecular label; and the current recognition result is recognized; if the current recognition result is, the text content of the first block of text corresponding to the current text in the first text block information sequence is reset to the corresponding current molecular label; if the current recognition result is no, the first text block information corresponding to the current text is deleted from the first text block information sequence; and at the end of this round of traversal, the latest first text block information sequence is used as the corresponding first molecular label information sequence.

5. The method for mining patent information in combination with a large language model according to claim 2, characterized in that: The performing other table filtering processing on the first table information sequence based on the third instruction template and the first model to obtain the corresponding second table information sequence specifically includes: A round of traversal is performed on all the first tables in the first table information sequence; and in this round of traversal, the first table currently traversed is used as the corresponding current table and input into the third instruction template to assemble the corresponding instruction text to obtain the corresponding current task instruction; and the current task instruction is input into the first model to perform table classification task processing to obtain the corresponding third instruction feedback; and the third recognition result of the third instruction is used as the corresponding current recognition result; and the current recognition result is recognized; if the current recognition result is a molecular activity table, the first table information corresponding to the current table is marked as the corresponding first type of table information; if the current recognition result is a table without a header, the first table information corresponding to the current table is marked as the corresponding second type of table information; if the current recognition result is other tables, the first table information corresponding to the current table is deleted from the first table information sequence; and at the end of this round of traversal, the latest first table information sequence is used as the corresponding second table information sequence.

6. The method for mining patent information in combination with a large language model according to claim 5, characterized in that: The step of performing splicing of the molecular activity table and the headerless table based on the fourth instruction template, the first model and the second table information sequence to obtain a corresponding first spliced ​​table information sequence specifically includes: Step 61, initializing an empty sequence as the corresponding third table information sequence; and taking the first of the first table information in the second table information sequence as the corresponding current table information; Step 62, identifying whether the current table information is marked as the type of table information; if the current table information is marked as the type of table information, identifying whether the row coordinates of the third lower left corner row and column coordinates of the current table information are the bottom row of the current page, if so, go to step 63, if not, go to step 64; if the current table information is not marked as the type of table information, go to step 65; Step 63, add 1 to the third page number of the current table information to obtain the corresponding next page number; and use the second type of table information in which each of the third page numbers in the second table information sequence matches the next page number and the row coordinates of the third upper left corner row and column coordinates are the top row of the current page as the corresponding candidate table information; and identify whether the candidate table information is empty; if the candidate table information is empty, go to step 64; if the candidate table information is not empty, record the first table of the candidate table information as the corresponding next table, and record the first table of the current table information as the corresponding previous table, and count the total number of the first row of unit texts of any one of the first table rows of the previous table to obtain the corresponding first unit total number, and count the The total number of the first row of unit text is counted to obtain the corresponding total number of the second unit, and whether the total number of the first and second units are equal is identified; if the total number of the first and second units are not equal, go to step 64; if the total number of the first and second units are equal, the second first table row of the front table and the first first table row of the back table are used as the corresponding first and second data rows to input the fourth instruction template to perform corresponding instruction text assembly to obtain the corresponding current task instruction, and the current task instruction is input into the first model to perform data row similarity recognition task processing to obtain the corresponding fourth instruction feedback, and the fourth recognition result in the fourth instruction feedback is used as the corresponding current recognition result, and the current recognition result is identified; if the current recognition result is not similar, go to step 64;If the current recognition result is similar, then a new first spliced ​​table information with empty content is added to the third table information sequence as the corresponding current spliced ​​table information, and a unique identifier is assigned to the current spliced ​​table information as the corresponding current spliced ​​table identifier, and the post-table is added to the end of the pre-table to obtain the corresponding current spliced ​​table, and the fourth identifier of the current spliced ​​table information is set to the corresponding current spliced ​​table identifier, and the pre- and post-page numbers of the current spliced ​​table information are set to the third page numbers corresponding to the pre- and post-tables, and the row and column coordinates of the upper left corner of the pre-page, the row and column coordinates of the upper right corner of the pre-page, and the row and column coordinates of the lower right corner of the pre-page of the current spliced ​​table information are set to the corresponding current spliced ​​table identifier. and the lower left corner row and column coordinates of the leading page are set to the third upper left corner row and column coordinates, the third upper right corner row and column coordinates, the third lower right corner row and column coordinates and the third lower left corner row and column coordinates corresponding to the leading table, and the upper left corner row and column coordinates, the upper right corner row and column coordinates, the lower right corner row and column coordinates and the lower left corner row and column coordinates of the trailing page of the current splicing table information are set to the third upper left corner row and column coordinates, the third upper right corner row and column coordinates, the third lower right corner row and column coordinates and the third lower left corner row and column coordinates corresponding to the trailing table, and the first splicing table of the current splicing table information is set to the corresponding current splicing table, and after the setting of the current splicing table information is completed, go to step 65; Step 64, record the first table of the current table information as the corresponding current table; add a new first spliced ​​table information with empty content in the third table information sequence as the corresponding current spliced ​​table information; assign a unique identifier to the current spliced ​​table information as the corresponding current spliced ​​table identifier; set the fourth identifier of the current spliced ​​table information to the corresponding current spliced ​​table identifier; set the leading page number of the current spliced ​​table information to the third page number corresponding to the current table; set the following page number of the current spliced ​​table information to empty; and The upper left corner row and column coordinates, the upper right corner row and column coordinates, the lower right corner row and column coordinates and the lower left corner row and column coordinates of the preceding page of the grid information are set to the third upper left corner row and column coordinates, the third upper right corner row and column coordinates, the third lower right corner row and column coordinates and the third lower left corner row and column coordinates corresponding to the current table; and the upper left corner row and column coordinates, the upper right corner row and column coordinates, the lower right corner row and column coordinates and the lower left corner row and column coordinates of the following page of the current spliced ​​table information are all set to null; and the first spliced ​​table of the current spliced ​​table information is set to the corresponding current table; Step 65, identifying whether the current table information is the last one of the first table information in the second table information sequence; if not, extracting the next one of the first table information in the second table information sequence as the new current table information and returning to step 62; if so, outputting the latest first splicing table information sequence as the result of this splicing processing.

7. The method for mining patent information in combination with a large language model according to claim 2, characterized in that: The step of constructing a label-image correspondence table based on the first molecular image information sequence, the first molecular label information sequence, the first splicing table information sequence, the fifth instruction template and the first model specifically includes: Step 71, taking the first of the first molecular image information sequences as the corresponding current molecular image; Step 72, taking the first page number corresponding to the current molecular image as the corresponding current image page number, and recording the area covered by the first upper left corner row and column coordinates, the first upper right corner row and column coordinates, the first lower right corner row and column coordinates and the first lower left corner row and column coordinates corresponding to the current molecular image as the corresponding current image area; Step 73, record the first spliced ​​table information in which the preceding page number or the succeeding page number in the first spliced ​​table information sequence matches the current image page number as the corresponding first same-page table information; and based on the row and column coordinates of the upper left corner of the preceding / successing page, the row and column coordinates of the upper right corner of the preceding / successing page, the row and column coordinates of the lower right corner of the preceding / successing page, and the row and column coordinates of the lower left corner of the preceding / successing page of each of the first same-page table information, confirm the coverage of the current first same-page table information on the current image page number to obtain the corresponding first same-page table area; and when the number of the first same-page table areas is not zero, confirm whether there is one of the first same-page table areas that intersects with the current image area to obtain the corresponding first confirmation result; and identify the first confirmation result; if the first confirmation result is not present, go to step 74; if the first confirmation result is present, take the first same-page table information corresponding to the first same-page table area that currently intersects with the current image area as the corresponding current same-page table information, and go to step 75; Step 74, record the first text block information whose second page number in the first molecule label information sequence matches the current image page number as the corresponding first same-page text block information; and confirm the coverage of the current first same-page text block information on the current image page number based on the second upper left corner row and column coordinates, the second upper right corner row and column coordinates, the second lower right corner row and column coordinates and the second lower left corner row and column coordinates of each of the first same-page text block information to obtain the corresponding first same-page text block area; and take the first block of text of the first text block information corresponding to the first same-page text block area that is closest to the current image area as the corresponding current molecule label; and go to step 76; Step 75, input the first spliced ​​table of the current same-page table information into the fifth instruction template for corresponding instruction text assembly to obtain the corresponding current task instruction; and input the current task instruction into the first model for molecular label column identification task processing to obtain the corresponding fifth instruction feedback; and extract the corresponding first column extraction information from the fifth instruction feedback; and record the full table coverage area of ​​the single page or double pages of the current same-page table information as the corresponding first table area; and record the first upper left corner row and column coordinates, the first upper right corner row and column coordinates, the first lower right corner row and column coordinates and the first molecular image information sequence of each of the first molecular images. The area covered by the first lower left corner row and column coordinates is recorded as the corresponding first image area; the first image area that intersects with the first table area is recorded as the corresponding first in-table image area, and all the first in-table image areas are sorted in order from front to back in the intersection area to form a corresponding first in-table image area sequence; the sequence index corresponding to the first in-table image area that matches the current image area in the first in-table image area sequence is used as the corresponding current row index; and the molecular label information of the first column unit information whose row index information in the first column extraction information matches the current row index is extracted as the corresponding current molecular label; Step 76, taking the current molecular image as a corresponding first molecular image field; taking the current molecular label as a corresponding first molecular label field; and forming a corresponding first corresponding relationship record by the obtained first molecular label field and the first molecular image field; Step 77, identifying whether the current molecular image is the last first molecular image of the first molecular image information sequence; if not, extracting the next first molecular image of the first molecular image information sequence as the new current molecular image, and returning to step 72; if so, forming the corresponding label-image correspondence table by all the first correspondence records obtained.

8. The method for mining patent information in combination with a large language model according to claim 2, characterized in that: The step of constructing a label-activity information correspondence table based on the fifth instruction template, the first model and the first concatenated table information sequence specifically includes: Step 81, extract the first second table row of all the first concatenated tables of the first concatenated table information sequence to form a corresponding second table row set; extract all the second row unit texts of the second table row set to form a corresponding second row unit text set; perform unit text deduplication processing on the second row unit text set; perform a round of traversal on all the second row unit texts of the second row unit text set that have completed deduplication; and in this round of traversal, use the second row unit text currently traversed as the corresponding current field name; and based on the preset molecular activity keyword dictionary, identify whether the current field name is a type of molecular activity name to obtain a corresponding activity name recognition result; and when the activity name recognition result is yes, mark the current field name as a corresponding molecular activity field name; and at the end of this round of traversal, all the obtained molecular activity field names form a corresponding molecular activity field name set; Step 82, perform a round of traversal of all the first splicing tables of the first splicing table information sequence; and in this round of traversal, use the first splicing table currently traversed as the corresponding current splicing table; and input the current splicing table into the fifth instruction template to assemble the corresponding instruction text to obtain the corresponding current task instruction; and input the current task instruction into the first model to perform molecular label column identification task processing to obtain the corresponding fifth instruction feedback; and extract the corresponding first column extraction information from the fifth instruction feedback; and use the first second table row of the current splicing table as the corresponding current header row; and use the second row unit text corresponding to the first column extraction information in the current header row as a corresponding Molecular label field name; and perform a poll on all the second-row unit texts except the molecular label field name in the current header row; and during this polling process, record the second-row unit text currently polled as the corresponding current field name, and identify whether the current field name is included in the molecular activity field name set, if so, take the second-row unit text currently polled as a corresponding molecular activity field name, if not, record the arrangement index of the second-row unit text corresponding to the current field name in the current header row as the corresponding column index to be deleted; and at the end of this polling, delete the second-row unit texts corresponding to all the column indexes to be deleted in all the second table rows of the current spliced ​​table; Step 83, after the current round of traversal of all the first splicing tables in the first splicing table information sequence is completed, all the field contents of all the first splicing tables are fully connected and merged using the molecular label field name as the connection key field name to obtain a new table with a molecular label field and all molecular activity fields as the corresponding label-activity information correspondence table.

9. A device for executing the processing method for mining patent information in combination with a large language model as described in any one of claims 1 to 8, characterized in that: The device comprises: a large language model fine-tuning module, a patent information receiving module, a patent information noise reduction module, a patent information mining module, and a mining report feedback module; The large language model fine-tuning module is used to select a type of pre-trained multimodal large language model as the corresponding first model; and specify five target tasks for the first model, recorded as the corresponding first, second, third, fourth and fifth tasks; and customize instruction text templates for the first, second, third, fourth and fifth tasks, recorded as the corresponding first, second, third, fourth and fifth instruction templates; and construct five task data sets by collecting public small molecule drug patent literature information, recorded as the corresponding first, second, third, fourth and fifth data sets; and fine-tune the first model based on the first, second, third, fourth and fifth data sets and the corresponding first, second, third, fourth and fifth instruction templates; the first task is to perform molecular image recognition on the input image; the second task is to perform molecular label recognition on the input text; the third task is to classify the input table; the fourth task is to perform similarity recognition on two input data rows, and the total number of data fields in the two data rows is consistent; the fifth task is to perform molecular label column recognition on the input table; The patent information receiving module is used to receive the PDF file of the small molecule drug patent input by the user as the corresponding first patent file after the first model fine-tuning is completed; and extract the image, text block and table of the first patent file based on the preset PDF content extraction tool to obtain the corresponding first image information sequence, first text block information sequence and first table information sequence; The patent information denoising module is used to perform molecular image screening processing on the first image information sequence based on the first instruction template and the first model to obtain a corresponding first molecular image information sequence; and to perform molecular label screening processing on the first text block information sequence based on the second instruction template and the first model to obtain a corresponding first molecular label information sequence; and to perform other table filtering processing on the first table information sequence based on the third instruction template and the first model to obtain a corresponding second table information sequence; and to perform molecular activity table and headerless table splicing processing based on the fourth instruction template, the first model and the second table information sequence to obtain a corresponding first spliced ​​table information sequence; The patent information mining module is used to construct a label-image correspondence table based on the first molecular image information sequence, the first molecular label information sequence, the first splicing table information sequence, the fifth instruction template and the first model; and to construct a label-activity information correspondence table based on the fifth instruction template, the first model and the first splicing table information sequence; and to merge multimodal tables based on the label-image correspondence table and the label-activity information correspondence table to obtain a corresponding multimodal comprehensive information table; The mining report feedback module is used to feed back a first mining report corresponding to the first patent document and the multimodal comprehensive information table to the current user.

10. An electronic device, characterized in that: include: memory, processors, and transceivers; The processor is used to couple with the memory, read and execute instructions in the memory, so as to implement the method according to any one of claims 1 to 8; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer is enabled to execute the method according to any one of claims 1 to 8.

Images