Processing method and device based on multi-modal input document, equipment and storage medium
By extracting chart information in multimodal documents and combining visual language models and large language models, the problem of incomplete multimodal data processing in the existing technology is solved, and a deep understanding and accurate summary of multimodal documents are achieved. Improves the efficiency and accuracy of document processing.
Patent Information
- Application Number
- CN202411079713.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-12
AI Technical Summary
Existing natural language processing technologies are difficult to effectively combine and understand multimodal data, especially when processing documents containing charts and pictures, resulting in incomplete and inaccurate information processing.
By receiving multimodal input documents, extracting chart information and recording position information, inputting a visual language model to extract high-dimensional visual features, combining text information to generate target text input information, and inputting the target large language model to generate accurate text output.
It realizes a deep understanding and accurate summary of multimodal documents, generates customized text output, significantly improving the processing efficiency and accuracy of complex documents.
Smart Images

Figure CN118940742A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to a processing method, device, equipment and storage medium based on multimodal input documents. Background Art
[0002] Most existing natural language processing (NLP) technologies focus on processing plain text data. In practical applications, many documents (such as research reports, technical documents, business reports, etc.) usually contain information in multiple modes, including text, charts, pictures, etc. These multimodal input data need to be processed comprehensively in order to generate meaningful text descriptions and summaries. However, existing technologies have many deficiencies in processing such complex documents and cannot effectively combine and understand multimodal data, resulting in incomplete and inaccurate information processing.
[0003] In traditional NLP models, significant progress has been made in processing pure text data. For example, models such as BERT and GPT-3 have performed well in many language understanding tasks. However, these models are often unable to cope with documents containing non-text information such as charts and pictures. Charts and pictures contain a lot of visual information, which is crucial for understanding the content of the document. Models that rely solely on text information find it difficult to fully understand and utilize this visual information.
[0004] Some existing multimodal processing techniques attempt to combine visual and language models to process image and text information. However, these methods still have the following problems when processing complex documents:
[0005] Separation and fusion of chart and text information: When extracting chart information, traditional methods usually find it difficult to effectively combine the key information in the chart with the text content, resulting in a lack of coherence and accuracy in the generated text description.
[0006] Accuracy of visual feature extraction: Existing visual encoders are often limited by training data and model architecture when extracting high-dimensional visual features. They are unable to accurately capture key information in the chart, affecting the effect of multimodal data processing. Summary of the invention
[0007] The main purpose of the present invention is to provide a processing method, device, equipment and storage medium based on multimodal input documents, aiming to solve the technical problem that the existing language model cannot accurately identify non-text information when processing multimodal input.
[0008] To achieve the above object, the present invention provides a processing method based on multimodal input documents, the processing method based on multimodal input documents comprising the following steps:
[0009] receiving a multimodal input document, extracting chart information from the multimodal input document, and recording location information of the chart information;
[0010] Inputting the chart information into a visual language model to obtain chart text language information;
[0011] Target text input information is generated according to the graphic text language information and the position information, and the target text input information is input into a target large language model to obtain a target text output of the target large language model.
[0012] Optionally, the step of extracting chart information from the multimodal input document includes:
[0013] Identify the format of the input document and extract the chart in the input document;
[0014] The text information in the chart information is extracted, and the contents of the chart other than the text information are used as the chart information.
[0015] Optionally, the step of generating target text input information according to the graphic text language information and the position information includes:
[0016] The graphic text language information and the text information are embedded into the multimodal input document according to the position information to generate the target text input information.
[0017] Optionally, the step of inputting the target text input information into a target large language model includes:
[0018] Acquire user demand information, and generate demand prompts according to the user demand information, wherein the demand prompts are specific descriptions of the user demand information and are used to guide the target large language model to generate outputs that meet the user demand;
[0019] The requirement prompt and the target text input information are input into the target large language model.
[0020] Optionally, the step of inputting the chart information into a visual language model to obtain the chart text language information includes:
[0021] Inputting the chart information into the visual language model, wherein the visual language model extracts high-dimensional visual features from the chart information;
[0022] defining and generating a potential query vector, and fusing the high-dimensional visual feature with the potential query vector using an Attention mechanism to obtain a fused feature representation;
[0023] The fused feature representation is input into a decoder to obtain the graphic text language information.
[0024] Optionally, the step of extracting high-dimensional visual features from the chart information by the visual language model includes:
[0025] The visual language model includes a visual encoder, which is trained by the NFNet model through a contrastive learning task, and the visual encoder extracts high-dimensional visual features from the chart information.
[0026] Optionally, after the step of obtaining the target text output of the target large language model, the method further includes:
[0027] A post-processing correction model is constructed, the target text output is input into the post-processing correction model, the target text output is corrected, and a correct output result is obtained.
[0028] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a processing device based on multimodal input documents, the processing device based on multimodal input documents comprising:
[0029] A multimodal input module receives a multimodal input document, extracts chart information from the multimodal input document, and records location information of the chart information;
[0030] A visual language module inputs the chart information into a visual language model to obtain chart text language information;
[0031] The output module generates target text input information according to the graphic text language information and the position information, inputs the target text input information into a target large language model, and obtains a target text output of the target large language model.
[0032] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a processing device based on multimodal input documents, wherein the processing device based on multimodal input documents comprises a memory, a processor, and a processing program based on multimodal input documents stored in the memory and executable on the processor, wherein the processing program based on multimodal input documents implements the steps of the processing method based on multimodal input documents as described above when executed by the processor.
[0033] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which a processing program based on multimodal input documents is stored, and when the processing program based on multimodal input documents is executed by a processor, the steps of the processing method based on multimodal input documents as described above are implemented.
[0034] The present invention relates to a processing method based on multimodal input documents, which receives a multimodal input document, extracts chart information therefrom, and records the position information of the chart information. The chart information is input into a visual language model to obtain chart text language information. Based on the chart text language information and position information, these information and text information are embedded into a multimodal input document to generate target text input information. The target text input information is input into a target large language model to obtain a target text output of the target large language model. The present invention can effectively solve the deficiencies of the prior art in processing multimodal documents, achieve a deep understanding and accurate summary of multimodal documents, and generate customized text output, which significantly improves the processing efficiency and accuracy of complex documents. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a structural schematic diagram of a device hardware operating environment involved in an embodiment of a processing device based on multimodal input documents of the present invention;
[0036] Figure 2 It is a flowchart of a first embodiment of a processing method based on multimodal input documents of the present invention;
[0037] Figure 3 It is a functional module diagram of a preferred embodiment of the processing device based on multimodal input documents of the present invention.
[0038] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0039] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0040] It should be noted that most existing natural language processing (NLP) technologies focus on processing plain text data, while in actual applications, many documents (such as research reports, technical documents, business reports, etc.) usually contain information in multiple modes, including text, charts, pictures, etc. These multimodal input data need to be processed comprehensively in order to generate meaningful text descriptions and summaries. However, existing technologies have many deficiencies in processing such complex documents and cannot effectively combine and understand multimodal data, resulting in incomplete and inaccurate information processing.
[0041] In traditional NLP models, significant progress has been made in processing pure text data. For example, models such as BERT and GPT-3 have performed well in many language understanding tasks. However, these models are often unable to cope with documents containing non-text information such as charts and pictures. Charts and pictures contain a lot of visual information, which is crucial for understanding the content of the document. Models that rely solely on text information find it difficult to fully understand and utilize this visual information.
[0042] Some existing multimodal processing techniques attempt to combine visual and language models to process image and text information. However, these methods still have the following problems when processing complex documents:
[0043] Separation and fusion of chart and text information: When extracting chart information, traditional methods usually find it difficult to effectively combine the key information in the chart with the text content, resulting in a lack of coherence and accuracy in the generated text description.
[0044] Accuracy of visual feature extraction: Existing visual encoders are often limited by training data and model architecture when extracting high-dimensional visual features. They are unable to accurately capture key information in the chart, affecting the effect of multimodal data processing.
[0045] Based on the above-mentioned defects, the present invention provides a processing device based on multimodal input documents, referring to Figure 1 , Figure 1 It is a structural schematic diagram of the device hardware operating environment involved in the embodiment of the processing device based on multimodal input documents of the present invention.
[0046] like Figure 1 As shown, the processing device based on multimodal input documents may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0047] Those skilled in the art will understand that Figure 1The hardware structure of the processing device based on multimodal input documents shown in the figure does not constitute a limitation on the processing device based on multimodal input documents, and may include more or less components than shown in the figure, or a combination of certain components, or a different arrangement of components.
[0048] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a processing program based on multimodal input documents. The operating system is a program for managing and controlling processing equipment and software resources based on multimodal input documents, and supports the operation of the network communication module, the user interface module, the processing program based on multimodal input documents, and other programs or software; the network communication module is used to manage and control the network interface 1004; and the user interface module is used to manage and control the user interface 1003.
[0049] exist Figure 1 In the hardware structure of the processing device based on multimodal input documents shown, the network interface 1004 is mainly used to connect to the backend server and perform data communication with the backend server; the user interface 1003 is mainly used to connect to the client and perform data communication with the client; the processor 1001 can call the processing program based on the multimodal input document stored in the memory 1005, and perform the following operations:
[0050] receiving a multimodal input document, extracting chart information from the multimodal input document, and recording location information of the chart information;
[0051] Inputting the chart information into a visual language model to obtain chart text language information;
[0052] Target text input information is generated according to the graphic text language information and the position information, and the target text input information is input into a target large language model to obtain a target text output of the target large language model.
[0053] Furthermore, the step of extracting chart information from the multimodal input document includes:
[0054] Identify the format of the input document and extract the chart in the input document;
[0055] The text information in the chart information is extracted, and the contents of the chart other than the text information are used as the chart information.
[0056] Furthermore, the step of generating target text input information according to the graphic text language information and the position information includes:
[0057] The graphic text language information and the text information are embedded into the multimodal input document according to the position information to generate the target text input information.
[0058] Furthermore, the step of inputting the target text input information into the target large language model includes:
[0059] Acquire user demand information, and generate demand prompts according to the user demand information, wherein the demand prompts are specific descriptions of the user demand information and are used to guide the target large language model to generate outputs that meet the user demand;
[0060] The requirement prompt and the target text input information are input into the target large language model.
[0061] Furthermore, the step of inputting the chart information into a visual language model to obtain the chart text language information includes:
[0062] Inputting the chart information into the visual language model, wherein the visual language model extracts high-dimensional visual features from the chart information;
[0063] defining and generating a potential query vector, and fusing the high-dimensional visual feature with the potential query vector using an Attention mechanism to obtain a fused feature representation;
[0064] The fused feature representation is input into a decoder to obtain the graphic text language information.
[0065] Furthermore, the step of extracting high-dimensional visual features from the chart information by the visual language model includes:
[0066] The visual language model includes a visual encoder, which is trained by the NFNet model through a contrastive learning task, and the visual encoder extracts high-dimensional visual features from the chart information.
[0067] Furthermore, after the step of obtaining the target text output of the target large language model, the method further includes:
[0068] Construct a post-processing correction model, input the target text output into the post-processing correction model, correct the target text output, and obtain the correct output result
[0069] The specific implementation of the device for processing multimodal input documents of the present invention is basically the same as the following embodiments of the method for processing multimodal input documents, and will not be described in detail here.
[0070] The present invention also provides a method for processing documents based on multimodal input based on the above-mentioned device for processing documents based on multimodal input.
[0071] Reference Figure 2 , Figure 2 It is a flowchart diagram of the first embodiment of the processing method based on multimodal input documents of the present invention.
[0072] The embodiment of the present invention provides an embodiment of a processing method based on multimodal input documents. It should be noted that although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.
[0073] In various embodiments of the processing method based on multimodal input documents, the execution subject is a processing device based on multimodal input documents.
[0074] The processing method based on multimodal input documents is applied to a processing device based on multimodal input documents, and the processing method based on multimodal input documents includes:
[0075] Step S10, receiving a multimodal input document, extracting chart information from the multimodal input document, and recording location information of the chart information;
[0076] Most existing natural language processing (NLP) technologies focus on processing plain text data. In practical applications, many documents (such as research reports, technical documents, business reports, etc.) usually contain information in multiple modes, including text, charts, pictures, etc. These multimodal input data need to be processed comprehensively in order to generate meaningful text descriptions and summaries. However, existing technologies have many deficiencies in processing such complex documents and cannot effectively combine and understand multimodal data, resulting in incomplete and inaccurate information processing.
[0077] In traditional NLP models, significant progress has been made in processing pure text data. For example, models such as BERT and GPT-3 have performed well in many language understanding tasks. However, these models are often unable to cope with documents containing non-text information such as charts and pictures. Charts and pictures contain a lot of visual information, which is crucial for understanding the content of the document. Models that rely solely on text information find it difficult to fully understand and utilize this visual information.
[0078] Some existing multimodal processing techniques attempt to combine visual and language models to process image and text information. However, these methods still have the following problems when processing complex documents:
[0079] Separation and fusion of chart and text information: When extracting chart information, traditional methods usually find it difficult to effectively combine the key information in the chart with the text content, resulting in a lack of coherence and accuracy in the generated text description.
[0080] Accuracy of visual feature extraction: Existing visual encoders are often limited by training data and model architecture when extracting high-dimensional visual features. They are unable to accurately capture key information in the chart, affecting the effect of multimodal data processing.
[0081] The processing method based on multimodal input documents in this embodiment is applicable to a device having the function of the processing method based on multimodal input documents.
[0082] In this embodiment, various documents containing multimodal inputs may be received, such as enterprise market analysis reports, legal document reviews, educational materials, and the like.
[0083] For example, receiving a scientific research report in PDF format containing text and figures.
[0084] Use document parsing tools (such as Adobe Acrobat API, PyMuPDF, etc.) to identify the structure of the document, extract all chart information from it, and record the location information of these charts in the document. The location information includes the page number and coordinate position of the chart in the document.
[0085] For example, use the PyMuPDF tool to load a PDF document. By parsing the PDF file, the system can identify each page of the document and extract the chart information on each page. The system traverses each page of the PDF document and identifies the chart information on the page, including the chart's image, title, label, and other content. The system records the information of each chart, including its page number and coordinate position in the document (such as the coordinates of the upper left and lower right corners of the chart on the page). The extracted chart information, including the chart's image data and location information, will be saved to a database or file for subsequent processing and analysis.
[0086] Suppose a research report contains the following charts: Page 1 contains a bar chart of experimental results, located in the upper left corner of the document (coordinate position: [72,150,300,450]). Page 3 contains a flowchart of the experimental setup, located in the middle of the document (coordinate position: [100,200,400,600]).
[0087] After the above steps, the extracted chart information will be recorded as follows:
[0088] Bar chart: located in the upper left corner of page 1, with coordinates of [72,150,300,450].
[0089] Flowchart: Located in the middle of page 3, with coordinates [100, 200, 400, 600].
[0090] Step S20, inputting the chart information into a visual language model to obtain chart text language information;
[0091] In this embodiment, the chart information obtained through the above steps will be passed to the visual language model to further extract high-dimensional visual features and generate corresponding text descriptions, providing accurate basic data for subsequent analysis and summary.
[0092] The steps may include: converting the extracted chart information into a format suitable for model processing, including the chart image data and related text labels, titles and other information. Standardizing the image, such as adjusting the image size, normalizing the image data, etc., to meet the input requirements of the visual model.
[0093] The preprocessed chart image is fed into the pre-trained Normalizer-Free ResNet (NFNet) model. The NFNet model is used to extract high-dimensional visual features of the chart, which contain visual information in the chart, such as color, shape, data points, etc.
[0094] In the process of visual feature extraction, the model extracts multi-scale features of the image through a multi-layer convolutional neural network, and finally obtains a high-dimensional feature vector representation.
[0095] If the chart contains text information (such as labels, titles, and data points), use OCR (Optical Character Recognition) technology to extract this text information.
[0096] The extracted text information is processed, including word segmentation, part-of-speech tagging, etc., and converted into a format suitable for language model processing.
[0097] Define and generate latent query vectors, which capture key feature information in the graph through learning. The latent query vector is a parameter learned during model training and is used to fuse with visual features in the Attention mechanism.
[0098] Generate multiple potential query vectors to ensure that all graph information is processed effectively.
[0099] The latent query vector is fused with the extracted high-dimensional visual features using a multi-head self-attention mechanism. The attention mechanism weights and combines features by calculating the similarity between the query vector and the key vector.
[0100] Each query vector is calculated as a weighted sum with all visual features in the Attention mechanism to generate a fused feature representation.
[0101] The fused feature representation is input into the decoder, which consists of a multi-layer Transformer decoder. The Transformer decoder converts the input fused features into a natural language text description through a multi-layer self-attention mechanism and a feedforward neural network layer to obtain the text language information of the chart. The text description generated by the decoder includes an explanation of the chart content, key points of the data, etc.
[0102] Step S30, generating target text input information according to the graphic text language information and the position information, inputting the target text input information into a target large language model, and obtaining a target text output of the target large language model.
[0103] In this embodiment, the generated chart text language information is combined with the text information extracted from the document. According to the recorded chart position information, this information is embedded into the original document to generate the target text input information. It is ensured that the generated text information is consistent with the original document content, and the overall structure and logic of the document are kept clear.
[0104] Format the generated target text input information to ensure that the text is clear in layout, grammar, and logical structure. Ensure that the embedded chart description is consistent with the style and tone of the surrounding text to improve the readability and professionalism of the text.
[0105] Input the formatted target text input information into the target large language model (such as the pre-trained language model GPT-3) to generate a preliminary target output. The target output can be a detailed research summary, data analysis report, or other target text output content that meets user needs. The final result can be generated in multiple formats such as PDF and Word for easy reading and publishing by users.
[0106] The present invention relates to a processing method based on multimodal input documents, which receives a multimodal input document, extracts chart information therefrom, and records the position information of the chart information. The chart information is input into a visual language model to obtain chart text language information. Based on the chart text language information and position information, these information and text information are embedded into a multimodal input document to generate target text input information. The target text input information is input into a target large language model to obtain a target text output of the target large language model. The present invention can effectively solve the deficiencies of the prior art in processing multimodal documents, achieve a deep understanding and accurate summary of multimodal documents, and generate customized text output, which significantly improves the processing efficiency and accuracy of complex documents.
[0107] Furthermore, a second embodiment of the processing method based on multimodal input documents of the present invention is proposed. In the above step S10, the step of extracting chart information from the multimodal input document includes:
[0108] Step a1, identifying the format of the input document and extracting charts in the input document;
[0109] Step a2, extracting the text information in the chart information, and taking the contents of the chart other than the text information as the chart information.
[0110] In this embodiment, detailed steps of extracting chart information from a multimodal input document are described, including identifying the format of the input document, extracting the chart, and processing the text information and visual information in the chart.
[0111] Receive a multimodal input document, which may be in PDF, Word, Excel or other formats. Use document parsing tools (such as Adobe Acrobat API, PyMuPDF, Apache POI, etc.) to load and parse the document content.
[0112] Identify the format of the document to determine the appropriate parsing tools and methods to use.
[0113] For example, use PyMuPDF for PDF documents, Apache POI for Word documents, and Apache POI or Pandas for Excel documents.
[0114] The system traverses each page of the document, identifies and extracts chart information, and uses parsing tools to identify images, charts, and embedded objects in the page.
[0115] Text information extraction: Use OCR (Optical Character Recognition) technology to extract text information from charts. OCR tools such as Tesseract can recognize text in images and convert them into editable text format.
[0116] Save the text information in the chart separately, and save the contents of the chart other than the text information as chart information. Record the detailed information of the chart, including chart image, text information, chart location, etc.
[0117] For example, a bar graph (page 1): extract the graph image and record the position information: [72, 150, 300, 450]. Use OCR to extract text information: "temperature (℃)", "reaction rate (mol / s)", "25℃", "0.5mol / s", "75℃", "1.8mol / s". Save the graph image and position information, and save the text information separately.
[0118] Flowchart (page 3): Extract the chart image and record the position information: [100, 200, 400, 600]. Use OCR to extract text information: "Sample preparation", "Reaction condition control", "Data acquisition", "Data analysis". Save the chart image and position information, and save the text information separately.
[0119] Through the above steps, this embodiment can accurately extract chart information and text information from multimodal input documents and record detailed location information. This information will be used for subsequent visual language model processing and text generation to ensure that the generated text description and summary are accurate and coherent.
[0120] Furthermore, a third embodiment of the processing method based on multimodal input documents of the present invention is proposed. In the above step a1, the step of generating target text input information according to the graphic text language information and the position information includes:
[0121] Step b1, embedding the graphic text language information and the text information into the multimodal input document according to the position information to generate the target text input information.
[0122] In this embodiment, detailed steps of how to generate target text input information according to the graphic text language information and position information are described, including embedding the graphic text language information and text information into the multimodal input document according to the position information to generate the target text input information. The specific steps may include:
[0123] Combine the chart text language information with the text information: Combine the generated chart text language information with the text information extracted from the document. According to the recorded chart position information, embed this information into the original document.
[0124] Determine the insertion position: Find the specific location of the chart in the original document. Locate the chart in the document based on the location information (page number, coordinates).
[0125] Insert chart description: Insert the generated chart text language information at the located chart position. Make sure the insertion position is consistent with the chart position in the original document to maintain the coherence and logic of the content.
[0126] Processing chart descriptions and annotations: For each chart, embed the generated text description (such as the chart title, label, data explanation, etc.) into the corresponding position in the document. For example, insert the generated description below or next to the chart to ensure that readers can understand the content of the chart.
[0127] Format the target text input information: Format the embedded target text input information to ensure that the text layout, grammar and logical structure are clear. Ensure that the embedded chart description is consistent with the style and tone of the surrounding text to improve the readability and professionalism of the text.
[0128] For example, suppose we have a scientific research report in PDF format that contains the following chart:
[0129] Bar chart (page 1): Extract chart information: The system recognizes and extracts the bar chart image and its text information (label, title), and records the location information of the chart (upper left corner of page 1: [72,150,300,450]).
[0130] Generate chart text language information: The chart image is input into the NFNet model to extract high-dimensional visual features.
[0131] Use OCR technology to extract text information: "temperature (℃)", "reaction rate (mol / s)", "25℃", "0.5mol / s", "75℃", "1.8mol / s". Generate a potential query vector and fuse it with visual features through the Attention mechanism.
[0132] The decoder generates a text description: " Figure 3 The reaction rates under different temperature conditions are shown. It can be seen that the reaction rate increases significantly with increasing temperature. At 25°C, the reaction rate is 0.5 mol / s, while at 75°C, the reaction rate reaches 1.8 mol / s.
[0133] Embed information into the document: Find the coordinates of the chart in the upper left corner of page 1: [72,150,300,450]. Insert the generated text description below or next to the chart location: " Figure 3 The reaction rates at different temperatures are shown. It can be seen that the reaction rate increases significantly with increasing temperature. At 25°C, the reaction rate is 0.5 mol / s, while at 75°C, the reaction rate reaches 1.8 mol / s. ” Make sure the description is closely related to the graph and easy to understand.
[0134] Through the above method, this embodiment can accurately embed the generated chart text language information into the multimodal input document, ensuring that the text description is consistent with the chart position and content. The generated target text input information content is coherent and logically clear, which improves the overall readability and professionalism of the document and provides accurate basic data for subsequent analysis and summary.
[0135] Furthermore, a fourth embodiment of the processing method based on multimodal input documents of the present invention is proposed. In the above step S30, the step of inputting the target text input information into the target large language model includes:
[0136] Step c1, obtaining user demand information, and generating demand prompts according to the user demand information, wherein the demand prompts are specific descriptions of the user demand information, and are used to guide the target large language model to generate outputs that meet the user demand;
[0137] Step c2: inputting the requirement prompt and the target text input information into the target large language model.
[0138] In this embodiment, when processing a multimodal input document, it is necessary to generate a demand prompt according to the user's demand, and input the demand prompt together with the target text input information into the target large language model to generate a target text output that meets the user's demand.
[0139] Obtain user demand information through the user interface or preset parameters. User demand information may include specific research questions, data analysis requirements, report formats, etc.
[0140] First, generate demand prompts: Generate demand prompts based on user demand information. Demand prompts are specific descriptions of user needs, guiding the model to generate outputs that meet user needs.
[0141] For example, a user requirement prompt might be: “Generate a detailed experimental summary, including the effect of temperature on reaction rate, and a visual graphical depiction of the data.
[0142] Secondly, combine the demand prompt with the target text input information: combine the generated demand prompt with the target text input information. For example, combine the user demand prompt with the generated chart text language information and other text information to form a complete input data.
[0143] Finally, input the data into the target large language model: Input the combined input data into the target large language model (such as the pre-trained language model GPT-3). The target large language model uses the input data to generate detailed output text, including research summaries, data analysis reports, etc.
[0144] In this embodiment, through the above steps, the user demand prompt clarifies the information that the user wants to obtain from the document and the focus of the report. By combining the demand prompt, the system can generate customized output to meet the specific needs of the user.
[0145] Furthermore, a fifth embodiment of the processing method based on multimodal input documents of the present invention is proposed, wherein the above step S20 comprises:
[0146] Step d1, inputting the chart information into the visual language model, and the visual language model extracting high-dimensional visual features from the chart information;
[0147] In this embodiment, a pre-trained Normalizer-Free ResNet (NFNet) model is used, and the chart image is input into the model to extract high-dimensional visual features.
[0148] During the visual feature extraction process, the NFNet model extracts multi-scale features of the image through a multi-layer convolutional neural network and generates a high-dimensional feature vector representation. The extracted high-dimensional visual features are standardized and preprocessed to ensure that the feature data is suitable for subsequent processing.
[0149] Step d2, defining and generating a potential query vector, and using an Attention mechanism to fuse the high-dimensional visual feature with the potential query vector to obtain a fused feature representation;
[0150] Step d3, inputting the fused feature representation into a decoder to obtain the graphic text language information.
[0151] Define potential query vectors based on the content of the graph and user needs. These vectors are used to capture the key feature information in the graph. Potential query vectors can be predefined vectors or parameters learned during model training.
[0152] The latent query vector (Q) is fused with the extracted high-dimensional visual features (K, V) using a multi-head self-attention mechanism to generate a fused feature representation.
[0153] The Attention mechanism weights and combines features by calculating the similarity between the query vector and the key vector, ensuring that all important information is processed effectively. The Attention calculation formula is:
[0154]
[0155] Where: Q represents the query vector (Query), K represents the key vector (Key), V represents the value vector (Value), d k represents the dimensionality of the keys. The softmax function ensures that the weights are normalized so that the sum of the weights is 1.
[0156] The fused feature representation is input into the decoder, which consists of a multi-layer Transformer decoder. The Transformer decoder converts the input fused features into a natural language text description through a multi-layer self-attention mechanism and a feedforward neural network layer. The text description generated by the decoder includes an explanation of the chart content, key points of the data, etc.
[0157] For example, there is a scientific research report that contains the following chart:
[0158] Bar graph (page 1): Extract chart information: Identify and extract the bar graph image and its text information (label, title), and record the position information of the chart (upper left corner of page 1: [72,150,300,450]).
[0159] Input visual language model: Input the histogram image into the NFNet model to extract high-dimensional visual features.
[0160] Extracting high-dimensional visual features: The NFNet model extracts multi-scale features of images through a multi-layer convolutional neural network and generates a high-dimensional feature vector representation.
[0161] Define and generate latent query vectors: Given multiple graphs as input, the model automatically generates learnable latent query vectors that are used to capture key feature information in the graph.
[0162] Feature fusion: Use a multi-head self-attention mechanism to fuse the latent query vector with the high-dimensional visual features to generate a fused feature representation. The Attention mechanism weights and combines features by calculating the similarity between the query vector and the key vector.
[0163] Generate graph text language information: Input the fused feature representation into the decoder to generate the graph text language information.
[0164] The decoder generates a text description: " Figure 3 The reaction rates under different temperature conditions are shown. It can be seen that the reaction rate increases significantly with increasing temperature. At 25°C, the reaction rate is 0.5 mol / s, while at 75°C, the reaction rate reaches 1.8 mol / s.
[0165] Through the above steps, this embodiment can use the pre-trained NFNet model to extract high-dimensional visual features, and fuse the potential query vector with the visual features through the multi-head self-attention mechanism to generate accurate chart text language information. These text descriptions can explain the chart content in detail and provide users with a clear understanding and analysis basis.
[0166] Furthermore, a sixth embodiment of the processing method based on multimodal input documents of the present invention is proposed. In the above step d1, the step of extracting high-dimensional visual features from the chart information by the visual language model includes:
[0167] Step e1, the visual language model includes a visual encoder, which is trained by the NFNet model through a contrastive learning task, and the visual encoder extracts high-dimensional visual features from the chart information.
[0168] In this embodiment, the visual language model consists of a visual encoder and a text decoder. The visual encoder is responsible for extracting high-dimensional visual features from the chart information, while the text decoder converts these features into natural language text descriptions.
[0169] The visual encoder uses a pre-trained Normalizer-Free ResNet (NFNet) model and is trained through contrastive learning tasks. During the training process, the InfoNCE loss function (InfoNCE loss) is used to optimize the model and improve the accuracy of feature extraction. The formula of InfoNCE Loss is:
[0170]
[0171] Where: x q The feature vector representing the query sample; y + The feature vector representing the positive sample; y i represents the feature vector of the i-th sample among all samples, including positive samples and negative samples; τ is a hyperparameter set by an artificial person; S(x q ,y i ) represents the similarity between the query sample and the i-th sample.
[0172] The high-dimensional features extracted by the visual encoder are transformed into 1×n dimensional vectors in two-dimensional space through the Flatten operation, which is convenient for subsequent processing and feature fusion. When multiple graphs are input, the model automatically generates learnable latent query vectors (Learned latent queries), which are used to capture the key feature information in the graph.
[0173] For example, there is a scientific research report that contains the following chart:
[0174] Flowchart (page 3): Extract chart information. The system recognizes and extracts the flowchart image and its text information (step instructions, comments), and records the location information of the chart (middle of page 3: [100, 200, 400, 600]).
[0175] Input the visual language model and feed the flowchart image into the NFNet model to extract high-dimensional visual features.
[0176] Train the visual encoder, use the pre-trained NFNet model, and optimize the InfoNCE loss function in the contrast learning task to improve the accuracy of feature extraction. The extracted high-dimensional features are transformed into 1×n-dimensional vectors in two-dimensional space through the Flatten operation.
[0177] Through the above steps, this embodiment can use the pre-trained NFNet model to extract high-dimensional visual features, and fuse the potential query vector with the visual features through the multi-head self-attention mechanism to generate accurate chart text language information. These text descriptions can explain the chart content in detail and provide users with a clear understanding and analysis basis.
[0178] Furthermore, a seventh embodiment of the processing method based on multimodal input documents of the present invention is proposed. In the above step S30, after the step of obtaining the target text output of the target large language model, the method further includes:
[0179] Step f1, constructing a post-processing correction model, inputting the target text output into the post-processing correction model, correcting the target text output, and obtaining a correct output result.
[0180] In this embodiment, after obtaining the preliminary target text output, a post-processing correction model is constructed, and the target text output is input into the model for correction to ensure the accuracy and consistency of the output result.
[0181] First, build a post-processing correction model and collect a large amount of training data containing preliminary output and correct output pairs. This data can come from previously processed documents and their correction results. The dataset should include various types of texts, covering different topics and fields to ensure the generalization ability of the model.
[0182] Choose a suitable correction model. You can use a Transformer-based pre-trained model (such as BERT, GPT, etc.) for fine-tuning, or you can use a traditional language model. Initialize the model parameters and prepare for training.
[0183] The model is trained using the collected training data, with the goal of enabling the model to generate correct correction outputs from preliminary outputs. During the training process, the model learns the difference between preliminary outputs and correct outputs and adjusts parameters to minimize correction errors. The model is evaluated using the validation set to check the model's correction capabilities and output quality. Model parameters and training strategies are adjusted based on the evaluation results to optimize model performance
[0184] Since the preliminary target text output is generated by the visual language model, it may contain some grammatical errors, logical inconsistencies, or missing information. The preliminary target text output is input into the post-processing correction model. The correction model corrects the output according to the rules and patterns learned in training. The correction model generates a corrected output result, correcting grammatical errors, logical inconsistencies, or missing information to ensure the accuracy and consistency of the output text.
[0185] Functions of the calibration model may include:
[0186] Grammatical and spelling error correction: The correction model can identify and correct grammatical and spelling errors in the preliminary output, improving the language quality of the text.
[0187] Logical consistency check: The correction model can check and correct logical inconsistencies in the text to ensure the coherence and rationality of the output content.
[0188] Information completion and refinement: The correction model can complete missing information based on the context and refine lengthy or repetitive content to make the output text more concise and clear.
[0189] Through the above steps, this embodiment can further optimize the text quality based on the preliminary target output and generate accurate, consistent and high-quality final output text. This process ensures the professionalism and readability of the output results and provides users with reliable document content.
[0190] The present invention also provides a processing device based on multi-modal input documents.
[0191] Reference Figure 3 , Figure 3 Schematic diagram of functional modules of a first embodiment of a processing device based on multimodal input documents of the present invention. The processing device based on multimodal input documents comprises:
[0192] A multimodal input module receives a multimodal input document, extracts chart information from the multimodal input document, and records location information of the chart information;
[0193] A visual language module inputs the chart information into a visual language model to obtain chart text language information;
[0194] The output module generates target text input information according to the graphic text language information and the position information, inputs the target text input information into a target large language model, and obtains a target text output of the target large language model.
[0195] In addition, an embodiment of the present invention further provides a computer storage medium.
[0196] The computer storage medium stores a processing program based on multimodal input documents, and when the processing program based on multimodal input documents is executed by a processor, the steps of the processing method based on multimodal input documents as described above are implemented.
[0197] The specific implementation of the storage medium of the present invention is basically the same as the above-mentioned embodiments of the processing method based on multimodal input documents, and will not be repeated here.
[0198] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.
[0199] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0200] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.
[0201] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0202] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A processing method based on multimodal input documents, characterized in that: The processing method based on multimodal input documents comprises the following steps: receiving a multimodal input document, extracting chart information from the multimodal input document, and recording location information of the chart information; Inputting the chart information into a visual language model to obtain chart text language information; Target text input information is generated according to the graphic text language information and the position information, and the target text input information is input into a target large language model to obtain a target text output of the target large language model.
2. The processing method based on multimodal input documents according to claim 1, characterized in that: The step of extracting chart information from the multimodal input document includes: Identifying the format of the input document and extracting charts from the input document; The text information in the chart information is extracted, and the content of the chart other than the text information is used as the chart information.
3. The method for processing multimodal input documents as claimed in claim 2, wherein: The step of generating target text input information according to the graphic text language information and the position information includes: The graphic text language information and the text information are embedded into the multimodal input document according to the position information to generate the target text input information.
4. The processing method based on multimodal input documents according to claim 1, characterized in that: The step of inputting the target text input information into the target large language model comprises: Obtaining user demand information, and generating a demand prompt based on the user demand information, wherein the demand prompt is a specific description of the user demand information, and is used to guide the target large language model to generate an output that meets the user demand; The requirement prompt and the target text input information are input into the target large language model.
5. The processing method based on multimodal input documents according to claim 1, characterized in that: The step of inputting the chart information into a visual language model to obtain the chart text language information comprises: inputting the chart information into the visual language model, wherein the visual language model extracts high-dimensional visual features from the chart information; defining and generating a latent query vector, and fusing the high-dimensional visual features with the latent query vector using an attention mechanism to obtain a fused feature representation; The fused feature representation is input into a decoder to obtain the graph text language information.
6. The processing method based on multimodal input documents according to claim 5, characterized in that: The step of extracting high-dimensional visual features from the chart information using the visual language model includes: The visual language model includes a visual encoder, which is trained by the NFNet model through a contrastive learning task, and the visual encoder extracts high-dimensional visual features from the chart information.
7. The processing method based on multimodal input documents according to claim 1, characterized in that: After the step of obtaining the target text output of the target large language model, the method further includes: A post-processing correction model is constructed, the target text output is input into the post-processing correction model, the target text output is corrected, and a correct output result is obtained.
8. A processing device based on multimodal input documents, characterized in that: The processing device based on multimodal input documents includes: a multimodal input module, receiving a multimodal input document, extracting chart information from the multimodal input document, and recording location information of the chart information; A visual language module inputs the chart information into a visual language model to obtain chart text language information; The output module generates target text input information according to the graphic text language information and the position information, inputs the target text input information into a target large language model, and obtains a target text output of the target large language model.
9. A processing device based on multimodal input documents, characterized in that: The processing device based on multimodal input documents includes a memory, a processor, and a processing program based on multimodal input documents stored on the memory and capable of running on the processor. When the processing program based on multimodal input documents is executed by the processor, the steps of the processing method based on multimodal input documents as described in any one of claims 1 to 7 are implemented.
10. A computer storage medium, characterized in that The storage medium stores a processing program based on multimodal input documents, and when the processing program based on multimodal input documents is executed by the processor, the steps of the processing method based on multimodal input documents according to any one of claims 1 to 7 are implemented.
Citation Information
Cited By
Method and system for generating text summary according to chart
CN119296117A
PPT file content extraction method and system based on large language model
CN119397040A
Document generation method, device and equipment
CN120197597A
Document generation method, device and equipment
CN120197597B
