Interactive intelligent document question and answer method and system and storage medium

Abstract

The invention provides an interactive intelligent document question-answering method and system and a storage medium, and the intelligent document question-answering method comprises the steps: S1, obtaining document data, and converting the document data into image data; s2, acquiring a text input by a user, and converting the input text into a visual instruction feature; and S3, performing visual observation processing on the image data according to the visual instruction features to obtain a target text answer. Due to the adoption of a pure vision scheme, the limitation of an OCR (Optical Character Recognition) technology is completely avoided, and unstructured contents which cannot be processed by a traditional Text-RAG method can be effectively processed. In addition, the reading behavior is self-adapted by simulating multiple steps of retrieval-search-focusing of human beings. And when needed, related pages are'watched 'or details are magnified, so that huge waste caused by the fact that all the pages are'watched at one time' in a large VLM is avoided, and the efficiency is higher.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, specifically relating to an interactive intelligent document question-answering method, an interactive intelligent document question-answering device, and a storage medium. Background Technology

[0002] With the development of Large Language Models (LLM) and Visual Language Models (VLM), automated document processing has become a hot topic. However, traditional technologies face bottlenecks when dealing with long documents, especially unstructured documents (such as scanned documents, handwritten notes, and reports containing complex charts).

[0003] Optical character recognition (OCR)-based methods suffer from high text extraction error rates when faced with unstructured content, leading to subsequent question-answering failures. Therefore, purely visual question answering (VLDQA), which directly processes images, has become an important technological direction.

[0004] Currently, there are two main document question-and-answer solutions:

[0005] 1. Text-based Retrieval Enhanced Generative Aggregator (RAG): This is currently the mainstream approach. It first converts document images into text using OCR, then stores the text chunks in a database. Subsequent steps (retrieval, generation) rely entirely on this extracted text.

[0006] 2. End-to-End Large Visual Language Model (VLM): such as GPT-4V. This approach attempts to take all page images of a document (up to hundreds of pages) as input, allowing the model to "see" all the content at once and generate the answer.

[0007] However, both of the above-mentioned existing technologies have serious drawbacks:

[0008] 1. Defects of Text-based RAG: (1) Reliance on OCR, failure to handle unstructured content: For scanned documents, handwritten text, formulas or charts, OCR technology will produce a large number of errors or fail completely, resulting in information loss, which is its fundamental defect. (2) Context fragmentation: Text slicing will destroy the visual layout and contextual continuity of the original text (e.g., a table is cut into two slices).

[0009] 2. Disadvantages of Large VLM: (1) Extremely high computational overhead and resource requirements: Processing hundreds of pages of high-resolution images requires huge computational resources and video memory, and the inference time is extremely long. (2) Cannot run on edge devices: Due to its huge model size and resource requirements, such models cannot be deployed on edge devices such as smartphones and tablets (these devices usually have less than 1GB of memory).

[0010] 3. Common Drawback (Lack of Flexibility): Both approaches lack the flexibility to simulate expert human reading. Humans read by "quickly flipping pages" (retrieval), "locating keywords" (searching), and "zooming in to see details" (focusing). Existing technologies, however, either follow a fixed text flow or forcefully "read through everything at once," resulting in inefficiency and a lack of intelligence. Summary of the Invention

[0011] In view of this, the purpose of this application is to provide an interactive intelligent document question-answering method, system, and storage medium to solve the above problems.

[0012] To solve the above-mentioned technical problems, this application adopts the following technical solution:

[0013] In a first aspect, this application provides an interactive intelligent document question answering method, which includes: step S1: acquiring document data and converting the document data into image data; step S2: acquiring user input text and converting the input text into visual instruction features; step S3: performing visual observation processing on the image data based on the visual instruction features to obtain the target text answer.

[0014] Furthermore, step S1 includes: step S11: obtaining document data, which includes at least one of the following: text, tables, and charts; step S12: processing the document data based on an image noise algorithm to obtain image data.

[0015] Furthermore, step S2 includes: step S21: obtaining user input text; step S22: processing the user input text based on the visual language model to obtain visual instruction features.

[0016] Furthermore, step S3 also includes: step S31: performing visual observation processing on the image data according to the visual instruction features to obtain the visual observation results, and saving the visual observation results to the cache area; step S32: using the image in the cache area as the target text answer and outputting it for display.

[0017] Furthermore, the visual instruction features include retrieval actions. Step S31 includes: Step A1: Embedding the visual instruction features into an instruction vector and embedding the image data into an image vector; Step A2: Calculating the cosine similarity between the instruction vector and the image vector to obtain the image data with the highest cosine similarity; Step A3: Using the complete page image in the image data as the visual observation result.

[0018] Furthermore, the visual instruction features include search actions, and step S31 includes: step B1: processing the visual instruction features based on a lightweight object detection model to obtain shape visual features; step B2: performing visual observation processing on the image data based on the shape visual features to obtain cropped image information of the image data; step B3: using the cropped image information as the visual observation result.

[0019] Furthermore, the visual instruction features include focusing actions, and step S31 also includes: step C1: determining the image region coordinates and reference image template based on the image data; step C2: determining the target focused image based on the image region coordinates; step C3: magnifying the target focused image based on the reference image template, and using the magnified target focused image as the visual observation result.

[0020] Secondly, this application provides an interactive intelligent document question-answering device, which includes an acquisition module, a conversion module, and an observation module. The acquisition module is used to acquire document data and convert the document data into image data; the conversion module is used to acquire user input text and convert the input text into visual instruction features; and the observation module is used to perform visual observation processing on the image data according to the visual instruction features to obtain the target text answer.

[0021] Thirdly, this application provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the above-described interactive intelligent document question-and-answer method.

[0022] Fourthly, this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described interactive intelligent document question-and-answer method.

[0023] As can be seen from the above technical solutions, the advantages and positive effects of the interactive intelligent document question-answering method, system, and storage medium proposed in this application are as follows:

[0024] 1. High robustness: This application, by adopting a purely visual approach, completely avoids the limitations of OCR technology. Therefore, it can effectively handle unstructured content that traditional Text-RAG methods cannot handle, such as scanned old documents, handwritten notes, and reports containing complex charts and formulas.

[0025] 2. High efficiency and low cost (edge ​​computing): This application uses a small neural network (approximately 5M parameters) instead of a large model with hundreds of billions of parameters. Its computational overhead and memory usage are extremely low, enabling it to run efficiently on edge devices (such as smartphones with <1GB of memory), which is something that large VLM solutions cannot achieve.

[0026] 3. Intelligence and Flexibility: The intelligent agent simulates the multi-step, adaptive reading behavior of humans, involving "retrieval-searching-focusing." It only "looks at" relevant pages or zooms in on details when needed, avoiding the huge waste of large VLMs that "view all pages at once," thus achieving higher efficiency. Attached Figure Description

[0027] The above description of this application and the following detailed embodiments will be better understood when read in conjunction with the accompanying drawings. It should be noted that the drawings are merely examples of the claimed technical solutions.

[0028] Figure 1 This is a flowchart of an interactive, intelligent document question-and-answer method;

[0029] Figure 2 This is an interactive diagram of an intelligent document question-and-answer system;

[0030] Figure 3 This is the execution logic diagram of the interactive intelligent document question-and-answer method. Detailed Implementation

[0031] The detailed features and advantages of this application are described below in the specific embodiments. The content of this description is sufficient to enable any person skilled in the art to understand the technical content of this application and implement it accordingly. Based on the specification, claims and drawings disclosed in this specification, a person skilled in the art can easily understand the related objectives and advantages of this application.

[0032] The invention will now be described with reference to the accompanying drawings, in which similar reference numerals denote similar elements. While specific structures and arrangements are discussed, it should be understood that this is done merely for illustrative purposes. Those skilled in the art will recognize that other structures and arrangements can be used without departing from the spirit and scope of the invention. It will be apparent to those skilled in the art that the invention can also be used in a variety of other applications.

[0033] In this specification and claims, several terms will be used, and unless otherwise indicated, these terms will be defined to have the following meanings:

[0034] The singular forms “a” and “the” include their corresponding plural forms. “At least one” means one or more, and “more” means two or more. “At least one of the following” or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be expressed as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0035] All figures used to represent component amounts, properties (e.g., molecular weight), reaction conditions, etc., should be considered to be modified in all cases by the terms "within the unavoidable margin of error" or "about". Therefore, the numerical values ​​set forth herein are approximate and may vary depending on the desired properties sought to be obtained by the present invention. The principle of equivalents, which is applied to a minimum and not intended to limit the scope of the claims, should be applied, for example, each value should be interpreted at least according to the specified significant digits and by applying conventional rounding techniques.

[0036] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0037] In the description of this embodiment, it should be noted that the terms "upper", "lower", "inner", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product is usually placed during use. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0038] Unless otherwise indicated, the following abbreviations have the following meanings, and any other abbreviations used herein but not defined have their generally accepted standard meanings:

[0039] All other terms used herein that are specifically defined herein shall have the general meaning understood by one of ordinary skill in the art, in particular meaning that, upon reading the claims, specification and drawings of this patent, one can directly and without doubt determine how the technical solution of this patent can be implemented.

[0040] Even if there are incomplete descriptions, omissions, or ambiguities in the grammar, words, punctuation, graphics, symbols, etc. of the claims, specification, and drawings of this patent, a person skilled in the art can still arrive at the only correct understanding by reading the claims, specification, and drawings as a whole without extensive reasoning or experimentation, and effectively exclude various incorrect interpretations that are not aimed at achieving the purpose of this patent.

[0041] Those skilled in the art would first choose to read the claims, specification, and drawings of this patent to reasonably interpret the terms; secondly, they would choose to refer to the relevant definitions in other documents published by the applicant before the filing date to reasonably interpret the terms; thirdly, they would choose the references cited in this patent to reasonably interpret the terms; and finally, they would choose to combine the technical dictionaries, technical manuals, reference books, textbooks, national or industry technical standards, etc., commonly used by those skilled in the art to reasonably interpret the terms.

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0043] Please refer to Figure 1 and Figure 2 This application provides an interactive intelligent document question-answering method, which is applied to an interactive intelligent document question-answering system.

[0044] An interactive intelligent document question-answering system mainly consists of three core components:

[0045] 1. Document Image Repository:

[0046] (1) Function: Preprocessing and storing document images

[0047] (2) Implementation: The input PDF document is first converted into a sequence of pure images (e.g., using the pdf2image library, each page is converted into a PNG image with a resolution of 1024x768). Then, the system uses a pre-trained CLIP ViT-B / 32 model to pre-compute a 512-dimensional visual embedding vector for each page image. The visual embeddings of all pages are stored in a Faiss index for fast visual similarity retrieval.

[0048] 2. Agent Model:

[0049] (1) Function: The “brain” of the system, responsible for decision-making.

[0050] (2) Implementation: This is a small multimodal neural network (e.g., based on the MobileViT-S variant, with approximately 5M total parameters). It includes:

[0051] ① Text encoder (such as DistilBERT): Used to encode user query Q.

[0052] ② Visual encoder (such as MobileNetV3 style CNN): used to encode the image in the current state (such as a page, cropped block).

[0053] ③ Decision head (e.g., 3-layer MLP): It integrates textual and visual features and outputs the action ID to be executed next (4 types of softmax) and the parameters required for the action (e.g., Top-K value, detection threshold, etc.).

[0054] 3. Action Execution Environment:

[0055] (1) Function: Execute the visual action determined by the agent and return the new visual observation.

[0056] (2) Implementation: This is a virtual image processing simulator (implemented purely in Python and OpenCV, with no external API dependencies). It maintains a current visual state (image buffer, for example, with a capacity of 10 images, using a FIFO strategy). When it receives an action instruction, it calls OpenCV functions (such as cv2.resize, cv2.getRectSubPix) to process the image and returns the result (a new NumPy array of images) to the agent.

[0057] The specific steps of this intelligent document question-answering method are as follows:

[0058] Step S1: Obtain document data and convert the document data into image data.

[0059] Step S1 includes:

[0060] Step S11: Obtain document data.

[0061] Document data can include at least one of the following: text, tables, and charts.

[0062] Step S12: Process the document data based on the image noise algorithm to obtain image data.

[0063] For example, tools such as LaTeX can be used to synthesize random PDF documents containing text, tables, and charts, and noise (such as Gaussian blur algorithm) can be added to simulate scanned documents, converting the document data into image data and saving it to an image library.

[0064] Step S2: Obtain the user input text and convert the input text into visual instruction features.

[0065] Step S2 includes:

[0066] Step S21: Obtain the user input text.

[0067] Step S22: Process the user input text based on the visual language model to obtain visual instruction features.

[0068] For example, a large teacher model (such as the GPT-4V API) is used. The teacher is provided with a synthesized query (such as "What are the [attributes] of [entity] in the document?") and asked to "simulate human step-by-step reading" using a specific prompt word.

[0069] Step S3: Perform visual observation processing on the image data based on the visual instruction features to obtain the target text answer.

[0070] For example, the teacher model outputs a complete action trajectory (e.g., sequence [retrieval (K=3), search (KW='entity'), focus (zoom=2.0), integrate]) and the final ground-truth answer.

[0071] Step S3 further includes:

[0072] Step S31: Perform visual observation processing on the image data according to the visual instruction features to obtain the visual observation results, and save the visual observation results to the cache area.

[0073] Step S32: Use the image in the cache as the target text answer and output it.

[0074] Please refer to Figure 3 Visual instruction features can include: retrieval actions, search actions, focusing actions, and integration actions.

[0075] When the visual instruction feature is a retrieval action, step S31 includes:

[0076] Step A1: Embed the visual instruction features into an instruction vector and the image data into an image vector.

[0077] Step A2: Calculate the cosine similarity between the instruction vector and the image vector to obtain the image data with the highest cosine similarity.

[0078] Step A3: Use the complete page image from the image data as the visual observation result.

[0079] Among them, the retrieval action is:

[0080] (1) Simulation: Humans quickly flip through documents to locate relevant pages.

[0081] (2) Parameter: Top-K value (e.g., 3-5, predicted by the agent).

[0082] (3) Execution logic: The agent first uses the CLIP text encoder to embed the user query Q into a vector. Then, in the Faiss index, calculate... Visual embedding with all pages in the library The system calculates the cosine similarity between the images. It then returns the top-K most similar full-page images (NumPy arrays) as new observations.

[0083] (4) Innovation: This is a pure visual retrieval that can handle non-text queries such as "circular charts".

[0084] When the visual instruction feature is a search action, step S31 includes:

[0085] Step B1: Process the visual instruction features based on the lightweight object detection model to obtain shape visual features.

[0086] Step B2: Perform visual observation processing on the image data based on shape visual features to obtain cropped image information of the image data.

[0087] Step B3: Use the cropped image information as a visual observation result.

[0088] Among them, the search action is:

[0089] (1) Simulation: Humans search using "Ctrl+F" on a page, but this invention is a purely visual matching, not a text search.

[0090] (2) Parameters: Keyword KW (generated by the agent) and detection threshold (e.g., 0.5-0.9, predicted by the agent).

[0091] (3) Execution logic: Use a pre-trained lightweight object detection model (e.g., YOLOv5n). This model is not for OCR, but is trained to detect the "visual shape of keywords" (e.g., trained by rendering keywords as template images using the Pillow library). The environment runs this detection model on the current page image and returns a list of cropped images (NumPy arrays) of all detected regions along with their coordinates.

[0092] (4) Innovation: Overcomes the limitations of OCR and can “search” handwritten, artistic, or slanted text.

[0093] When the visual instruction feature is a focusing action, step S31 includes:

[0094] Step C1: Determine the coordinates of the image region and the reference image template based on the image data.

[0095] Step C2: Determine the target focused image based on the image region coordinates.

[0096] Step C3: Based on the reference image template, magnify the target focused image and use the magnified target focused image as the visual observation result.

[0097] Among them, focus action:

[0098] (1) Simulation: Humans magnify reading details.

[0099] (2) Parameters: scaling factor (e.g., 1.5-3.0, predicted by the agent).

[0100] (3) Execution logic: Receive the region coordinates C and reference image R returned by the "search action". Use OpenCV's cv2.getRectSubPix to crop the region and use cv2.resize (bilinear interpolation) to enlarge it.

[0101] (4) Output: Returns a single, high-resolution magnified image patch.

[0102] When the visual instruction feature is an integrated action, then step S32 above is executed.

[0103] Integration Action:

[0104] (1) Simulation: Terminate the action and generate an answer by integrating all information.

[0105] (2) Execution logic: The agent invokes its decoding head (e.g., a small 4-layer Transformer decoder). This decoder fuses the embedding of the user query Q with all accumulated image patches in the state buffer S. It then autoregressively generates the final text response (e.g., a maximum of 128 tokens). The loop stops when a specific terminator (e.g., ) is generated.

[0106] Based on the same inventive concept, this application also provides an interactive intelligent document question-and-answer device, which includes: an acquisition module, a conversion module, and an observation module.

[0107] The acquisition module is used to acquire document data and convert the document data into image data.

[0108] The conversion module is used to acquire user input text and convert the input text into visual instruction features.

[0109] The observation module is used to perform visual observation processing on image data based on visual instruction features to obtain the target text answer.

[0110] Furthermore, the acquisition module is also used to acquire document data, which includes at least one of the following: text, tables, and charts; and to process the document data based on an image noise algorithm to obtain image data.

[0111] Furthermore, the conversion module is also used to acquire user input text; based on the visual language model, the user input text is processed to obtain visual instruction features.

[0112] Furthermore, the observation module is also used to perform visual observation processing on image data according to visual instruction features, obtain visual observation results, and save the visual observation results to the cache area; the image in the cache area is used as the target text answer and output for display.

[0113] Furthermore, the visual instruction features include retrieval actions. The observation module is also used to embed the visual instruction features into instruction vectors and image data into image vectors; calculate the cosine similarity between the instruction vector and the image vector to obtain the image data with the highest cosine similarity; and take the complete page image in the image data as the visual observation result.

[0114] Furthermore, the visual instruction features include search actions. The observation module is also used to process the visual instruction features based on a lightweight object detection model to obtain shape visual features; perform visual observation processing on the image data based on the shape visual features to obtain cropped image information of the image data; and use the cropped image information as the visual observation result.

[0115] Furthermore, the visual instruction features include focusing actions. The observation module is also used to determine the image region coordinates and reference image template based on the image data; determine the target focused image based on the image region coordinates; magnify the target focused image based on the reference image template; and use the magnified target focused image as the visual observation result.

[0116] It is understood that the interactive intelligent document question-and-answer device provided in this application corresponds to the interactive intelligent document question-and-answer method provided in this application. In order to keep the specification concise, the same or similar parts can be referred to the content of the interactive intelligent document question-and-answer method section, and will not be repeated here.

[0117] The modules in the aforementioned interactive intelligent document question-and-answer device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the server's processor in hardware form or independent of it, or stored in the server's memory in software form, so that the processor can call and execute the operations corresponding to each module. The processor can be a central processing unit (CPU), a microprocessor, a microcontroller, etc.

[0118] The aforementioned interactive intelligent document question-answering method and / or interactive intelligent document question-answering device can be implemented in the form of a computer-readable instruction that can run on a computer system.

[0119] This application also provides a computer system including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described interactive intelligent document question-and-answer method.

[0120] The computer system can be a server. The computer system includes a processor, non-volatile storage medium, internal memory, input device, display screen, and network interface connected via a system bus. The non-volatile storage medium of the computer system can store an operating system and computer-readable instructions. When executed, these computer-readable instructions cause the processor to execute an interactive intelligent document question-and-answer method according to various embodiments of this application. The specific implementation process of this method can be found in [reference needed]. Figure 1 The specific details will not be elaborated here.

[0121] The processor of this computer system provides computing and control capabilities, supporting the operation of the entire system. The internal memory can store computer-readable instructions, which, when executed by the processor, enable the processor to perform an interactive, intelligent document question-and-answer method. The computer system's input devices are used for inputting various parameters, the display screen is used for display, and the network interface is used for network communication.

[0122] Based on the same inventive concept, this application provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps in the above-described interactive intelligent document question-and-answer method.

[0123] The memory in the embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0124] The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache.

[0125] By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0126] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs.

[0127] When computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions according to the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means.

[0128] Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more sets of available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. Semiconductor media can be solid-state drives (SSDs).

[0129] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0130] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0133] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0135] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] In this specification, references to "an embodiment" or "a specific implementation" mean that a particular feature, structure, or characteristic described in connection with that embodiment / specific implementation is included in at least one embodiment / specific implementation of the invention. Therefore, the phrase "in one embodiment / specific implementation" appearing in various places in this specification does not necessarily refer to the same embodiment / setting, but rather to potentially different embodiments. Furthermore, specific features, structures, or characteristics may be combined in one or more embodiments / settings in any suitable manner, as will be apparent to those skilled in the art from this disclosure.

[0137] Similarly, it should be understood that in the above description of exemplary embodiments / specific implementations of the invention, various features of the invention are sometimes combined in a single embodiment / specific implementation or its figures and description, with the aim of simplifying the disclosure and aiding in the understanding of one or more of the various aspects of the invention. However, the method of description in this patent should not be construed as reflecting an intention that the claimed features of the invention are more than those expressly stated in each claim, except where explicitly stated otherwise or in obvious technical contradiction or exclusion. Rather, the inventive aspect reflected in the claims lies in not all the features of a single foregoing disclosed embodiment / specific implementation. Therefore, the claims following the detailed description are expressly incorporated herein by reference, each claim existing independently as a separate embodiment / specific implementation of the invention.

[0138] Furthermore, while some embodiments / specific implementations described herein include, but are not limited to, other features included in other embodiments / specific implementations, combinations of features from different embodiments / specific implementations are intended to be within the scope of the invention and form different embodiments / specific implementations, as will be understood by those skilled in the art. For example, in the following claims, embodiments / specific implementations of any claim can be used in any combination.

[0139] The terms and expressions used in this specification are for illustrative purposes and not for limitation. In using these terms and expressions, it is not intended to exclude any equivalents of the features or portions thereof shown and described, but rather to recognize that various modifications may be possible within the scope of the invention.

[0140] Therefore, it should be understood that although the invention has been specifically disclosed through preferred embodiments, exemplary embodiments and optional features, those skilled in the art may take variations or modifications of the concepts disclosed herein, and such variations and modifications are therefore considered to be within the scope of the invention as defined by the appended claims.

[0141] The specific embodiments given in this specification are examples of useful implementations of the present invention. It will be apparent to those skilled in the art that the present invention can be implemented using many variations of the devices, device components, and method steps disclosed in this specification.

[0142] The foregoing description of specific embodiments fully discloses the general features of the present invention, enabling others to easily modify and / or adapt such specific embodiments for various applications by applying knowledge within the scope of the art, without conducting excessive experimentation and without departing from the general concept of the present invention.

[0143] Therefore, based on the teachings and guidance provided herein, it is intended that such modifications and alterations be included within the meaning and scope of equivalents of the disclosed embodiments. It should be understood that the wording or terminology used herein is for descriptive purposes and is not intended to be limiting; thus, the wording or terminology in this specification will be interpreted by those skilled in the art based on the foregoing teachings and guidance.

[0144] Furthermore, the scope of the invention should not be limited to any of the exemplary embodiments described above, but only to the appended claims and their equivalents.

Claims

1. An interactive intelligent document question-answering method, characterized in that, The intelligent document question-answering method includes: Step S1: Acquire document data and convert the document data into image data; Step S2: Obtain the user input text and convert the input text into visual instruction features; Step S3: Perform visual observation processing on the image data according to the visual instruction features to obtain the target text answer.

2. The interactive intelligent document question-answering method according to claim 1, characterized in that, Step S1 includes: Step S11: Obtain the document data, which includes at least one of the following: text, tables, and charts; Step S12: Process the document data based on the image noise algorithm to obtain the image data.

3. The interactive intelligent document question-answering method according to claim 1, characterized in that, Step S2 includes: Step S21: Obtain the user input text; Step S22: Process the user input text based on the visual language model to obtain the visual instruction features.

4. The interactive intelligent document question-answering method according to claim 1, characterized in that, Step S3 further includes: Step S31: Perform visual observation processing on the image data according to the visual instruction features to obtain visual observation results, and save the visual observation results to the cache area; Step S32: Use the image in the cache as the target text answer and output and display it.

5. The interactive intelligent document question-answering method according to claim 4, characterized in that, The visual instruction features include a retrieval action, and step S31 includes: Step A1: Embed the visual instruction features into an instruction vector, and embed the image data into an image vector; Step A2: Calculate the cosine similarity between the instruction vector and the image vector to obtain the image data with the highest cosine similarity; Step A3: Use the complete page image from the image data as the visual observation result.

6. The interactive intelligent document question-answering method according to claim 4, characterized in that, The visual instruction features include a search action, and step S3 includes: Step B1: Process the visual instruction features based on the lightweight object detection model to obtain shape visual features; Step B2: Perform visual observation processing on the image data based on the shape visual features to obtain the cropped image information of the image data; Step B3: Use the cropped image information as the visual observation result.

7. The interactive intelligent document question-answering method according to claim 4, characterized in that, The visual instruction feature includes a focusing action, and step S31 further includes: Step C1: Determine the image region coordinates and reference image template based on the image data; Step C2: Determine the target focused image based on the coordinates of the image region; Step C3: Based on the reference image template, magnify the target focused image and use the magnified target focused image as the visual observation result.

8. An interactive intelligent document question-and-answer device, characterized in that, The intelligent document question-answering device includes: an acquisition module, a conversion module, and an observation module. The acquisition module is used to acquire document data and convert the document data into image data; The conversion module is used to acquire user input text and convert the input text into visual instruction features; The observation module is used to perform visual observation processing on the image data according to the visual instruction features to obtain the target text answer.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the interactive intelligent document question-answering method according to any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the interactive intelligent document question-answering method according to any one of claims 1-7.

Images