Multi-modal intelligent question-answering system based on large model and construction method and device

Abstract

The invention provides a multi-modal intelligent question-answering system based on a large model and a construction method and device, and the method comprises the steps: carrying out the region recognition of a PDF document, and obtaining a table region picture, a text region picture and / or an image region picture; the Word document is analyzed, paragraph content of the Word document is obtained, and the paragraph content is composed of the first text, table data and / or Word pictures; constructing a knowledge base; wherein the knowledge base comprises vector representation of document slices; the document slice is determined based on the table area picture, the character area picture and / or the image area picture, or the document slice is determined based on the paragraph content and the corresponding chapter information thereof. Visibly, according to the method, layout recognition can be carried out on the PDF document, catalog chapter recognition can be carried out on the Word document, the knowledge base is constructed based on the layout recognition, the semantic integrity and coherence of multi-modal text question and answer recall can be improved, and the answer accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multimodal intelligent question-answering system based on a large model and a construction method and device. Background Art

[0002] With the development of artificial intelligence technology, deep learning and large model technology have made significant breakthroughs, and intelligent question-answering systems based on large models have become a hot topic in research and application. Intelligent question-answering systems based on large models can respond to user input, understand user intent, and provide answers to users based on knowledge bases. The knowledge base includes rich structured or unstructured data from various data sources, such as public data sets on the Internet, professional databases, etc., which enables the intelligent question-answering system to quickly retrieve information related to user questions from the knowledge base and generate accurate and comprehensive answers.

[0003] In order to provide more accurate answers and improve user experience, building a knowledge base based on multimodal data is an effective strategy. Multimodal data includes various types of data such as text, images, audio, etc. Multimodal data can be integrated into the knowledge base to enhance the capabilities of the intelligent question-answering system.

[0004] However, for multimodal data in certain formats, conventional integration methods often cannot parse out the characteristics of the document, thus limiting the accuracy and comprehensiveness of the question-answering system. For example, PDF documents contain a wealth of elements, and Word documents also contain rich semantic information. If these documents are simply segmented and stored based on conventional integration methods, it is difficult to achieve the purpose of improving the accuracy of question-answering, and the user experience is poor. Summary of the invention

[0005] The embodiments of the present application provide a multimodal intelligent question-answering system based on a large model and a construction method and device to solve the problem of poor accuracy of questions and answers in traditional intelligent question-answering systems.

[0006] In a first aspect, an embodiment of the present application provides a method for constructing a multimodal intelligent question and answer system based on a large model, the method comprising: obtaining a knowledge file, the knowledge file comprising at least a PDF document and a Word document, the PDF document and the Word document comprising table content, text content and / or image content; performing area recognition on the PDF document to obtain a table area image, a text area image and / or an image area image; parsing the Word document to obtain the paragraph content of the Word document, the paragraph content being composed of a first text, table data and / or a Word image; constructing a knowledge base so that the multimodal intelligent question and answer system provides intelligent question and answer services based on the knowledge base; wherein the knowledge base comprises a vector representation of a document slice; the document slice is determined based on the table area image, the text area image and / or the image area image, or the document slice is determined based on the paragraph content and its corresponding chapter information.

[0007] In one implementable manner, performing region recognition on a PDF document to obtain a table region image, a text region image and / or an image region image comprises: converting the PDF document page by page into at least one PDF image; performing region recognition on at least one PDF image using a PaddleDetection model to obtain a table region image, a text region image and / or an image region image; after performing region recognition on the PDF document to obtain a table region image, a text region image and / or an image region image, the step further comprises: using a pre-trained single-line text detection model to scan the table region image, identifying all text blocks in the table region image, and drawing a bounding box around each text block to mark the position of the text block; outputting a text block coordinate list through the single-line text detection model, the text block coordinate list including text block coordinates of each text block, the text block coordinates including at least two vertex coordinates of a bounding box corresponding to the text block. The SLANet model is used to extract line features from the table area image to determine the table structure information of the table in the table area image. The table structure information at least includes the number of rows and columns of the table, table boundary information, table row and column line information and cell merging information; the SLANet model is used to predict the cell coordinates of each cell in the table based on the table structure information; the table structure information and the cell coordinate list are output through the SLANet model, and the cell coordinate list includes all cell coordinates.

[0008] In one feasible method, after the step of outputting the table structure information and the cell coordinate list through the SLANet model, it also includes: using the OCR engine to perform text recognition on the table area image to obtain the text recognition result corresponding to each text block; traversing each cell coordinate in the cell coordinate list, comparing it with the text block coordinates, and obtaining the matching relationship between each cell in the table and each text recognition result; outputting the recognition result list, the recognition result list includes the matching relationship.

[0009] In one implementable manner, the document slice includes a table HTML string; after the step of outputting the recognition result list, it also includes: determining the tag information of the HTML table based on the table structure information, the tag information includes the table tag table, the row tag tr and / or the cell tag td; building the HTML table based on the tag information; traversing the recognition result list, and inserting each text recognition result into the corresponding cell of the HTML table based on the matching relationship; adding attribute tags to the merged cells in the HTML table, the attribute tags include the cross-row tag rowspan and the cross-column tag colspan; and outputting the table HTML string corresponding to the HTML table.

[0010] In one achievable method, the document slice includes a first text slice and / or an image area picture, wherein the image area picture includes formula content; after performing area recognition on the PDF document to obtain a table area picture, a text area picture and / or an image area picture, the method further includes: performing area recognition on the text area picture to obtain a title text area and a paragraph text area; performing text recognition on the title text area and the paragraph text area using an OCR engine to obtain title text and paragraph text; establishing a first mapping relationship between the title text and the paragraph text; and generating a first text slice based on the first mapping relationship.

[0011] In one achievable method, document slicing includes chapter content slicing, chapter information includes chapter titles, and paragraph content includes chapter paragraph content; parsing a Word document to obtain the paragraph content of the Word document includes: parsing the Word document to determine whether the Word document includes a directory structure; if the Word document includes a directory structure, extracting all chapter titles in the Word document; for each chapter title, identifying and extracting its corresponding chapter paragraph content; establishing a second mapping relationship between the chapter title and its corresponding chapter paragraph content; based on the second mapping relationship, generating chapter content slicing; wherein, if the number of words in the first text in the chapter paragraph content exceeds a preset threshold, the chapter content slicing includes a summary and / or FAQ question-answer pair corresponding to the chapter paragraph content, and the summary and FAQ question-answer pair are generated based on the chapter paragraph content by calling a large language model.

[0012] In one achievable method, the document slice also includes a second text slice, and the paragraph content includes segmented paragraph content; after parsing the Word document and determining whether the Word document includes a directory structure, it also includes: if the Word document does not include a directory structure, segmenting the Word document according to semantics to obtain multiple segmented paragraph contents; generating a second text slice based on the segmented paragraph contents; wherein, if the number of words in the first text in the segmented paragraph contents exceeds a preset threshold, the second text slice includes a summary and / or FAQ question-and-answer pair corresponding to the segmented paragraph contents.

[0013] In one achievable manner, the step of constructing a knowledge base includes: mapping each document slice to an embedding vector to obtain multiple first embedding vectors; constructing an index for each first embedding vector, and saving all first embedding vectors to a preset database based on the index to form a knowledge base.

[0014] In a second aspect, an embodiment of the present application further provides a multimodal intelligent question-answering system based on a large model, the multimodal intelligent question-answering system comprising a knowledge base, which is constructed based on the method for constructing a multimodal intelligent question-answering system based on a large model provided by the aforementioned first aspect and various implementations thereof;

[0015] The system also includes: an input module for receiving a query request input by a user; a vector module for mapping the query request into a second embedding vector; a retrieval module for searching in a preset database based on the second embedding vector to calculate the relevance between the second embedding vector and the first embedding vector corresponding to each document slice in the knowledge base; sorting the document slices according to the magnitude of the relevance, and selecting the first k document slices as relevant document slices, where k is equal to a preset value; a calling module for generating an input prompt that conforms to a prompt engineering template based on the relevant document slices, and calling a large language model, and inputting the relevant document slices and the input prompt into the large language model, so that the large language model can infer and generate answers based on the relevant document slices; and an output module for displaying the answers to the user.

[0016] In the third aspect, an embodiment of the present application also provides a device for constructing a multimodal intelligent question and answer system based on a large model, the device comprising: an acquisition module, used to acquire knowledge files, the knowledge files comprising at least PDF documents and Word documents, the PDF documents and Word documents containing table content, text content and / or image content; an area recognition module, used to perform area recognition on PDF documents, and obtain table area pictures, text area pictures and / or image area pictures; a parsing module, used to parse the Word document, and obtain the paragraph content of the Word document, the paragraph content being composed of a first text, table data and / or Word pictures; a construction module, used to construct a knowledge base, so that the multimodal intelligent question and answer system provides intelligent question and answer services based on the knowledge base; wherein the knowledge base includes a vector representation of document slices; the document slices are determined based on the table area pictures, text area pictures and / or image area pictures, or the document slices are determined based on the paragraph content and its corresponding chapter information.

[0017] From the above content, it can be seen that the embodiment of the present application provides a multimodal intelligent question-answering system based on a large model and a construction method and device, the method comprising: obtaining a knowledge file, the knowledge file at least comprising a PDF document and a Word document, the PDF document and the Word document containing table content, text content and / or image content; performing area recognition on the PDF document to obtain a table area image, a text area image and / or an image area image; parsing the Word document to obtain the paragraph content of the Word document, the paragraph content being composed of a first text, table data and / or a Word image; constructing a knowledge base so that the multimodal intelligent question-answering system provides intelligent question-answering services based on the knowledge base; wherein the knowledge base includes a vector representation of document slices; the document slices are determined based on the table area image, the text area image and / or the image area image, or the document slices are determined based on the paragraph content and its corresponding chapter information. The method can perform layout recognition on the PDF document, and can also perform catalog chapter recognition on the Word document. Based on this, the knowledge base can be constructed to improve the semantic integrity and coherence of the multimodal text question-answering recall, improve the accuracy of the answer, and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A first flow chart of a method for constructing a multimodal intelligent question-answering system based on a large model provided in an embodiment of the present application;

[0019] Figure 2 A second flow chart of the method for constructing a multimodal intelligent question-answering system based on a large model provided in an embodiment of the present application;

[0020] Figure 3 Schematic diagram of the PDF document and Word document provided in the embodiment of the present application;

[0021] Figure 4 A schematic diagram of a first process for parsing a PDF document provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of a second process for parsing a PDF document provided in an embodiment of the present application;

[0023] Figure 6 A third flow chart of parsing a PDF document provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of parsing a table area image provided in an embodiment of the present application;

[0025] Figure 8 A fourth flow chart of parsing a PDF document provided in an embodiment of the present application;

[0026] Fig. 9 A schematic diagram of a process for parsing a Word document provided in an embodiment of the present application;

[0027] Fig.10 A schematic diagram of the structure of a multimodal intelligent question-answering system based on a large model provided in an embodiment of the present application;

[0028] Fig.11 A schematic diagram of the structure of a device for constructing a multimodal intelligent question-answering system based on a large model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0030] In order to facilitate understanding of the technical solution of the embodiment of the present application, before describing the specific implementation of the embodiment of the present application, some technical terms in the technical field to which the embodiment of the present application belongs are briefly explained. Among them:

[0031] 1. Multimodal Data: A data set that contains two or more different types of information.

[0032] 2. Model as a Service (MaaS) platform: It is a cloud platform that provides model development, training and deployment services. The MaaS platform encapsulates complex artificial intelligence algorithm models and their related capabilities into services to lower the threshold for technology use, control application construction costs, simplify system operation and maintenance management complexity, and significantly improve the comprehensive application efficiency of artificial intelligence technology. The emergence of this model provides a solid foundation for the construction and application of multimodal intelligent document systems.

[0033] 3. Question Answer Recall: refers to the process of extracting answers or information related to user queries from a database or knowledge base in the field of natural language processing. It mainly includes the following steps:

[0034] (1) User query: The user enters a question or query.

[0035] (2) Information retrieval: The system searches for relevant information in pre-stored documents, knowledge bases, or databases based on the input question.

[0036] (3) Candidate answer generation: The system generates multiple candidate answers related to the query.

[0037] (4) Answer sorting and screening: The candidate answers are sorted by an algorithm and the most relevant or accurate answers are selected for display.

[0038] (5) Result return: present the filtered answers to the user.

[0039] With the development of artificial intelligence technology, deep learning and large model technology have made significant breakthroughs, and intelligent question-answering systems based on large models have become a hot topic in research and application. Intelligent question-answering systems based on large models can respond to user input, understand user intent, and provide answers to users based on knowledge bases. The knowledge base includes rich structured or unstructured data from various data sources, such as public data sets on the Internet, professional databases, etc., which enables the intelligent question-answering system to quickly retrieve information related to user questions from the knowledge base and generate accurate and comprehensive answers.

[0040] In order to provide more accurate answers and improve user experience, building a knowledge base based on multimodal data is an effective strategy. Multimodal data includes various types of data such as text, images, audio, etc. Multimodal data can be integrated into the knowledge base to enhance the capabilities of the intelligent question-answering system.

[0041] However, for multimodal data in certain formats, conventional integration methods often cannot parse out the characteristics of the document, thus limiting the accuracy and comprehensiveness of the question-answering system. For example, PDF documents contain a wealth of elements, and Word documents also contain rich semantic information. If these documents are simply segmented and stored based on conventional integration methods, it is difficult to achieve the purpose of improving the accuracy of question-answering, and the user experience is poor.

[0042] It can be seen that the current intelligent question-answering system faces a challenge, that is, it is difficult to effectively process multimodal content such as PDF files, scanned documents and pictures. These multimodal documents not only contain regular text, but also incorporate complex elements such as double-column layouts, handwritten notes, official seals, structured wired tables and free-form wireless tables. These elements are important components of document knowledge and are crucial for in-depth understanding of document content and enhancing the insight and judgment capabilities of intelligent question-answering systems. However, conventional data integration methods lack the ability to effectively analyze these multimodal characteristics, which limits their application in complex document processing.

[0043] When processing question and answer for Word documents, conventional data integration methods often adopt a relatively simple text segmentation strategy, that is, mechanically dividing the document according to the order of lines and paragraphs, and recalling the content based on a fixed-size window. This method ignores the inherent logical association and semantic connection between document-level titles and paragraphs, resulting in incomplete semantic coverage during question and answer retrieval. Specifically, due to the failure to fully utilize document structure information, the system may not be able to accurately capture the potential connection between the user's query intent and the deep content of the document, thereby limiting the accuracy and comprehensiveness of the question and answer system.

[0044] The embodiment of the present application provides a multimodal intelligent question-and-answer system based on a large model and a construction method and device. The construction method can perform effective data analysis on PDF documents and Word documents, and build a knowledge base based on this, thereby improving the accuracy of the answers of the intelligent question-and-answer system and improving the user experience. In a specific implementation, the embodiment of the present application can also combine the MaaS large model platform with multimodal question-and-answer to form a multimodal intelligent question-and-answer system based on the MaaS large model platform. The MaaS platform greatly reduces the threshold for developing complex multimodal applications by providing various pre-trained large models, and supports data input and output in multiple modes, so as to better realize intelligent question-and-answer of multimodal document data.

[0045] The MaaS platform plays a vital role in the multimodal question and answer of large models. This is mainly reflected in the following aspects: (1) Providing one-stop model development services. The MaasS platform provides developers and enterprises with one-stop services ranging from basic computing services, data services, model training services to application deployment and construction. (2) Supporting multi-source heterogeneous basic large models. (3) Simplifying the model development process. (4) Optimizing model performance. (5) Improving the intelligence level of the question and answer system. The multimodal question and answer system built on the MassS platform can make full use of the powerful capabilities of large models to achieve a more intelligent and personalized question and answer experience. The system can provide answers and suggestions that better meet user needs based on the user's historical data and behavioral habits. At the same time, the large model can also continue to learn and evolve to adapt to the ever-changing question and answer scenarios and needs.

[0046] Figure 1 A first flow chart of a method for constructing a multimodal intelligent question-answering system based on a large model provided in an embodiment of the present application.

[0047] Figure 2 A second flow chart of the method for constructing a multimodal intelligent question-answering system based on a large model provided in an embodiment of the present application.

[0048] like Figure 1 and Figure 2 As shown, the method for constructing a multimodal intelligent question-answering system provided in an embodiment of the present application includes the following steps S100-S400.

[0049] S100: Acquire a knowledge file, where the knowledge file includes at least a PDF document and a Word document, and the PDF document and the Word document contain table content, text content and / or image content.

[0050] Among them, the data sources of knowledge files may include internal sources and external sources. Internal sources may be local files, such as internal data generated during the production and operation of an enterprise, including sales data, inventory data, etc. External sources are, for example, public online databases, which are not specifically limited in the embodiments of the present application. This step can ensure that the intelligent question-and-answer system can access a variety of information sources and provide a rich data foundation for questions and answers. Especially in vertical field applications, by accurately selecting and integrating internal and external data sources, the intelligent question-and-answer system can provide users with more professional and in-depth knowledge answers to meet their information needs in specific fields.

[0051] Figure 3 A PDF document and a Word document are shown as examples. It is understandable that different documents usually contain different contents, such as only text content, or table content, text content and image content. The embodiments of the present application do not make specific limitations on this.

[0052] In the embodiment of the present application, the knowledge file is not limited to PDF format and Word format, but may also include Excel documents, PPT documents or TXT documents, etc.

[0053] S200: Performing region recognition on the PDF document to obtain a table region image, a text region image and / or an image region image.

[0054] Figure 3 The PDF document in exemplifies the table area, text area and image area after area recognition. The embodiments of the present application can execute different document parsing strategies for different documents. The step of performing area recognition on the PDF document can be specifically performed based on the PaddleDetection model, which can be obtained from the MaaS platform or developed based on actual needs. The embodiments of the present application do not make specific limitations on this.

[0055] Based on the PaddleDetection model, PDF documents can be divided into regions and key areas such as text, titles, tables, pictures, etc. can be located. In this way, different areas can be distinguished from each other to accurately extract various types of information in PDF documents, thereby improving the efficiency of document processing and the accuracy of information acquisition.

[0056] S300: Parse the Word document to obtain paragraph content of the Word document, where the paragraph content is composed of a first text, table data and / or a Word image, wherein the first text corresponds to the text content in the Word document, the table data corresponds to the table content in the Word document, and the Word image corresponds to the image content in the Word document.

[0057] Figure 3 The Word document in exemplifies the paragraph content. The embodiment of the present application can perform structural processing on the Word document and obtain the chapter information of the paragraph content, such as the chapter title, table of contents, paragraph number, line number, page number and other information. In this way, the paragraph relationship of the Word document can be accurately parsed, and the paragraph content can be associated with its chapter to maintain semantic consistency, which can avoid the separation of the paragraph content and the chapter title, and further avoid the problem of incomplete recall text in the question and answer recall stage.

[0058] S400: Build a knowledge base so that the multimodal intelligent question-answering system provides intelligent question-answering services based on the knowledge base; wherein the knowledge base includes vector representations of document slices; document slices are determined based on table area images, text area images and / or image area images, or, document slices are determined based on paragraph content and its corresponding chapter information.

[0059] For document slicing, the embodiment of the present application can perform vectorization operations on it, and then save it in the database to form a knowledge base for subsequent retrieval. The document slicing is specifically based on the table html string determined by the PDF document, the first text slice, the image area picture obtained by area recognition, and can also be based on the chapter content slice and the second text slice determined by the Word document. The steps of parsing the document will be described in detail below and will not be repeated here.

[0060] From the above content, it can be seen that the embodiment of the present application provides a method for constructing a multimodal intelligent question-answering system based on a large model, the method comprising: obtaining a knowledge file, the knowledge file at least comprising a PDF document and a Word document, the PDF document and the Word document comprising table content, text content and / or image content; performing region recognition on the PDF document to obtain a table region image, a text region image and / or an image region image; parsing the Word document to obtain the paragraph content of the Word document, the paragraph content being composed of a first text, table data and / or a Word image; constructing a knowledge base so that the multimodal intelligent question-answering system provides intelligent question-answering services based on the knowledge base; wherein the knowledge base comprises a vector representation of document slices; the document slices are determined based on table region images, text region images and / or image region images, or the document slices are determined based on the paragraph content and its corresponding chapter information. The method can perform layout recognition on a PDF document, and can also perform catalog chapter recognition on a Word document. Based on this, the knowledge base can be constructed to improve the semantic integrity and coherence of multimodal text question-answering recall, improve the accuracy of answers, and improve the user experience.

[0061] The following is a detailed introduction to the PDF document parsing steps.

[0062] Figure 4 A schematic diagram of a first process for parsing a PDF document provided in an embodiment of the present application.

[0063] Figure 5 A second flow chart of parsing a PDF document provided in an embodiment of the present application.

[0064] like Figure 4 and Figure 5 As shown, step S200 may specifically include the following steps S201-S202.

[0065] S201: Convert a PDF document into at least one PDF image page by page.

[0066] In this step, if the PDF document is a single-page document, you can directly convert the PDF document to an image format, such as JPEG or PNG format, to obtain a PDF image. If the PDF document is a multi-page document, you can first split the PDF document into pages, and then convert each page into a PDF image. In this step, you can use a PDF to image tool to process.

[0067] S202: Use the PaddleDetection model to perform region recognition on at least one PDF image to obtain a table region image, a text region image and / or an image region image.

[0068] It can be understood that the PaddleDetection model here is only introduced as an example. In practical applications, other models can be used for region recognition, and the embodiments of the present application do not make specific limitations on this.

[0069] After region recognition, the annotation information of each region can be obtained, including the coordinates, category, region pixel value information and confidence level of each region. Specifically, the following steps may be included:

[0070] 1. Image preprocessing: Before inputting the PDF image into the model, it is first preprocessed, including resizing, standardization, and data enhancement, to improve the recognition accuracy of the model.

[0071] 2. Region detection: Input the processed image into the PaddleDetection model, and the model can identify different regions in the image based on the trained weights. The recognition results may include:

[0072] ① Table area image: Identify the table part in the image, and the generated area can be extracted as a separate image file, namely the table area image, to facilitate subsequent data analysis.

[0073] ②Text area image: Identify the area containing text and generate the corresponding text area image, i.e., the text area image, which can be subsequently combined with optical character recognition (OCR) technology for text extraction.

[0074] ③ Image area picture: Identify other important contents in the image, such as illustrations, formulas, icons, etc., and save them as independent image files, i.e., image area pictures. It is worth noting that the embodiment of the present application can identify formulas in PDF images and output them as image area pictures to improve the intelligent question-answering system's ability to understand formulas.

[0075] Furthermore, region recognition can also include a step of result visualization, in which the recognition results can be displayed in a visual form, such as drawing a bounding box on the original image (PDF image) and marking the categories and confidence levels of different regions. This not only facilitates verification of the recognition effect of the model, but also provides an intuitive basis for subsequent processing. Categories include, for example, table, text, and title.

[0076] Furthermore, the embodiment of the present application can save the identified area as an image file, and the file name can adopt a standardized format, such as "table_area_1.jpg", "text_area_1.png", etc., to facilitate subsequent search and use.

[0077] In addition, the identified result information (such as coordinates and categories) can also be exported in JSON or CSV format to facilitate further data analysis and processing.

[0078] Furthermore, the embodiment of the present application includes a step of parsing the table area image in the PDF document.

[0079] Figure 6 A third flow chart of parsing a PDF document provided in an embodiment of the present application.

[0080] Figure 7 A schematic diagram of parsing a table area image provided in an embodiment of the present application.

[0081] like Figure 6 and Figure 7 As shown, after step S200, the following steps S501-S505 may also be included.

[0082] S501: Use a pre-trained single-line text detection model to scan the table area image, identify all text blocks in the table area image, and draw a bounding box around each text block to mark the position of the text block.

[0083] This step needs to use a text detection algorithm. First, the table area image can be input into a single-line text detection model. The single-line text detection model has been trained with a large amount of data and has a highly accurate text detection capability. It can automatically identify all text blocks (or single-line texts) in the table area image. In order to better visualize the recognition result, the embodiment of the present application can draw a bounding box around each text block to mark the position of the text block. This is not only convenient for the user to view the recognition effect, but also provides accurate text position information for subsequent processing. The text block can be the content in a single cell (cell) in the table, and can include various characters such as numbers, letters, symbols, etc. The embodiment of the present application does not specifically limit this. It is understandable that for some cells where frame lines are invisible, the embodiment of the present application can distinguish them from merged cells.

[0084] In some implementations, the single-line text detection model can be obtained from the MaaS platform, which is not specifically limited in the embodiments of the present application.

[0085] S502: Outputting a text block coordinate list through a single-line text detection model, the text block coordinate list including text block coordinates of each text block, the text block coordinates including at least two vertex coordinates of a bounding box corresponding to the text block.

[0086] It is understandable that the text block coordinate list records the location information of each text block, and at least two vertices can be the upper left corner coordinates and the lower right corner coordinates of the bounding box. The storage format of the at least two vertex coordinates can be (x1, y1, x2, y2), or (x1, y1) is the upper left corner coordinate and (x2, y2) is the lower right corner coordinate. In some implementations, the text block coordinates can also include the four vertex coordinates of its corresponding bounding box, which is not specifically limited in the embodiments of the present application.

[0087] S503: Use the SLANet model to extract line features from the table area image, and determine the table structure information of the table in the table area image. The table structure information at least includes the number of rows and columns of the table, table boundary information, table row and column line information, and cell merging information.

[0088] In this step, the SLANet model can identify features such as the table's boundaries, row lines, and column lines, and infer the boundaries of each cell in the table based on this. Specifically, the SLANet model is a deep learning model for table structure parsing, which can accurately identify line features in the table, including the table's boundary lines, row and column lines, etc. Through these line features, the SLANet model can infer the number of rows and columns in the table, the table's boundary information, and the specific location of the table's row and column lines. In addition, the model can also identify cell merging information, that is, which cells are merged together. Together, this information constitutes the complete structural information of the table, providing a key basis for subsequent cell coordinate prediction.

[0089] In some implementations, the SLANet model may be obtained from the MaaS platform, which is not specifically limited in the embodiments of the present application.

[0090] S504: Predict the cell coordinates of each cell in the table based on the table structure information using the SLANet model.

[0091] In an embodiment of the present application, the cell coordinates may be the coordinates of the four vertices of the cell, or may be the coordinates of the upper left corner and the lower right corner. The specific storage format may be (x3, y3, x4, y4), or (x2, y2) for the upper left corner coordinates and (x3, y3) for the lower right corner coordinates. The embodiment of the present application does not make any specific limitation on this.

[0092] S505: Outputting table structure information and a cell coordinate list through the SLANet model, wherein the cell coordinate list includes all cell coordinates.

[0093] It is understandable that the cell coordinate data accurately describes the position of each cell in the table area image. By comparing and analyzing these coordinate data, we can obtain information such as the size and shape of each cell and the relative position relationship between each cell. This facilitates the subsequent extraction and processing of table data.

[0094] Continue to see Figure 6 , the following steps S506-S508 may also be included after step S505.

[0095] S506: Use the OCR engine to perform text recognition on the table area image to obtain a text recognition result corresponding to each text block.

[0096] This step requires the use of a file recognition algorithm, and a text recognition model such as an OCR engine can be used for text recognition. In this step, the OCR engine can accurately extract text information from the table area image and convert it into an editable and searchable text format.

[0097] In some implementations, the OCR engine may be obtained from the MaaS platform, which is not specifically limited in the embodiments of the present application.

[0098] S507: traverse each cell coordinate in the cell coordinate list, compare it with the text block coordinate, and obtain the matching relationship between each cell in the table and each character recognition result.

[0099] This step is a step of cell coordinate aggregation and cell text aggregation. It can be understood that step S507 includes three input data, namely the text recognition result obtained from the text recognition model, the cell coordinate list obtained from the SLANet model, and the text block coordinate list obtained from the single-line text detection model.

[0100] It is worth noting that “each cell in the table” in step S507 includes all cells with visible borders and all cells with invisible borders.

[0101] In the embodiment of the present application, the specific cell corresponding to each text block can be determined through a matching operation.

[0102] S508: Output a recognition result list, which includes matching relationships.

[0103] Exemplarily, the recognition result list may be as follows:

[0104]

[0105]

[0106] It is worth noting that the format of the recognition result list can be a structured data format, such as JSON or CSV, to facilitate subsequent data analysis and processing.

[0107] Furthermore, after step S508, the following steps S509-S513 may also be included.

[0108] S509: Determine tag information of the HTML table based on the table structure information, where the tag information includes a table tag table, a row tag tr, and / or a cell tag td.

[0109] In this step, we first determine the basic tags of the HTML table based on the identified table structure information. Specifically, we can use Tags to define the entire table, and then use Labels identify rows, and each cell uses The tags indicate that these tags together form the basic framework of the table. It should be noted that if the table has merged cells, you need to define corresponding attributes for these cells, including the attribute tag rowspan used to indicate the number of rows spanned by the cell, and the attribute tag colspan used to indicate the number of columns spanned by the cell.

[0110] S510: Constructing an HTML table based on the tag information.

[0111] In this step, the string structure of the HTML table can be constructed based on the tag information generated in the previous step. Specifically, each tag needs to be combined in the correct order and nested relationship to form a complete HTML table code. In this process, a certain tag closing and nesting order can be followed to ensure that the generated HTML table code complies with the HTML standard and can be correctly parsed and displayed by the browser.

[0112] S511: traverse the recognition result list, and insert each character recognition result into a corresponding cell of the HTML table based on the matching relationship.

[0113] In this step, you can traverse the previously generated recognition result list, and insert the corresponding text content into the corresponding HTML table cell according to the matching relationship between each text recognition result and the table cell. In this way, the table data originally in the form of images can be converted into editable and searchable HTML table data.

[0114] S512: Add attribute tags to the merged cells in the HTML table, where the attribute tags include a rowspan tag and a colspan tag.

[0115] In the embodiment of the present application, the rowspan attribute can be used to mark the merged cells across rows, and the colspan attribute can be used to mark the merged cells across columns. These attributes will clearly indicate the number of rows or columns spanned by the merged cells, thereby ensuring that the HTML table can accurately reflect the structure and layout of the original table.

[0116] S513: Output the table HTML string corresponding to the HTML table, that is, generate document slices.

[0117] In this way, the document slices generated by the table information in the PDF file can be included in the knowledge base and used by the intelligent question answering system. Specifically, the table html string contains the complete HTML table code, which can be directly embedded in the web page running the intelligent question answering system or transmitted and used by other means.

[0118] It is worth noting that the embodiment of the present application can construct markdown tags based on the table structure information to achieve structured storage of table information, and the embodiment of the present application does not specifically limit this.

[0119] In some implementations, when the table result in the table area image is too large, cropping and splitting steps can be taken for parsing, such as cropping by rows, columns, or according to specific areas, and then performing the parsing steps. The embodiments of the present application do not make specific limitations on this.

[0120] Furthermore, the embodiment of the present application also includes the step of parsing the text area image in the PDF document.

[0121] Figure 8 A fourth flow chart of parsing a PDF document provided in an embodiment of the present application.

[0122] like Figure 8 As shown, the document slice includes the first text slice. After step S200, the following steps S601-S604 may also be included.

[0123] S601: Performing region recognition on the text region image to obtain a title text region and a paragraph text region.

[0124] This step can be performed based on the PaddleDetection model and will not be described in detail here.

[0125] S602: Use an OCR engine to perform text recognition on the title text area and the paragraph text area to obtain the title text and the paragraph text.

[0126] In this way, the text information in the text area image can be accurately extracted and converted into an editable and searchable text format.

[0127] S603: Establish a first mapping relationship between the title text and the paragraph text.

[0128] In this step, a mapping relationship between the title text and the paragraph text can be established. This mapping relationship helps to understand the structure of the text, reflects the logical connection between the title and the paragraph, and clarifies which title corresponds to which paragraph. In this way, the semantic consistency of the text can be maintained, and the semantic integrity and coherence during question-answering recall can be improved.

[0129] S604: Generate a first text slice based on the first mapping relationship.

[0130] It is understandable that text slicing is a way to divide and organize document content according to a specific logic or structure. The embodiment of the present application can combine the title and paragraph text based on the first text slice, so that the first text slice not only retains the structure and content of the original document, but also makes the information easy to understand and use. The first text slice is, for example, in JSON or XML format, which is convenient for storage and text analysis.

[0131] In the embodiment of the present application, the document slice may also include an image area picture, which may include formula content, logo, etc. Specifically, the embodiment of the present application may directly save the image area picture as a document slice, so that the image area picture is included in the knowledge base.

[0132] It is worth noting that the embodiment of the present application may also include a layout restoration step, in which the layout restoration model may be called to reconstruct and restore the layout of the table area image, the text area image, and the image area image. After the layout restoration is completed, a digital document that is highly consistent with the original document layout may be obtained. This document not only contains all the original information, but also provides table structure information, etc. After the layout is restored, the embodiment of the present application may also save the layout restoration document in HTML format or markdown format, and then save it in a preset database. The embodiment of the present application does not specifically limit this. During the question-and-answer process, the intelligent question-and-answer system can directly output a PDF document with a restored layout to the user, so that the user can more conveniently browse, search and share the content of the PDF document.

[0133] It should be noted that the step of parsing the PDF document can be specifically performed by modules such as a table recognition module, a layout detection module (Layout Detection), a text recognition module, a layout restoration module, a formula recognition module and / or an image recognition module, and the embodiments of the present application do not specifically limit this.

[0134] The following is a detailed introduction to the parsing steps of Word documents.

[0135] It is understandable that a Word document may have a directory structure such as hierarchical titles, or may not have a directory structure. The embodiment of the present application may perform different parsing steps for Word documents with a directory structure and Word documents without a directory structure to obtain chapter content slices and second text slices.

[0136] Specifically, the document slice includes a chapter content slice and / or a second text slice, the chapter information may include a chapter title, and the paragraph content may include a chapter paragraph content and / or a segmented paragraph content. Figure 3 An exemplary introduction is given to the contents of chapters and paragraphs and the contents of sub-paragraphs.

[0137] Fig. 9 A schematic diagram of the process of parsing a Word document provided in an embodiment of the present application.

[0138] like Fig. 9 As shown, step S300 may include the following steps S301-S305.

[0139] S301: Parse the Word document to determine whether the Word document includes a directory structure.

[0140] This step can parse the Word document using the Office Open XML specification to analyze whether there are text paragraphs using specific styles in the Word document, such as "Heading 1", "Heading 2", etc. If these styles exist in the Word document, it can be determined that it has a directory structure.

[0141] S302: If the Word document includes a directory structure, extract all chapter titles in the Word document.

[0142] The embodiment of the present application can traverse each paragraph in the Word document one by one, identify and collect all title information. It is worth noting that in this step, not only the name of the chapter title can be extracted, but also the hierarchical relationship of each chapter title in the document, such as the first-level title, the second-level title, the sub-title, etc., can be extracted for subsequent processing.

[0143] S303: For each chapter title, identify and extract the corresponding chapter paragraph content.

[0144] In this step, for each chapter title, its corresponding chapter and paragraph content can be further identified and extracted. The embodiment of the present application can locate the start and end positions of each chapter and extract all the chapter and paragraph content in the chapter. It should be emphasized here that the chapter and paragraph content includes not only text, but also tables and pictures.

[0145] S304: Establish a second mapping relationship between the chapter title and its corresponding chapter paragraph content.

[0146] In the embodiment of the present application, the second mapping relationship can clarify the correspondence between each chapter title and all the chapter and paragraph contents it contains. This mapping relationship helps to understand the structure of the text, reflects the logical connection between the title and the paragraph, and clarifies which title corresponds to which paragraph. In this way, the semantic consistency of the text can be maintained, and the semantic integrity and coherence during question and answer recall are improved.

[0147] S305: Generate chapter content slices based on the second mapping relationship. If the word count of the first text in the chapter content exceeds a preset threshold, the chapter content slices include a summary and / or FAQ question-answer pairs corresponding to the chapter content, and the summary and FAQ question-answer pairs are generated based on the chapter content by calling a large language model (LLM).

[0148] The embodiment of the present application can combine the title and paragraph text based on chapter content slicing, so that the chapter content slicing not only retains the structure and content of the original document, but also makes the information easy to understand and use. The chapter content slicing is in JSON or XML format, for example, which is convenient for storage and text analysis.

[0149] Further, if the number of words in the first text in a certain chapter paragraph content exceeds a preset preset threshold, then the paragraph can be further processed to generate a summary and / or FAQ question and answer pair. Wherein, the preset threshold is, for example, 1000 words or 2000 words, and the embodiment of the present application does not specifically limit this. The summary can extract the core information of the chapter, and the Frequently Asked Questions and Answers (FAQ) pairs, referred to as FAQ question and answer pairs, are common questions and their corresponding answers generated based on the chapter content. This process will be achieved by calling a large language model to ensure the accuracy and relevance of the generated content, improve comprehension ability, and improve the accuracy of question and answer matching.

[0150] Further, in the embodiment of the present application, the document slice may further include a second text slice, and the paragraph content may include segmented paragraph content. Fig. 9 , after step S301, the following steps S306-S307 may also be included.

[0151] S306: If the Word document does not include a directory structure, segment the Word document according to semantics to obtain a plurality of segmented paragraph contents.

[0152] In the embodiment of the present application, the step of segmenting the Word document according to semantics can be achieved by calling a large language model. Specifically, combined with the context understanding ability of the large language model, the semantic relationship between the paragraphs of the Word document is identified, and the paragraphs with similar semantics are divided into the same segmented paragraph content. In this way, the paragraph segmentation can be prevented from being too fragmented.

[0153] In some implementations, the number of words in the first text included in the content of a single segment paragraph is, for example, 1,000 words or 2,000 words, which is not specifically limited in the embodiments of the present application.

[0154] S307: Generate a second text slice based on the segmented paragraph content; wherein, if the number of words in the first text in the segmented paragraph content exceeds a preset threshold, the second text slice includes a summary and / or FAQ question-and-answer pair corresponding to the segmented paragraph content.

[0155] In the embodiment of the present application, the second text slice is in JSON or XML format, for example, to facilitate storage and text analysis. The summary and FAQ question-answer pair in the second text slice can be determined by calling a large language model.

[0156] In some implementations, the embodiments of the present application may also call image parsing models and table parsing models to parse images and tables in chapter and paragraph content and segmented paragraph content, and add the parsed information to the chapter content slice or the second text slice to improve the parsing effect and thereby improve the accuracy of questions and answers.

[0157] It is necessary to add that, if Figure 2 As shown in FIG. 1 , various knowledge files are parsed to obtain structured data (document slices), specifically, the step of constructing documents. Figure 2 "Documents" is used to refer to "document slices".

[0158] Continue to see Figure 2 In the embodiment of the present application, step S400 may include the following steps S401-S402.

[0159] S401: Map each document slice to an embedding vector to obtain a plurality of first embedding vectors.

[0160] This step is a vectorization step. The embedding vector can capture the semantic information and contextual relationship of the text, so that similar or related texts are located close to each other in the vector space. This step can be implemented by calling the embedding vector model or the embedding layer of other models.

[0161] S402: construct an index for each first embedding vector, and save all the first embedding vectors to a preset database based on the index to form a knowledge base.

[0162] In the embodiment of the present application, the preset database is, for example, a Milvus database, and the index is, for example, a Milvus index. This is only an exemplary introduction. The preset database may also be an SQL database, which is not limited in the embodiment of the present application. In this way, the knowledge base can be used for retrieval by the intelligent question-answering system.

[0163] It is worth noting that the embodiment of the present application can also parse Excel files and PPT files to achieve structured storage. For example, for Excel files, the embodiment of the present application can use Open XML technology and large language model technology for efficient processing. Specifically, the embodiment of the present application can extract all the content from the Excel file. Then, the table splitting operation is performed. This step is completed based on the number of tokens (Token) of each row of content. When splitting, the context length limit of the large language model can also be referred to to ensure the rationality of each small table, and the header information of the original table can be retained in this step. Finally, the large language model is used to perform in-depth analysis of the content of the small table, generate a summary, and extract FAQ questions to improve the accuracy of table matching, thereby improving the accuracy of question and answer.

[0164] For PPT documents, the embodiment of the present application can parse the PPT content based on the Office Open XML specification, accurately detect and extract the directory structure tree information. Subsequently, the embodiment of the present application can search and extract the paragraph content according to the division of the seed chapter of the directory structure tree, and can parse out key information such as titles, texts, tables, and pictures according to the layout of the PPT. On this basis, the embodiment of the present application can use a large language model to generate summaries and extract questions for the content of each chapter to improve the accuracy of PPT content matching, thereby improving the accuracy of question and answer.

[0165] It can be seen that the structured processing steps of the embodiments of the present application, such as vectorizing document slices, can enable the intelligent question and answer system to perform comparisons and operations in the vector space, thereby improving the semantic understanding capabilities of the intelligent question and answer system, and thereby providing users with more accurate and intelligent services.

[0166] Fig.10 A schematic diagram of the structure of a large-model-based multimodal intelligent question-answering system provided in an embodiment of the present application.

[0167] like Figure 2 and Fig.10 As shown, an embodiment of the present application can also provide a multimodal intelligent question-answering system based on a large model, which can include a knowledge base 1001, and the knowledge base can be constructed based on the aforementioned multimodal intelligent question-answering system construction method based on a large model.

[0168] The system may also include:

[0169] The input module 1002 is used to receive a query request input by a user.

[0170] The query request can also be called "text to be searched". The query request can be in text format or image format, for example, "Please tell me how to tokenize text."

[0171] The vector module 1003 is used to map the query request into a second embedding vector.

[0172] This step can be implemented by calling the embedding vector model. In this way, the embedding vector of the search text (query request) can be obtained.

[0173] The retrieval module 1004 is used to search in a preset database based on the second embedding vector to calculate the relevance of the second embedding vector with the first embedding vector corresponding to each document slice in the knowledge base 1001; sort the document slices according to the size of the relevance, and select the first k document slices as relevant document slices, where k is equal to a preset value.

[0174] For example, when a user asks a question, the retrieval module 1004 can convert the question into a second embedding vector and then retrieve the most similar first embedding vector in the knowledge base to determine the most relevant answer.

[0175] The step of calculating the correlation is, for example, calculating the cosine similarity or Euclidean distance between the first embedding vector and the second embedding vector, which is not specifically limited in the embodiments of the present application. For example, k can be equal to 2, 3 or 5, which can be adjusted based on actual needs, which is not specifically limited in the embodiments of the present application.

[0176] The calling module 1005 is used to generate an input prompt that conforms to the prompt engineering template based on the relevant document slices, and to call the large language model 1006 and input the relevant document slices and the input prompt to the large language model 1006, so that the large language model 1006 can infer and generate answers, i.e., search results, based on the relevant document slices.

[0177] The prompt engineering template may be preset, and the prompt engineering template specifically defines the format of the input prompt, such as length, structure, and wording, based on which the input prompt can guide the large language model 1006 to better understand and process the information of the relevant document slices.

[0178] The output module 1007 is used to display the answer to the user.

[0179] In this way, the semantic integrity and coherence of text question and answer recall can be improved, the accuracy of answers can be improved, and the user experience can be improved. Users can easily and quickly obtain the required information and answers, and achieve efficient knowledge query and acquisition.

[0180] Fig.11 A schematic diagram of the structure of a device for constructing a multimodal intelligent question-answering system based on a large model provided in an embodiment of the present application.

[0181] like Fig.11 As shown, the embodiment of the present application also provides a device for constructing a multimodal intelligent question-answering system based on a large model, the device comprising:

[0182] The acquisition module 2001 is used to acquire knowledge files, which include at least PDF documents and Word documents, and the PDF documents and Word documents contain table content, text content and / or image content;

[0183] The region identification module 2002 is used to perform region identification on the PDF document to obtain a table region image, a text region image and / or an image region image;

[0184] The parsing module 2003 is used to parse the Word document to obtain the paragraph content of the Word document, where the paragraph content is composed of the first text, table data and / or Word picture;

[0185] Construction module 2004 is used to construct a knowledge base so that the multimodal intelligent question-answering system provides intelligent question-answering services based on the knowledge base; wherein the knowledge base includes vector representations of document slices; document slices are determined based on table area images, text area images and / or image area images, or document slices are determined based on paragraph content and its corresponding chapter information.

[0186] In some implementations, the region identification module 2002 is specifically used to: convert a PDF document into at least one PDF image page by page; use a PaddleDetection model to perform region identification on at least one PDF image to obtain a table region image, a text region image and / or an image region image.

[0187] The area recognition module 2002 is also used for: using a pre-trained single-line text detection model to scan the table area image, identifying all text blocks in the table area image, and drawing a bounding box around each text block to mark the position of the text block; outputting a text block coordinate list through the single-line text detection model, the text block coordinate list including the text block coordinates of each text block, the text block coordinates including at least two vertex coordinates of the bounding box corresponding to the text block; using the SLANet model to extract line features from the table area image, determining the table structure information of the table in the table area image, the table structure information at least including the number of rows and columns of the table, table boundary information, table row and column line information and cell merge information; using the SLANet model to predict the cell coordinates of each cell in the table based on the table structure information; outputting the table structure information and the cell coordinate list through the SLANet model, the cell coordinate list including all cell coordinates.

[0188] In some implementations, the area identification module 2002 is also used to: use the OCR engine to perform text recognition on the table area image to obtain the text recognition result corresponding to each text block; traverse the cell coordinates in the cell coordinate list, compare them with the text block coordinates, and obtain the matching relationship between each cell in the table and each text recognition result; output the recognition result list, which includes the matching relationship.

[0189] In some implementations, the document slice includes a table HTML string; the region identification module 2002 is also used to: determine the tag information of the HTML table based on the table structure information, the tag information includes the table tag table, the row tag tr and / or the cell tag td; build the HTML table based on the tag information; traverse the recognition result list, and insert each text recognition result into the corresponding cell of the HTML table based on the matching relationship; add attribute tags to the merged cells in the HTML table, the attribute tags include the row span tag rowspan and the column span tag colspan; output the table HTML string corresponding to the HTML table.

[0190] In some implementations, the document slice includes a first text slice and / or an image area picture, and the image area picture includes formula content; the area identification module 2002 is also used to: perform area identification on the text area picture to obtain a title text area and a paragraph text area; use an OCR engine to perform text recognition on the title text area and the paragraph text area to obtain title text and paragraph text; establish a first mapping relationship between the title text and the paragraph text; and generate a first text slice based on the first mapping relationship.

[0191] In some implementations, document slices include chapter content slices, chapter information includes chapter titles, and paragraph content includes chapter paragraph content; the parsing module 2003 is specifically used to: parse the Word document to determine whether the Word document includes a directory structure; if the Word document includes a directory structure, extract all chapter titles in the Word document; for each chapter title, identify and extract its corresponding chapter paragraph content; establish a second mapping relationship between the chapter title and its corresponding chapter paragraph content; based on the second mapping relationship, generate chapter content slices; wherein, if the number of words in the first text in the chapter paragraph content exceeds a preset threshold, the chapter content slices include a summary and / or FAQ question-and-answer pair corresponding to the chapter paragraph content, and the summary and FAQ question-and-answer pair are generated based on the chapter paragraph content by calling a large language model.

[0192] In some implementations, the document slice also includes a second text slice, and the paragraph content includes segmented paragraph content; the parsing module 2003 is also used to: if the Word document does not include a directory structure, segment the Word document according to semantics to obtain multiple segmented paragraph contents; generate a second text slice based on the segmented paragraph content; wherein, if the number of words in the first text in the segmented paragraph content exceeds a preset threshold, the second text slice includes a summary and / or FAQ question-and-answer pair corresponding to the segmented paragraph content.

[0193] In some implementations, the construction module 2004 is specifically used to: map each document slice to an embedding vector to obtain multiple first embedding vectors; construct an index for each first embedding vector, and save all first embedding vectors to a preset database based on the index to form a knowledge base.

[0194] In a specific implementation, the present invention further provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment of the method for constructing a multimodal intelligent question-answering system based on a large model provided by the present invention. The storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0195] It is easy to understand that those skilled in the art can combine, split, reorganize, etc. the embodiments of the present application to obtain other embodiments based on the several embodiments provided in the present application, and these embodiments do not exceed the protection scope of the present application.

[0196] The above specific implementation methods further explain in detail the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above are only specific implementation methods of the embodiments of the present application and are not used to limit the protection scope of the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A method for constructing a multimodal intelligent question-answering system based on a large model, characterized in that: The method comprises: Acquire a knowledge file, wherein the knowledge file includes at least a PDF document and a Word document, wherein the PDF document and the Word document contain table content, text content and / or image content; Performing region recognition on the PDF document to obtain a table region image, a text region image and / or an image region image; Parsing the Word document to obtain paragraph content of the Word document, wherein the paragraph content is composed of a first text, table data and / or a Word picture; Construct a knowledge base so that the multimodal intelligent question-answering system provides intelligent question-answering services based on the knowledge base; wherein the knowledge base includes vector representations of document slices; the document slices are determined based on the table area image, the text area image and / or the image area image, or the document slices are determined based on the paragraph content and its corresponding chapter information.

2. The method for constructing a multimodal intelligent question-answering system based on a large model according to claim 1, characterized in that: The step of performing area recognition on the PDF document to obtain a table area picture, a text area picture and / or an image area picture comprises: Convert the PDF document page by page into at least one PDF image; Using the PaddleDetection model to perform region recognition on at least one of the PDF images to obtain the table region image, the text region image and / or the image region image; After performing region recognition on the PDF document to obtain a table region image, a text region image and / or an image region image, the step further includes: Scanning the table area image using a pre-trained single-line text detection model to identify all text blocks in the table area image, and drawing a bounding box around each of the text blocks to mark the position of the text block; Outputting a text block coordinate list through a single-line text detection model, wherein the text block coordinate list includes text block coordinates of each text block, and the text block coordinates include at least two vertex coordinates of the bounding box corresponding to the text block; Use the SLANet model to extract line features from the table area image, and determine table structure information of the table in the table area image, wherein the table structure information at least includes the number of rows and columns of the table, table boundary information, table row and column line information, and cell merging information; Predicting the cell coordinates of each cell in the table based on the table structure information using the SLANet model; The table structure information and the cell coordinate list are outputted through the SLANet model, and the cell coordinate list includes all the cell coordinates.

3. The method for constructing a multimodal intelligent question-answering system based on a large model according to claim 2, characterized in that: After the step of outputting the table structure information and the cell coordinate list through the SLANet model, the method further includes: Use an OCR engine to perform text recognition on the table area image to obtain a text recognition result corresponding to each text block; Traversing each of the cell coordinates in the cell coordinate list, comparing them with the text block coordinates, and obtaining a matching relationship between each cell in the table and each of the text recognition results; Output a recognition result list, wherein the recognition result list includes the matching relationship.

4. The method for constructing a multimodal intelligent question-answering system based on a large model according to claim 3 is characterized in that: The document slice includes a table HTML string; after the step of outputting the recognition result list, it also includes: Determine tag information of an HTML table based on the table structure information, wherein the tag information includes a table tag table, a row tag tr and / or a cell tag td; Constructing an HTML table based on the tag information; Traversing the recognition result list, and inserting each of the character recognition results into a corresponding cell of the HTML table based on the matching relationship; Adding attribute tags to the merged cells in the HTML table, wherein the attribute tags include a row span tag rowspan and a column span tag colspan; Output the table html string corresponding to the HTML table.

5. The method for constructing a multimodal intelligent question-answering system based on a large model according to claim 1, characterized in that: The document slice includes a first text slice and / or the image area picture, and the image area picture includes formula content; After performing region recognition on the PDF document to obtain a table region image, a text region image and / or an image region image, the step further includes: Performing region recognition on the text region image to obtain a title text region and a paragraph text region; Using an OCR engine to perform text recognition on the title text area and the paragraph text area to obtain the title text and the paragraph text; Establishing a first mapping relationship between the title text and the paragraph text; Based on the first mapping relationship, the first text slice is generated.

6. The method for constructing a multimodal intelligent question-answering system based on a large model according to claim 1, characterized in that: The document slices include chapter content slices, the chapter information includes chapter titles, and the paragraph content includes chapter paragraph content; the step of parsing the Word document to obtain the paragraph content of the Word document includes: Parsing the Word document to determine whether the Word document includes a directory structure; If the Word document includes the directory structure, extract all the chapter titles in the Word document; For each of the chapter titles, identify and extract the corresponding chapter paragraph content; Establishing a second mapping relationship between the chapter title and the corresponding chapter paragraph content; Based on the second mapping relationship, the chapter content slice is generated; wherein, if the number of words of the first text in the chapter paragraph content exceeds a preset threshold, the chapter content slice includes a summary and / or FAQ question-and-answer pair corresponding to the chapter paragraph content, and the summary and the FAQ question-and-answer pair are generated based on the chapter paragraph content by calling a large language model.

7. The method for constructing a multimodal intelligent question-answering system based on a large model according to claim 6, characterized in that: The document slice further includes a second text slice, and the paragraph content includes segmented paragraph content; after parsing the Word document and determining whether the Word document includes a directory structure, the step further includes: If the Word document does not include the directory structure, segmenting the Word document according to semantics to obtain a plurality of segmented paragraph contents; Based on the segmented paragraph content, the second text slice is generated; wherein, if the number of words in the first text in the segmented paragraph content exceeds the preset threshold, the second text slice includes a summary and / or FAQ question-and-answer pair corresponding to the segmented paragraph content.

8. The method for constructing a multimodal intelligent question-answering system based on a large model according to any one of claims 1 to 7, characterized in that: The steps to build a knowledge base include: Mapping each of the document slices into an embedding vector respectively to obtain a plurality of first embedding vectors; An index is constructed for each of the first embedding vectors, and all the first embedding vectors are saved to a preset database based on the index to form the knowledge base.

9. A multimodal intelligent question-answering system based on a large model, characterized in that: The multimodal intelligent question-answering system includes a knowledge base, and the knowledge base is constructed based on the method for constructing a multimodal intelligent question-answering system based on a large model according to any one of claims 1 to 8; The system further comprises: An input module, used for receiving a query request input by a user; A vector module, used for mapping the query request into a second embedding vector; A retrieval module, configured to search in a preset database based on the second embedding vector to calculate the relevance between the second embedding vector and the first embedding vector corresponding to each document slice in the knowledge base; sort the document slices according to the magnitude of the relevance, and select the first k document slices as relevant document slices, where k is equal to a preset value; A calling module is used to generate an input prompt that conforms to a prompt engineering template based on the relevant document slices, and to call a large language model and input the relevant document slices and the input prompt to the large language model, so that the large language model can infer and generate an answer based on the relevant document slices; The output module is used to display the answer to the user.

10. A device for constructing a multimodal intelligent question-answering system based on a large model, characterized in that: The device comprises: An acquisition module, used for acquiring a knowledge file, wherein the knowledge file includes at least a PDF document and a Word document, wherein the PDF document and the Word document contain table content, text content and / or image content; A region recognition module, used to perform region recognition on the PDF document to obtain a table region image, a text region image and / or an image region image; A parsing module, used for parsing the Word document to obtain paragraph content of the Word document, wherein the paragraph content is composed of a first text, table data and / or a Word picture; A construction module is used to construct a knowledge base so that the multimodal intelligent question and answer system provides intelligent question and answer services based on the knowledge base; wherein the knowledge base includes vector representations of document slices; the document slices are determined based on the table area image, the text area image and / or the image area image, or the document slices are determined based on the paragraph content and its corresponding chapter information.

Images