Knowledge question and answer control method and system based on picture data and electronic equipment

Abstract

The invention provides a knowledge question and answer control method and system based on picture data and electronic device.The method comprises the steps that firstly, a to-be-processed document containing the picture data is obtained, after a semantic result of the picture data is determined according to the document hierarchy of the to-be-processed document, a description text corresponding to the picture data is generated based on the semantic result; determining a markup language corresponding to the picture data according to the description text; then, constructing a vector database corresponding to the to-be-processed document based on the markup language, and obtaining a corresponding query text block from the vector database based on the knowledge question and answer questioning instruction; and finally, acquiring context data corresponding to the query text block according to a preset context assembly strategy, and generating a knowledge question-answer answer result corresponding to the knowledge question-answer question instruction by using a markup language contained in the context data. According to the method, picture data can be fully utilized in a knowledge question and answer related system, answers with pictures and texts are provided, and the user experience and the answer quality are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a knowledge question and answer control method, system and electronic equipment based on image data. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLM) have achieved remarkable results in the field of natural language processing. In knowledge question-answering related systems, LLM combined with retrieval-augmented generation (RAG) technology can provide efficient and intelligent question-answering services. Traditional knowledge question-answering systems mainly process text information, but have limited processing capabilities for documents containing image data such as pictures and charts. However, in many fields, such as education, medical care, industrial design and other usage scenarios, processing image data in documents is crucial for understanding and answering questions. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a knowledge question and answer control method, system and electronic device based on image data, which can make full use of image data in knowledge question and answer related systems, can dynamically quote images in original documents during the knowledge answering process, and provide answers with both pictures and texts, thereby improving user experience and answer quality.

[0004] In a first aspect, an embodiment of the present invention provides a knowledge question answering control method based on image data, the method comprising: Preprocessing step: obtaining a document to be processed containing image data, determining a semantic result of the image data according to the document level of the document to be processed, generating a description text corresponding to the image data based on the semantic result, and determining a markup language corresponding to the image data according to the description text; Vectorized interaction step: construct a vector database corresponding to the document to be processed based on the markup language, and obtain the corresponding query text block from the vector database based on the knowledge question-answering instruction; Answer generation step: Obtain context data corresponding to the query text block according to a preset context assembly strategy, and use the markup language contained in the context data to generate a knowledge question and answer answer result corresponding to the knowledge question and answer question instruction.

[0005] Optional pre-processing steps include: Document structure analysis step: obtaining the document level corresponding to the document to be processed, and determining the context information of the image data based on the document level; Image semantic understanding step: determine the semantic result corresponding to the image data according to the context information of the image data, and use the semantic result to determine the visual analysis result corresponding to the image data; Image description generation step: obtain the description text corresponding to the image data based on the visual analysis results and context information; Image link conversion steps: Use the tag format parameters corresponding to the markup language to convert the image data and description text into the markup language.

[0006] Optional document structure analysis steps include: After parsing the document structure of the document to be processed, obtain the document level; Get the chapter titles and paragraphs of the document to be processed based on the document level; Get the image data contained in the paragraph and extract the description text corresponding to the image data; Use chapter titles and description text to determine the contextual information of image data.

[0007] Optional, image semantic understanding steps include: Acquire picture type data and picture function data of the picture data based on the context information, and determine first description data corresponding to the picture data using the picture type data and the picture function data; Acquire product name data, operation step data, structure description data, annotation prompt data and warning information data of the image data based on the context information, and determine second description data corresponding to the image data using the product name data, operation step data, structure description data, annotation prompt data and warning information data; A semantic result is determined using the first description data and the second description data, and after the semantic result is input into a trained multimodal model, a visual analysis result of an output of the multimodal model is obtained.

[0008] Optionally, the image description generation step includes: Acquire the position structure information of the image data in the document to be processed based on the context information; Obtain identification annotation information of the image data, and generate description information of the image data using the identification annotation information, position structure information, and visual analysis results; After the description information is input into the multimodal model, the description text of the output of the multimodal model is obtained.

[0009] Optional, image link conversion steps include: Determine the markup format parameters corresponding to the markup language; Convert the image data into a format link based on the tag format parameters, and convert the description text into a format text based on the tag format parameters; Generate the markup language corresponding to the image data using format links and format text.

[0010] Optional, vectorized interaction steps include: After the markup language is updated to the document to be processed, the document to be processed is divided into a plurality of text blocks based on the semantic result of the markup language; A vector database is constructed using the text block, and a vector index of the text block is generated based on the vector database; Determine the query vector corresponding to the knowledge question and answer instruction, and obtain the vector index corresponding to the query vector; The query text block is determined according to the text block corresponding to the vector index.

[0011] Optional response generation steps include: Determine the prompt data corresponding to the query text block, and assemble the prompt data and the query text block using a context assembly strategy to obtain context data; After inputting the context data into the preset language model, the output result of the language model is obtained; The markup language included in the parsed and obtained output result is used to generate the knowledge question and answer result by using the markup language.

[0012] In a second aspect, the present invention provides a knowledge question answering control system based on image data, the system comprising: Preprocessing module: used to obtain a document to be processed containing image data, determine the semantic result of the image data according to the document level of the document to be processed, generate a description text corresponding to the image data based on the semantic result, and determine the markup language corresponding to the image data according to the description text; Vectorized interaction module: used to build a vector database corresponding to the document to be processed based on the markup language, and obtain the corresponding query text block from the vector database based on the knowledge question-answering instruction; Answer generation module: used to obtain context data corresponding to the query text block according to the preset context assembly strategy, and use the markup language contained in the context data to generate the knowledge question and answer answer result corresponding to the knowledge question and answer question instruction.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the steps of the knowledge question and answer control method based on image data provided in the first aspect.

[0014] In a fourth aspect, an embodiment of the present invention further provides a storage medium storing computer executable instructions. When the computer executable instructions are called and executed by a processor, the computer executable instructions prompt the processor to implement the steps of the knowledge question and answer control method based on image data provided in the first aspect.

[0015] The embodiment of the present invention provides a method, system and electronic device for controlling knowledge questions and answers based on image data. In the process of controlling knowledge questions and answers, the method first obtains a document to be processed containing image data, determines the semantic result of the image data according to the document level of the document to be processed, generates a description text corresponding to the image data based on the semantic result, and determines the markup language corresponding to the image data according to the description text; then, a vector database corresponding to the document to be processed is constructed based on the markup language, and the corresponding query text block is obtained from the vector database based on the knowledge question and answer instruction; finally, the context data corresponding to the query text block is obtained according to the preset context assembly strategy, and the markup language contained in the context data is used to generate the knowledge question and answer answer result corresponding to the knowledge question and answer instruction. The method can make full use of the image data in the knowledge question and answer related system, can dynamically quote the image in the original document during the knowledge answering process, and provide an answer with pictures and texts, thereby improving the user experience and the answer quality.

[0016] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of a knowledge question and answer control method based on image data provided by an embodiment of the present invention; Figure 2 A flowchart of step S101 in a knowledge question and answer control method based on image data provided by an embodiment of the present invention; Figure 3A flowchart of step S201 in a knowledge question and answer control method based on image data provided by an embodiment of the present invention; Figure 4 A flowchart of step S202 in a knowledge question and answer control method based on image data provided by an embodiment of the present invention; Figure 5 A flowchart of step S203 in a knowledge question and answer control method based on image data provided by an embodiment of the present invention; Figure 6 A flowchart of step S204 in a knowledge question and answer control method based on image data provided by an embodiment of the present invention; Figure 7 A flowchart of step S102 in another knowledge question and answer control method based on picture data provided by an embodiment of the present invention; Figure 8 A flowchart of step S103 in another knowledge question and answer control method based on picture data provided by an embodiment of the present invention; Fig. 9 A flowchart of another knowledge question and answer control method based on picture data provided by an embodiment of the present invention; Fig.10 A schematic diagram of a knowledge question answering control system based on image data provided by an embodiment of the present invention; Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0020] icon: 1010-preprocessing module; 1020-vectorization interaction module; 1030-response generation module; 101 - processor; 102 - memory; 103 - bus; 104 - communication interface. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] With the rapid development of artificial intelligence technology, the LLM (Large Language Model) has achieved remarkable results in the field of natural language processing. In knowledge question-answering systems, LLM combined with retrieval enhancement generation (RAG) technology can provide efficient and intelligent question-answering services. In many fields, such as education, medical care, industrial design and other usage scenarios, it is necessary to process image data in documents, but traditional knowledge question-answering systems mainly process text information and have limited processing capabilities for documents containing image data such as pictures and charts.

[0023] Specifically, most knowledge question-answering systems in the prior art are mainly based on text retrieval and matching, and are unable to effectively process image information in documents; and although some systems can recognize and describe images, they are unable to closely integrate image information with text content and context, and are even unable to dynamically reference relevant images during the knowledge question-answering process.

[0024] It can be seen that the knowledge question and answer control process in the prior art cannot make full use of image information: there is a lack of effective processing of images in documents, and images cannot be included in the answers, resulting in limited completeness and accuracy of the answers. In addition, the contextual relevance of image data is poor. Even if the image can be described, it cannot be closely integrated with the text context, which affects the user's understanding of the answer. In actual scenarios, the results obtained by existing knowledge questions also lack the form of answers with both pictures and texts, and users are inefficient in understanding complex questions.

[0025] Based on this, the present invention provides a knowledge question and answer control method, system and electronic device based on image data. The method can make full use of image data in knowledge question and answer related systems, can dynamically reference images in original documents during the knowledge answering process, and provide answers with both pictures and texts, thereby improving user experience and answer quality.

[0026] To facilitate understanding of this embodiment, firstly, a knowledge question answering control method based on image data disclosed in an embodiment of the present invention is described in detail. The method is applied to a navigation system including a map interface. Figure 1 As shown, including: Preprocessing step S101: obtaining a document to be processed containing image data, determining a semantic result of the image data according to the document level of the document to be processed, generating a description text corresponding to the image data based on the semantic result, and determining a markup language corresponding to the image data according to the description text; Vectorized interaction step S102: constructing a vector database corresponding to the document to be processed based on the markup language, and obtaining a corresponding query text block from the vector database based on the knowledge question-answering instruction; Answer generation step S103: obtaining context data corresponding to the query text block according to a preset context assembly strategy, and generating a knowledge question and answer answer result corresponding to the knowledge question and answer question instruction using the markup language contained in the context data.

[0027] The method first performs a preprocessing step, parses and determines the document hierarchical relationship after obtaining the document to be processed, and then determines the semantic result of the image data, and uses the semantic result to obtain the description text of the image data. When processing the description text, the image data can be embedded in the data form of a markup language such as Markdown. Using Markdown syntax makes it more convenient to reference and render images, reduces the complexity of subsequent processing, and ensures that the images are displayed correctly in the answer. At the same time, the alternative text in the Markdown image reference can contain the image description, which facilitates the understanding of the image data content.

[0028] By building a vector database corresponding to the document to be processed and updating the data in the vector database using markup language, it can be used as the data source for the retrieval enhancement generation process. The vectorization interaction step realizes the document vectorization and query process, and uses the knowledge question and answer question instruction to request the vector database, thereby obtaining the query text block corresponding to the knowledge question and answer question instruction.

[0029] The answer generation step can be achieved through the answer of the relevant language model. The query text block is assembled through the context assembly strategy corresponding to the language model to obtain context data. The language model generates relevant answer results containing image data based on the context data. The answer results are parsed according to the markup language to obtain Markdown data, and the Markdown data is used to generate the knowledge question and answer answer results for output so that they correspond to the knowledge question and answer question instructions.

[0030] Optionally, the preprocessing step S101 is as follows: Figure 2 As shown, including: Document structure analysis step S201: obtaining the document level corresponding to the document to be processed, and determining the context information of the image data based on the document level; Image semantic understanding step S202: determining a semantic result corresponding to the image data according to context information of the image data, and determining a visual analysis result corresponding to the image data using the semantic result; Image description generation step S203: obtaining description text corresponding to the image data based on the visual analysis result and context information; Picture link conversion step S204: converting the picture data and description text into the markup language using the markup format parameters corresponding to the markup language.

[0031] The preprocessing step requires the execution of document structure analysis, image semantic understanding, image description generation, and image link conversion processes, and finally the image data is converted into Markdown markup language. Specifically, the document level corresponding to the document to be processed is obtained, and the context information of the image data is determined based on the document level; then the semantic result of the image data is determined using the context information, and the visual analysis result of the image data is obtained based on the semantic result. After the visual analysis result is obtained, the description text of the image data is determined in combination with the context information, and then the Markdown tag format parameters corresponding to the markup language are used to convert the image data and the description text, thereby obtaining the Markdown markup language.

[0032] Optionally, the document structure analysis step S201, such as Figure 3 As shown, including: Step S301, obtaining the document level after parsing the document structure of the document to be processed; Step S302, obtaining chapter titles and paragraphs of the document to be processed based on the document level; Step S303, obtaining the image data contained in the paragraph, and extracting the description text corresponding to the image data; Step S304: determine the context information of the picture data using the chapter title and description text.

[0033] During the document structure analysis process, the document hierarchical structure is first parsed to obtain the location structure information of the image data in the document, such as directory location, chapter relationship, etc. Then the contextual content around the image data is extracted, and the existing image title, annotation and other information are obtained as descriptive text, and then the contextual information of the image data is obtained in combination with the chapter title.

[0034] In the specific implementation process, the document structure can be parsed using Python's python-docx library to read and parse the docx document to be processed. The structure in the document is usually organized in the form of paragraphs and titles; then locate the chapter title and identify the chapter title (such as using different levels of title styles), for example: Heading 1 represents the main title, Heading 2 represents the secondary title; then extract the relevant information of the image data, which can be achieved by finding the image object directly inserted in the paragraph. Images are usually part of the paragraph in the docx document, rather than references in Markdown format. Then extract the context information. When the image is identified, go back to the previous paragraph to obtain the context information and record the descriptive text after the image. Then combine the context information, and combine the extracted chapter title, image title and its related description into complete context information.

[0035] Optionally, the image semantic understanding step S202, such as Figure 4 As shown, including: Step S401, acquiring picture type data and picture function data of the picture data based on the context information, and determining first description data corresponding to the picture data using the picture type data and the picture function data; Step S402, obtaining product name data, operation step data, structure description data, annotation prompt data and warning information data of the image data based on the context information, and determining second description data corresponding to the image data using the product name data, operation step data, structure description data, annotation prompt data and warning information data; Step S403, using the first description data and the second description data to determine a semantic result, and after inputting the semantic result into the trained multimodal model, obtaining a visual analysis result of the output of the multimodal model.

[0036] The image semantic understanding process can use a multimodal large model (such as GPT-4 Vision) to analyze the image content and generate visual analysis results. The specific execution process involves the basic information and core content description in the prompt word construction process. When constructing prompt words, it is necessary to consider its basic information and core content. The basic information is used as the first description data. The image type data (such as operation step diagrams, product structure diagrams, interface diagrams, etc.) and image function data (the functional positioning of the image in the document) of the image data are mainly obtained based on context information, so as to use the image type data and image function data to determine the first description data corresponding to the image data.

[0037] The core content description is used as the second description data. It mainly obtains the product / component name data, operation step data, structure description data, annotation prompt data and warning information data of the image data based on the context information. The second description data of the core content can be obtained by using the above data. The second description data needs to use professional and accurate terms in the description process, and maintain the logic and sequence of the description. It needs to highlight the key points of the operation or description, and it also needs to retain the numbers and logos in the original picture.

[0038] The semantic result is then determined using the first description data and the second description data, and is input into the multimodal large model to obtain a specific visual analysis result. An example of a visual analysis result is as follows: - Basic information; - Type: Product structure breakdown diagram; -Function: Display the overall structure and component composition of the XW-200 smart bracelet.

[0039] On this basis, the picture description generation step S203 is as follows: Figure 5 As shown, including: Step S501, obtaining position structure information of the image data in the document to be processed based on the context information; Step S502, obtaining identification annotation information of the image data, and generating description information of the image data using the identification annotation information, position structure information and visual analysis results; Step S503, after the description information is input into the multimodal model, the description text of the output of the multimodal model is obtained.

[0040] In the process of describing an image, it is necessary to integrate various information to obtain the corresponding description information, including the location structure information of the image in the document, the identification annotation information of the original image, the visual analysis result information, and other information, so as to generate the description information of the image data. Then, the description information is input into the multimodal model for further analysis and extraction to generate a concise and complete description text. An example of a description text is as follows: "XW-200 bracelet structure diagram | core components diagram | mainboard display battery sensor | FPC connection relationship | maintenance and installation reference".

[0041] Optionally, the picture link conversion step S204, such as Figure 6 As shown, including: Step S601, determining the markup format parameters corresponding to the markup language; Step S602, converting the image data into a format link based on the markup format parameter, and converting the description text into a format text based on the markup format parameter; Step S603: Generate a markup language corresponding to the image data using the format link and the format text.

[0042] During the image link conversion process, the image data is converted into a Markdown format link based on the markup format parameters, and the description text is converted into formatted text based on the markup format parameters. By replacing the original image corresponding to the image data with a Markdown format link, the description text is used as the replacement text for the markdown link, and a unique link is generated for the image. An example of the markup language is as follows: "".

[0043] Optionally, vectorize the interaction step S102, such as Figure 7 As shown, including: Step S701, after updating the markup language to the document to be processed, the document to be processed is divided into a plurality of text blocks based on the semantic result of the markup language; Step S702, constructing a vector database using the text block, and generating a vector index of the text block based on the vector database; Step S703, determining the query vector corresponding to the knowledge question and answer instruction, and obtaining the vector index corresponding to the query vector; Step S704: determining the query text block according to the text block corresponding to the vector index.

[0044] After the Markdown markup language is updated to the document to be processed during the vectorized interaction process, the document to be processed is divided into multiple text blocks through semantic segmentation based on the semantic results of the markup language; through semantic segmentation, the integrity of the Markdown image link in the text block is maintained. Then, the vector database is built using the text block, the text block is converted into a vector representation using the embedding model, and the vector index of the text block is generated based on the vector database. After receiving the knowledge question and answer instruction issued by the user, the query process is converted into a vector representation by obtaining the query vector corresponding to the knowledge question and answer instruction, and the relevant text block is retrieved and obtained in the vector index.

[0045] Optionally, the response generation step S103 is as follows: Figure 8 As shown, including: Step S801, determining the prompt data corresponding to the query text block, and assembling the prompt data and the query text block using a context assembly strategy to obtain context data; Step S802, after inputting the context data into a preset language model, obtaining the output result of the language model; Step S803, parsing and obtaining the markup language contained in the output result, and generating the knowledge question and answer result by using the markup language.

[0046] The answer generation process can be realized through the answer of the relevant language model. After determining the prompt data corresponding to the query text block, the prompt data and the query text block are assembled using the context assembly strategy to obtain the context data. Then, after the context data is input into the preset language model, the Markdown content output by the language model is obtained, and the Markdown content is used to generate the knowledge question answer result. For example: "Based on your question about the internal structure of the product, you can refer to the following diagram to understand its components and connections:  As shown in the figure, the internal structure of the product consists of multiple main components, which cooperate with each other through specific connections to achieve the functions of the product.

[0047] like Fig. 9Another flow chart of a knowledge question-answering control method based on image data is shown in the figure. This method generates accurate image descriptions in the preprocessing stage by collecting information about the position structure of the image in the document, the contextual content around the image, the identification and annotations of the original image, and visual analysis results (using a large multimodal visual model such as GPT-4o), and using the large model to summarize and generate image descriptions. Through the introduction of the multimodal model, the system can accurately obtain the content description of the image and associate the context in which the image is located, laying the foundation for accurate reference to the image in the subsequent answer.

[0048] In addition, this method uses image links in Markdown syntax in input construction. When preprocessing text, images are embedded in the form of Markdown image links, which makes it easier to retain image references when LLM generates answers. Using Markdown syntax makes it easier to reference and render images, reduces the complexity of subsequent processing, and ensures that images are displayed correctly in answers. At the same time, the alternative text in the markdown image reference can contain image descriptions, which is easier for large models to understand.

[0049] This method optimizes the prompt engineering and guides LLM to cite images in the answer. By designing effective prompt words, LLM is guided to appropriately include image references when generating answers and retain Markdown image links. Through carefully designed prompt words, it is ensured that LLM will not ignore or modify image links when generating answers, so that the answers contain both text explanations and relevant images.

[0050] The knowledge question answering control method based on picture data in the above embodiment has the following advantages: Completeness: The answer can dynamically quote images from the original document, providing an illustrated answer format to enhance the completeness and intuitiveness of the answer.

[0051] Relevance: Tie the image closely to its context to help users better understand the answer.

[0052] Ease of use: Using Markdown syntax simplifies the image reference and rendering process, reducing the complexity of system implementation and maintenance.

[0053] Intelligence: Through the introduction of multimodal models, the system has the ability to comprehensively understand images and texts, which improves the accuracy of answers.

[0054] From the knowledge question and answer control method based on image data mentioned in the above embodiment, it can be seen that this method additionally sets up a drawing thread on the basis of the main thread of the navigation system, and uses the drawing thread to generate canvas trajectory data to reduce the resource consumption of the main thread; at the same time, this method combines the visible area to streamline the canvas trajectory data, further reducing the data processing volume, thereby improving the page fluency of the navigation system.

[0055] Corresponding to the knowledge question answering control method based on image data provided in the above embodiment, the embodiment of the present invention provides a knowledge question answering control system based on image data, such as Fig.10 As shown, the system includes: Preprocessing module 1010: used to obtain a document to be processed containing image data, determine the semantic result of the image data according to the document level of the document to be processed, generate a description text corresponding to the image data based on the semantic result, and determine the markup language corresponding to the image data according to the description text; Vectorized interaction module 1020: used to construct a vector database corresponding to the document to be processed based on the markup language, and obtain the corresponding query text block from the vector database based on the knowledge question and answer instruction; Answer generation module 1030: used to obtain context data corresponding to the query text block according to a preset context assembly strategy, and use the markup language contained in the context data to generate a knowledge question and answer answer result corresponding to the knowledge question and answer question instruction.

[0056] From the knowledge question and answer control system based on image data mentioned in the above embodiments, it can be seen that the system can make full use of image data in knowledge question and answer related systems, can dynamically quote images in original documents during the knowledge answering process, and provide answers with both pictures and texts, thereby improving user experience and answer quality.

[0057] The knowledge question and answer control system based on image data provided in the embodiment of the present invention has the same implementation principle and technical effects as the aforementioned knowledge question and answer control method embodiment based on image data. For the sake of brief description, for matters not mentioned in the system embodiment, reference may be made to the corresponding contents in the aforementioned knowledge question and answer control method embodiment based on image data.

[0058] This embodiment also provides an electronic device. The structural diagram of the electronic device is as follows: Fig.11 As shown, the device includes a processor 101 and a memory 102; wherein the memory 102 is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor to implement the steps of the above-mentioned knowledge question and answer control method based on image data.

[0059] Fig.11 The electronic device shown further includes a bus 103 and a communication interface 104 , and the processor 101 , the communication interface 104 and the memory 102 are connected via the bus 103 .

[0060] The memory 102 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The bus 103 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0061] The communication interface 104 is used to connect to at least one user terminal and other network units through a network interface, and send the encapsulated IPv4 message or IPv4 message to the user terminal through the network interface.

[0062] The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 101. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 102, and the processor 101 reads the information in the memory 102 and completes the steps of the method of the above embodiment in combination with its hardware.

[0063] An embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the knowledge question and answer control method based on image data in the aforementioned embodiment are executed.

[0064] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, equipment and methods can be implemented in other ways. The system embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0065] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0066] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0067] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that can be executed by a processor. Based on this understanding, the technical solution of the present invention can essentially or in other words, the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0068] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A knowledge question answering control method based on image data, characterized in that: The method comprises: Preprocessing step: obtaining a document to be processed containing image data, determining a semantic result of the image data according to the document level of the document to be processed, generating a description text corresponding to the image data based on the semantic result, and determining a markup language corresponding to the image data according to the description text; Vectorized interaction step: constructing a vector database corresponding to the document to be processed based on the markup language, and obtaining a corresponding query text block from the vector database based on the knowledge question-answering instruction; Answer generation step: obtaining context data corresponding to the query text block according to a preset context assembly strategy, and using the markup language contained in the context data to generate a knowledge question and answer answer result corresponding to the knowledge question and answer instruction.

2. The knowledge question-answering control method based on image data according to claim 1 is characterized in that: The pre-processing step comprises: Document structure analysis step: obtaining the document level corresponding to the document to be processed, and determining the context information of the image data based on the document level; Image semantic understanding step: determining the semantic result corresponding to the image data according to the context information of the image data, and determining the visual analysis result corresponding to the image data using the semantic result; Picture description generating step: obtaining the description text corresponding to the picture data based on the visual analysis result and the context information; Picture link conversion step: using the markup format parameters corresponding to the markup language, converting the picture data and the description text into the markup language.

3. The knowledge question-answering control method based on image data according to claim 2 is characterized in that: The document structure analysis step comprises: After parsing the document structure of the document to be processed, obtain the document level; Acquire the chapter titles and paragraphs of the document to be processed based on the document level; Acquire the image data contained in the paragraph, and extract the description text corresponding to the image data; The context information of the picture data is determined using the chapter title and the description text.

4. The knowledge question-answering control method based on image data according to claim 3 is characterized in that: The picture semantic understanding step includes: Acquire picture type data and picture function data of the picture data based on the context information, and determine first description data corresponding to the picture data by using the picture type data and the picture function data; Acquire product name data, operation step data, structure description data, annotation prompt data, and warning information data of the image data based on the context information, and determine second description data corresponding to the image data using the product name data, the operation step data, the structure description data, the annotation prompt data, and the warning information data; The semantic result is determined using the first description data and the second description data, and after the semantic result is input into a trained multimodal model, the visual analysis result of the output of the multimodal model is obtained.

5. The knowledge question-answering control method based on image data according to claim 4 is characterized in that: The picture description generating step comprises: Acquire the position structure information of the image data in the document to be processed based on the context information; Acquire identification annotation information of the image data, and generate description information of the image data using the identification annotation information, the position structure information, and the visual analysis result; After the description information is input into the multimodal model, the description text output by the multimodal model is obtained.

6. The knowledge question-answering control method based on image data according to claim 2, characterized in that: The picture link conversion step includes: Determine the markup format parameters corresponding to the markup language; Converting the image data into a format link based on the markup format parameters, and converting the description text into a format text based on the markup format parameters; The markup language corresponding to the image data is generated using the format link and the format text.

7. The knowledge question-answering control method based on image data according to claim 1, characterized in that: The vectorized interaction step includes: After the markup language is updated to the document to be processed, the document to be processed is divided into a plurality of text blocks based on the semantic result of the markup language; constructing the vector database using the text block, and generating a vector index of the text block based on the vector database; Determine a query vector corresponding to the knowledge question and answer instruction, and obtain the vector index corresponding to the query vector; The query text block is determined according to the text block corresponding to the vector index.

8. The knowledge question-answering control method based on image data according to claim 1, characterized in that: The response generation step comprises: Determine prompt data corresponding to the query text block, and assemble the prompt data and the query text block using the context assembly strategy to obtain the context data; After inputting the context data into a preset language model, obtaining an output result of the language model; The markup language included in the output result is parsed and obtained, and the knowledge question and answer result is generated using the markup language.

9. A knowledge question-answering control system based on image data, characterized in that: The system comprises: Preprocessing module: used for obtaining a document to be processed containing image data, determining the semantic result of the image data according to the document level of the document to be processed, generating a description text corresponding to the image data based on the semantic result, and determining the markup language corresponding to the image data according to the description text; Vectorization interaction module: used to construct a vector database corresponding to the document to be processed based on the markup language, and obtain a corresponding query text block from the vector database based on the knowledge question and answer instruction; Answer generation module: used to obtain the context data corresponding to the query text block according to a preset context assembly strategy, and use the markup language contained in the context data to generate a knowledge question and answer answer result corresponding to the knowledge question and answer instruction.

10. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the steps of the knowledge question and answer control method based on image data as described in any one of claims 1 to 8.

Images