Question processing method and device and electronic equipment

Abstract

The invention discloses a problem processing method and device and electronic equipment. The method relates to the field of artificial intelligence, and comprises the following steps: under the condition that a target question of a user is received, determining at least one target document block required for answering the target question from a plurality of document blocks contained in a pre-constructed database; generating a prompt statement based on the at least one target document block and the target question, the prompt statement being used for guiding the target large model to determine a text answer corresponding to the target question according to the at least one target document block, under the condition that image link information associated with the text answer exists in at least one target document block, the target large model is guided to output the text answer and an image corresponding to the image link information; and answering the target question according to the prompt statement through the target large model. The technical problem of low answering accuracy due to the fact that a large model cannot answer user questions in combination with pictures in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a problem processing method, device and electronic equipment. Background Art

[0002] At present, large models are widely used in various scenarios. However, they face challenges in certain specific tasks or knowledge-intensive tasks. For example, when processing information beyond their training data or specific domain-specific information, "hallucination" may occur, that is, generating factually incorrect content. To overcome these challenges, RAG (Retrieval-augmented Generation) technology was proposed and applied to large models.

[0003] RAG technology enhances the function of large models by retrieving relevant documents from external knowledge bases and calculating semantic similarity. The documents injected by users into the knowledge base are usually diversified documents containing text and pictures. However, the existing RAG usually ignores the information of pictures when processing the content of documents with pictures, which makes the large model have the problem of low answer accuracy when answering questions.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a method, device and electronic device for handling questions, so as to at least solve the technical problem in the related art that a large model cannot answer user questions in combination with pictures and has low answer accuracy.

[0006] According to one aspect of an embodiment of the present invention, a question processing method is provided, comprising: upon receiving a target question from a user, determining at least one target document block required to answer the target question from a plurality of document blocks contained in a pre-constructed database, wherein the document contents of at least some of the plurality of document blocks include image link information and text content associated with the image link information; generating a prompt statement based on at least one target document block and a target question, wherein the prompt statement is used to guide a target macro model to determine a text answer corresponding to the target question according to at least one target document block, and to guide the target macro model to output a text answer and an image corresponding to the image link information when image link information associated with the text answer exists in at least one target document block; and answering the target question according to the prompt statement through the target macro model.

[0007] Furthermore, the multiple document blocks include a first type of document block and a second type of document block, the document content of the first type of document block includes image link information and text content associated with the image link information, and the second type of document block does not include image link information. The problem handling method also includes: for each initial document among the multiple initial documents, when the document content of the initial document includes an image, generating image link information corresponding to the image; replacing the image in the document content according to the image link information to obtain the target document content; and generating at least the first type of document block according to the target document content.

[0008] Furthermore, the problem handling method also includes: obtaining image information of the image in the initial document, wherein the image information includes at least one of the following: text information in the image, position information of the image in the initial document, and image size of the image; identifying interference images from the images in the initial document based on the image information; and generating image link information for images in the initial document other than the interference images.

[0009] Furthermore, the problem handling method also includes: for each image in the initial document, counting the total number of times the image containing the text information in the image appears in the initial document, and counting the total number of pages of the initial document; determining whether the image is an interference image based on the numerical relationship between the total number of times corresponding to the image and the total number of pages.

[0010] Furthermore, the problem handling method also includes: for each image in the initial document, when the image size of the image is less than or equal to a first preset value, determining that the image is an interference image; when the image size of the image is greater than or equal to a second preset value, determining that the image is not an interference image, wherein the second preset value is greater than the first preset value; when the image size of the image is greater than the first preset value and less than the second preset value, determining whether the image is an interference image based on the similarity between the image and a preset sample interference image.

[0011] Furthermore, the problem handling method also includes: before generating at least the first type of document block based on the target document content, for each target document content including image link information, determining whether the target document content contains text data; if the target document content does not contain text data, performing content analysis on the image corresponding to the image link information in the target document content to obtain image introduction information; adding the image introduction information as text content associated with the image link information to the target document content to obtain updated target document content.

[0012] Furthermore, the problem handling method also includes: performing data type conversion on the document content in the initial document to obtain a dictionary corresponding to the initial document; and when the dictionary includes image data, generating image link information of an image corresponding to the image data based on the image data.

[0013] Furthermore, the problem handling method also includes: decoding the image data to obtain an image corresponding to the image data; storing the image corresponding to the image data in a target storage area, and generating image link information based on the storage position of the image in the target storage area.

[0014] Furthermore, the problem handling method also includes: generating a number of the image corresponding to the image link information based on the position information of the image corresponding to the image link information in the initial document; combining the image link information and the number of the image corresponding to the image link information to obtain combined information; replacing the image in the document content based on the combined information to obtain the target document content, wherein the prompt statement is also used to guide the target large model to determine the processing order of the image link information based on the numbers corresponding to the image link information when there are multiple image link information associated with the text answer.

[0015] Furthermore, the problem handling method also includes: slicing the target document content to obtain multiple slices; extracting the document name and summary information of the initial document from the initial document to which the slices belong; splicing the slices, document name and summary information to obtain spliced ​​slices; and generating at least a first type of document block based on the spliced ​​slices.

[0016] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the above-mentioned problem handling method when running.

[0017] According to another aspect of an embodiment of the present invention, an electronic device is also provided, which includes one or more processors; a memory for storing one or more programs, so that when the one or more programs are executed by the one or more processors, the one or more processors are implemented to run the programs, wherein the programs are configured to execute the above-mentioned problem handling method when running.

[0018] In an embodiment of the present invention, a method of answering user questions based on a combination of images and texts is adopted. When receiving a user's target question, at least one target document block required to answer the target question is determined from a plurality of document blocks contained in a pre-constructed database, and then a prompt statement is generated based on the at least one target document block and the target question, so that the target question is answered according to the prompt statement by the target macro model. Among them, the document content of at least some of the document blocks in the plurality of document blocks includes image link information and text content associated with the image link information, and the prompt statement is used to guide the target macro model to determine the text answer corresponding to the target question according to at least one target document block, and guide the target macro model to output the text answer and the image corresponding to the image link information when there is image link information associated with the text answer in at least one target document block.

[0019] In the above process, by setting image link information in the document block, the image content of the original image in the document block is retained. By generating a prompt statement based on at least one target document block and the target question required to answer the user's target question, and setting the prompt statement to guide the target large model to output the text answer and the image corresponding to the image link information when there is image link information associated with the text answer in at least one target document block, when answering the question through the target large model, if there is image link information associated with the text answer in the relevant document block, the answer is given in combination with the picture corresponding to the text answer and its associated image link information, thereby achieving attention to the relationship between the image and the text in the document and retaining the integrity of the document content, improving the accuracy and richness of the information used to answer the user's questions, and avoiding the low quality of answers caused by missing image link information or improper processing of image content in related technologies.

[0020] It can be seen that the solution provided in this application achieves the purpose of answering user questions based on a combination of pictures and texts, thereby achieving the technical effect of improving the accuracy of answers, and further solving the technical problem that the large model in the related technology cannot answer user questions in combination with pictures, resulting in low answer accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0022] Figure 1 This is a flow chart of an optional problem solving method according to an embodiment of the present invention. Figure 1 ;

[0023] Figure 2 is a schematic diagram of document content of an optional initial document according to an embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of optional target document content according to an embodiment of the present invention;

[0025] Figure 4 is a schematic diagram of an image corresponding to an optional picture link according to an embodiment of the present invention;

[0026] Figure 5 is a schematic diagram of an optional method for processing target document content according to an embodiment of the present invention;

[0027] Figure 6 This is a flow chart of an optional problem solving method according to an embodiment of the present invention. Figure 2 ;

[0028] Figure 7 is a schematic diagram of an optional target large model answering question according to an embodiment of the present invention;

[0029] Figure 8 is a schematic diagram of an optional problem handling device according to an embodiment of the present invention;

[0030] Fig. 9 is a schematic diagram of an optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0034] Example 1

[0035] According to an embodiment of the present invention, an embodiment of a problem handling method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] RAG technology includes two steps: indexing and retrieval. In the indexing stage, documents are divided into blocks, encoded into vectors, and stored in the vector database. In the retrieval stage, the top k document blocks most relevant to the question are retrieved based on semantic similarity. These document blocks are used as additional input to the large model, helping the large model to better understand the question and generate more accurate answers.

[0037] With the current RAG technology, the knowledge base indexing stage cannot effectively handle scenarios in which a document contains both images and text, resulting in missing images and missing associations between images and text in the index slice content. Based on the problem handling method provided in this embodiment, the images and text in the knowledge are treated as an organic whole rather than separate individuals to implement the RAG process, so as to ensure the integrity and effectiveness of knowledge injection throughout the RAG process.

[0038] Figure 1 This is a flow chart of an optional problem solving method according to an embodiment of the present invention. Figure 1 ,like Figure 1 As shown, the method comprises the following steps:

[0039] Step S101, upon receiving a target question from a user, determining at least one target document block required to answer the target question from a plurality of document blocks contained in a pre-constructed database, wherein document contents of at least some of the plurality of document blocks include image link information and text content associated with the image link information.

[0040] Optionally, electronic devices, application systems, servers and other devices may be used as the execution subject of the present application. In this embodiment, the target processing system is used as the execution subject to execute the aforementioned problem handling method.

[0041] In some embodiments, the user's target question may vary depending on the actual application scenario and actual needs. For example, in an operation specification query scenario, the user's target question may be "how to perform business A"; in a character query scenario, the user's target question may be "please introduce person B"; in a business query scenario, the user's target question may be "how to handle business C". Therefore, in this embodiment, the user's target question is not specifically limited.

[0042] In some embodiments, the user's target question refers to a question used to query the internal knowledge of the organization in which the user is located. The internal knowledge of the organization includes but is not limited to relevant knowledge about the products within the organization, relevant knowledge about the processes within the organization, relevant knowledge about the operating specifications within the organization, etc.

[0043] Optionally, the multiple document blocks in the pre-built database can be generated by the target processing system based on multiple initial documents acquired in advance. The multiple initial documents can be acquired by the target processing system from a designated storage area, for example, from the Internet, an organization's intranet, etc. The multiple initial documents can also be pre-input by the user.

[0044] Optionally, the document content of at least some of the multiple document blocks includes image link information and text content associated with the image link information, the image link information is used to replace the original image in the document block, the text content associated with the image link information includes: text content obtained by semantically recognizing the image corresponding to the image link information, and / or context content of the image corresponding to the image link information, the context content of the image may include text content before the image position of the image in the original document and / or text content after the image position.

[0045] It can be understood that the document blocks that do not include image link information in the multiple document blocks do not include the corresponding original images in the original document, that is, the document blocks that do not include image link information in the multiple document blocks only include texts in the original document.

[0046] Optionally, the positional relationship between the image link information and the text in the document block is determined as a first positional relationship, and the positional relationship between the original image corresponding to the image link information and the text in the document block is determined as a second positional relationship. The first positional relationship is the same as the second positional relationship, thereby achieving the retention of the semantic relationship between the original image and the context words in the document block.

[0047] Optionally, the method provided in this embodiment can be applied to any scenario of document parsing with images, for example, a scenario where there is no text in the image, or a scenario where the image is closely related to the context words, to ensure the integrity and relevance of the knowledge injection. In addition, the image corresponding to the image link information varies according to the actual application scenario and actual needs, and the image corresponding to the image link information is not limited to any specific content or form of the image. For example, the image corresponding to the image link information can be a high-precision circuit diagram, landscape painting, portrait, pure text display or art picture, etc. For another example, the image corresponding to the image link information can be the operation picture description corresponding to each operation step text description in the firewall UI operation manual.

[0048] Optionally, when the target processing system receives a target question from a user, it can determine at least one target document block required to answer the target question from multiple document blocks.

[0049] For example, a user may specify a target document block from a plurality of document blocks, and the target processing system may determine at least one target document block according to the specified information input by the user.

[0050] For another example, the target processing system can calculate the similarity between the vector of the text (excluding image link information) in each document block among multiple document blocks and the vector of the target question, thereby determining at least one target document block from the multiple document blocks according to the magnitude of the similarity. For example, the first N document blocks with the highest similarity are determined as the target document blocks. In this way, it is avoided to calculate the similarity between the vector of the text and image in the document block and the vector of the target question, thereby, on the one hand, reducing the occupation of computing resources, and on the other hand, effectively improving the matching accuracy, that is, improving the accuracy of the determined target document block.

[0051] Step S102, generating a prompt statement based on at least one target document block and a target question, wherein the prompt statement is used to guide the target large model to determine a text answer corresponding to the target question according to at least one target document block, and to guide the target large model to output the text answer and an image corresponding to the image link information when there is image link information associated with the text answer in at least one target document block.

[0052] Optionally, in order to enable the large model to accurately capture and process image content when the target document block includes image link information, the target processing system can generate prompt statements based on at least one target document block and a target question, so that the large model can return answers and pictures related to the question as required, and realize visual display on a Web (World Wide Web) page.

[0053] In some embodiments, the target large model may be a large language model such as ChatGPT or ChatGLM, and the prompt statement is a prompt for the target large model.

[0054] In some embodiments, the target large model determines a text answer from at least one document block, and the image link information associated with the text answer may refer to image link information within a preset range of the text answer in the target document block where the text answer is located. For example, the preset range may be three consecutive paragraphs above the text answer. For another example, the preset range may be five consecutive paragraphs below the text answer. For another example, the preset range may be the previous paragraph of the text answer, etc.

[0055] For example, assuming that each of the 1st to 5th paragraphs in the target document block where the text answer is located has an image link information, and the text answer is the 6th paragraph in the target document block, if the preset range is the previous paragraph of the text answer, then the image link information of the 5th paragraph in the target document block where the text answer is located will be determined as the image link information associated with the text answer.

[0056] In some embodiments, the image link information associated with the text answer may also refer to all image link information in the target document block where the text answer is located.

[0057] Optionally, the target processing system may obtain a prompt template, thereby importing at least one target document block and a target question into the prompt template to obtain a prompt statement. For example, an optional prompt statement is as follows:

[0058] Please answer the question (i.e. the target question) based on the given information. If there is an HTTP (Hypertext Transfer Protocol) link or path (i.e. image link information) of a PNG image in the reference content (i.e. the target document block where the text answer is located), you need to find the image (i.e. the image corresponding to the image link information) from the HTTP link or path, write it in 3 / 8 / Markdown, without backslashes or code blocks, and then put the image and the answer to the question (i.e. the text answer) together as the answer.

[0059] Question: What is the operation process of Operation A in firewall operation?

[0060] material:

[0061] -{Text content of document A}

[0062] -{Text content of document B}

[0063] -{Text content of document C}

[0064] In the above prompt statement, "Question:" is used to fill in the target question, and "Material:" is used to fill in the document content of the target document block.

[0065] Step S103, answer the target question according to the prompt statement through the target large model.

[0066] Optionally, after the prompt statement is determined, the target processing system may input the prompt statement into the target macro model, so that the target macro model outputs target information for answering the target question according to the prompt statement.

[0067] Optionally, when there is image link information associated with a text answer in at least one target document block, the target information output by the target macro model includes the text answer and the image corresponding to the image link information matching the text answer, and the image corresponding to the image link information is determined by the target macro model based on the image link information; when there is no image link information matching the text answer in at least one target document block, the target information output by the target macro model includes the text answer.

[0068] Based on the scheme defined in the above steps S101 to S103, it can be known that in the embodiment of the present invention, a method based on a combination of images and texts is adopted to answer user questions. When receiving the user's target question, at least one target document block required to answer the target question is determined from multiple document blocks contained in a pre-built database, and then a prompt statement is generated based on at least one target document block and the target question, so that the target question is answered according to the prompt statement through the target macro model. Among them, the document content of at least part of the multiple document blocks includes image link information and text content associated with the image link information, and the prompt statement is used to guide the target macro model to determine the text answer corresponding to the target question according to at least one target document block, and guide the target macro model to output the text answer and the image corresponding to the image link information when there is image link information associated with the text answer in at least one target document block.

[0069] It is easy to notice that in the above process, by setting image link information in the document block, the image content of the original image in the document block is retained, and a prompt statement is generated based on at least one target document block and target question required to answer the user's target question, and the prompt statement is set to guide the target large model to output the text answer and the image corresponding to the image link information when there is image link information associated with the text answer in at least one target document block. When answering questions through the target large model, if there is image link information associated with the text answer in the relevant document block, the answer is given in combination with the picture corresponding to the text answer and its associated image link information, thereby achieving attention to the relationship between images and text in the document and retaining the integrity of the document content, improving the accuracy and richness of information used to answer user questions, and avoiding the low quality of answers caused by missing image link information or improper processing of image content in related technologies.

[0070] It can be seen that the solution provided in this application achieves the purpose of answering user questions based on a combination of pictures and texts, thereby achieving the technical effect of improving the accuracy of answers, and further solving the technical problem that the large model in the related technology cannot answer user questions in combination with pictures, resulting in low answer accuracy.

[0071] In an optional embodiment, multiple document blocks include a first type of document block and a second type of document block, the document content of the first type of document block includes image link information and text content associated with the image link information, and the second type of document block does not include image link information. The generation process of the first type of document block includes: for each initial document among multiple initial documents, when the document content of the initial document includes an image, generating image link information corresponding to the image; replacing the image in the document content according to the image link information to obtain the target document content; and generating at least the first type of document block according to the target document content.

[0072] In some examples, after the target document content is obtained by replacing the image in the document content according to the image link information, the target document content is used to generate the first type of document block and the second type of document block may be generated at the same time.

[0073] In some examples, before determining at least one target document block required to answer a target question from multiple document blocks contained in a pre-built database, the target processing system can obtain multiple initial documents, and then for each initial document, when the document content of the initial document includes an image, generate image link information corresponding to the image, then replace the image in the document content based on the image link information to obtain the target document content, and when the document content of the initial document does not include an image, determine the target document content based on the document content of the initial document, thereby generating multiple document blocks based on the target document contents corresponding to the multiple initial documents.

[0074] Optionally, the initial document may be obtained by the target processing system from a designated storage area, for example, from the Internet, an organization's intranet, etc., and multiple initial documents may also be pre-input by the user.

[0075] In some embodiments, the initial document may be in PDF format. PDF format has cross-platform and cross-device stability, and can maintain consistent display effects on different operating systems and devices. This format conversion ensures the consistency of subsequent processing, making the management and use of documents more efficient. Optionally, the initial document may also be in other formats, such as word format.

[0076] Optionally, after acquiring multiple initial documents, for each initial document, the target processing system may generate image link information corresponding to the image when the document content of the initial document includes an image, and replace the image in the document content according to the image link information to obtain the target document content. For example, the target processing system may directly generate image link information corresponding to all images in the initial document. For another example, the target processing system may first filter the images in the initial document, and then generate image link information for the filtered images.

[0077] In some embodiments, the target processing system may first perform data type conversion on the document content of the initial document to convert the document content into a dictionary, such as a Python dictionary (Dict). Then, the target processing system may generate image link information corresponding to the image based on the image data in the dictionary.

[0078] In some embodiments, assuming that the document content of the initial document is as follows Figure 2 As shown, the target document content can be obtained as Figure 3 shown.

[0079] Optionally, for each initial document, the target processing system may determine the target document content directly based on the document content of the initial document when the document content of the initial document does not include an image. For example, the target processing system may perform OCR (Optical Character Recognition) on the document content of the initial document, thereby determining the recognized content as the target document content. For another example, the target processing system may perform data type conversion on the document content of the initial document to convert the document content into an operable dictionary, and then determine the data content of the operable dictionary as the target document content, so as to facilitate the subsequent determination of the document block.

[0080] Optionally, after determining the target document contents corresponding to multiple initial documents, the target processing system may perform slice processing on each target document content to obtain multiple slices corresponding to the target document content, thereby determining the slices corresponding to each target document content as a document block (first-category document block or second-category document block).

[0081] In some embodiments, the target processing system can slice the target document content according to the actual needs of the user. For example, it can be sliced ​​by page, and the content corresponding to each page of the initial document in the target document content is processed as an independent slice; for another example, it can be separated by the number of text units (Tokens), that is, specific elements in the document (such as paragraphs, words, etc.) are used as separation units to obtain multiple slices. In addition, more complex separation methods can be defined according to actual needs, such as multi-page separation with overlapping content of upper and lower pages.

[0082] Optionally, for the target document content including image link information, among the multiple slices obtained by slicing the target document content, there may be slices including image link information, and there may also be slices not including image link information (i.e., only including text content). Therefore, the target processing system can generate at least a first category of document blocks based on the target document content, and may also generate a second category of document blocks.

[0083] Optionally, for target document content that does not include image link information, among the multiple slices obtained by slicing the target document content, there are only slices that do not include image link information (i.e., only include text content). Therefore, the target processing system can generate a second type of document block based on the target document content.

[0084] Therefore, in this embodiment, the first type of document block is generated based on the target document content including image link information, and the second type of document block is generated based on the target document content including image link information and / or target document content not including image link information.

[0085] Optionally, after obtaining the multiple document blocks, the target processing system may store the multiple document blocks according to actual application requirements. Optionally, the target processing system may also perform vector conversion processing on the multiple document blocks to obtain vector representations corresponding to each document block, and store the vector representations corresponding to each document block, so that the target processing system can determine the target document block from the multiple document blocks.

[0086] It should be noted that by generating a link corresponding to the image in the document and inserting the link back into the original position of the document to replace the original image in the document, the semantic relationship between the image and the context words in the document is retained, thereby ensuring the integrity of the information in the document block and facilitating improving the accuracy of the target large model in answering questions.

[0087] In an optional embodiment, in the process of generating at least a first type of document block based on the target document content, the target processing system can slice the target document content to obtain multiple slices, and then extract the document name and summary information of the initial document from the initial document to which the slices belong, and splice the slices, document name and summary information to obtain spliced ​​slices, thereby generating at least a first type of document block based on the spliced ​​slices.

[0088] In this embodiment, after obtaining the target document content from the initial document containing the image, the target document content can be segmented into multiple slices, and then for each slice, the document name and summary information of the initial document are extracted from the initial document to which the slice belongs, and the slice, document name and summary information are spliced ​​to obtain a spliced ​​slice. Such a spliced ​​slice can be used as a document block. For example, the spliced ​​slice includes image link information, and the spliced ​​slice is used as a first-category document block. The spliced ​​slice does not include image link information, and the spliced ​​slice is used as a second-category document block.

[0089] Since the slicing process may cause each fragment to lose information related to the original document, thus affecting the understanding and consistency of the context, the name of the initial document to which the slice belongs and the summary information of the entire document can be spliced ​​at the beginning of each slice to form an overall string containing contextual information, ensuring that each fragment can retain the overall information of the document.

[0090] For example, after obtaining multiple slices corresponding to the target document content, the target processing system can extract the document name and summary information of the document from the initial document to which the slice belongs, and then for each slice, concatenate the document name and summary information at the beginning of the slice to obtain the concatenated slice (i.e., the above-mentioned overall string), thereby determining the concatenated slice as a document block. Optionally, if the concatenated slice includes image link information, the concatenated slice is determined as a first-type document block; if the concatenated slice does not include image link information, the concatenated slice is determined as a second-type document block.

[0091] For example, an alternative concatenated slice looks like this:

[0092] “Document name: xxx project report.pdf;

[0093] Document Summary: This report details the project's progress and achievements.

[0094] The document is sliced ​​as follows:

[0095] This report details the progress and achievements of the project. The project started in 2024 and has completed three phases. The first phase was mainly market research, and we found that the market demand was very strong. The second phase was product design, and we designed multiple prototypes and conducted user tests. The third phase was product development, and we have developed a preliminary product prototype and are currently conducting internal testing. Next we will enter the fourth phase, which is the product promotion phase. "

[0096] In an optional embodiment, for target document content that does not include image link information, the target processing system can use the above method to generate the second type of document blocks based on the target document content, so it will not be repeated here.

[0097] In some embodiments, after obtaining the target document content from the initial document that does not contain an image, in the process of generating the second type of document block based on the target document content, the target document content can be sliced ​​to obtain multiple slices, and then the document name and summary information of the initial document to which the slice belongs are extracted, and the slices, document names and summary information are spliced ​​to obtain spliced ​​slices, so as to generate the second type of document block based on the spliced ​​slices. In this way, the name of the initial document (i.e., a plain text document) that does not contain an image to which the slice belongs and the summary information of the entire document can be spliced ​​at the beginning of each slice to form an overall string containing context information, thereby ensuring that each segment can retain the overall information of the document.

[0098] Optionally, upon receiving the user's target question, the target processing system may convert the user's target question into a vector representation, and then perform similarity matching between the vector generated by the target question and the vector of the document block in the pre-built database, thereby sorting the retrieval results according to the similarity score and returning the most relevant document block. It should be noted that since each document block carries the document name and summary information, the retrieval results can more accurately reflect the contextual relationship of the document, thereby improving the accuracy and practicality of the retrieval.

[0099] In an optional embodiment, in the process of generating image link information corresponding to an image, the target processing system can obtain image information of the image in the initial document, and then identify interference images from the images in the initial document based on the image information, thereby generating image link information for images other than the interference images in the initial document.

[0100] When processing documents containing a large number of images, it is often encountered that some images do not actually contribute to the document content, but instead interfere with text parsing and information extraction. These images may include but are not limited to corporate logos, decorative gradient images, background images, solid color images, and other small icons with no practical meaning. When text paragraphs or graphic context information are broken at the page break, the complete contextual graphic information will be destroyed by the logo in the header and footer, which will lose contextual relevance, resulting in reduced slicing quality and reduced parsing quality. In order to improve the quality of document processing and ensure that the extracted content has high contextual relevance and practicality, these useless images can be identified before generating image link information to avoid generating image link information for them.

[0101] Optionally, the above-mentioned corporate logo, decorative gradient image, background image, solid color image and other small icons without practical meaning are equivalent to interference images. Interference images can be understood as images in the initial document whose importance is lower than a preset level.

[0102] Optionally, the target processing system may obtain image information of the image in the initial document, thereby identifying the interference image from the image in the initial document according to the image information. The image information includes at least one of the following: text information in the image, position information of the image in the initial document, and image size of the image.

[0103] For example, the target processing system may determine that the image is a logo, that is, an interference image, when text information in the image appears a large number of times.

[0104] For another example, the target processing system may determine that the image is an icon, that is, a distraction image, when the image size of the image is too small.

[0105] For another example, when the image information includes the position information of the image in the initial document, the target processing system can obtain a preset header position reference range and a footer position reference range, and thus determine that the image is an interference image when it is determined that the image position is within the header position reference range or the footer position reference range; and determine that the image is not an interference image when it is determined that the image position is outside the header position reference range and the footer position reference range.

[0106] Optionally, after determining the interfering images in the initial document, the target processing system may generate image link information for images other than the interfering images in the initial document.

[0107] It should be noted that, through the above process, it is possible to effectively filter out relatively useless images in the initial document before generating image link information. This can, on the one hand, improve the coherence of the document content and the consistency of the context, thereby improving the accuracy and reliability of document processing, and on the other hand, reduce the occupation of computing resources.

[0108] In an optional embodiment, when the image information includes text information in the image, in the process of identifying interference images from images in the initial document based on the image information, the target processing system can, for each image in the initial document, count the total number of times that images containing the text information in the image appear in the initial document, and count the total number of pages of the initial document, thereby determining whether the image is an interference image based on the numerical relationship between the total number of times corresponding to the image and the total number of pages.

[0109] Optionally, the target processing system may recognize text information in the image based on OCR technology.

[0110] For example, Figure 5 is a schematic diagram of an optional processing target document content according to an embodiment of the present invention, such as Figure 5As shown, the target processing system may first perform data type conversion on the document content of the initial document to convert the document content into a dictionary ( Figure 5 In the dictionary, text represents text and image represents image). After that, the target processing system can perform optical character recognition on the images corresponding to the image data in the dictionary to obtain the text information in each image (such as Figure 5 The text information recognized in the third step in the image is "logo"), and for each image, a counter mechanism is used to count the total number of times the image containing the text information in the image appears in the initial document (such as Figure 5 The total number of times identified in the third step is 3), and the total number of pages of the initial document is counted, for example, Figure 5 The total number of pages in the initial document is 3.

[0111] Furthermore, the target processing system can determine whether the image is an interference image based on the numerical relationship between the total number of times the image corresponds to and the total number of pages. Figure 5 As shown, if the total number of times the image corresponds to is the same as the total number of pages, the image is determined to be an interference image, and if the total number of times the image corresponds to is different from the total number of pages, the image is determined not to be an interference image. For another example, if the total number of times the image corresponds to is an integer multiple of the total number of pages, the image is determined to be an interference image, and if the total number of times the image corresponds to is not an integer multiple of the total number of pages, the image is determined not to be an interference image.

[0112] Optional, such as Figure 5 As shown, if there is an interfering image in the dictionary, the image data of the interfering image in the dictionary is deleted, that is, the interfering image is deleted.

[0113] It should be noted that, through the above process, effective recognition of interfering images in the document content is achieved, thereby improving the effectiveness of the target document content.

[0114] In an optional embodiment, when the image information includes the image size of the image, in the process of identifying interference images from images in the initial document based on the image information, the target processing system can, for each image in the initial document, determine that the image is an interference image when the image size of the image is less than or equal to a first preset value, and determine that the image is not an interference image when the image size of the image is greater than or equal to a second preset value, wherein the second preset value is greater than the first preset value, and when the image size of the image is greater than the first preset value and less than the second preset value, determine whether the image is an interference image based on the similarity between the image and a preset sample interference image.

[0115] For example, a size check is performed on each image in the initial document. If the image size is less than or equal to a certain threshold (i.e., the first preset value mentioned above, for example, 5KB), it is considered that the image may be a small prompt icon, a gradient image, a background image, a solid color image, or other interference images with no practical meaning, and the image is directly ignored.

[0116] If the image size is greater than or equal to a larger threshold (ie, the second preset value mentioned above, for example, 50KB), the image is considered to have practical significance and needs to be replaced with image link information and placed in the corresponding context text.

[0117] If the image size is between the two thresholds (ie, greater than the first preset value and less than the second preset value, for example, greater than 5 KB and less than 50 KB), it is determined whether the image is an interference image based on the similarity.

[0118] Optionally, the target processing system can maintain a sample library containing decorative images commonly used by enterprises or individual users when making documents, including some logos, animation blocks, large and small icons of various styles, etc., that is, including multiple sample interference images. These images are mainly used for decoration and have no practical meaning.

[0119] For images whose sizes are between the above two thresholds, the similarity can be compared with the sample interference images in the sample library. If the similarity between a certain image and any sample interference image is greater than a preset threshold (for example, 90%), the image is considered to be a decorative image (i.e., an interference image) and has no practical meaning.

[0120] It should be noted that by identifying whether an image is an interference image based on image size, the image recognition efficiency can be effectively improved. By combining the similarity between the image and the sample interference image for image recognition when the image size cannot obviously represent whether the image is a useless image, the image recognition accuracy can be effectively improved.

[0121] In an optional embodiment, before generating at least the first type of document blocks based on the target document content, the target processing system may determine whether the target document content includes text data for each target document content including image link information. If the target document content does not include text data, a content analysis is performed on the image corresponding to the image link information in the target document content to obtain image introduction information, thereby adding the image introduction information to the target document content as text content associated with the image link information to obtain updated target document content.

[0122] Optionally, if the target document content does not contain text data, it is determined that the image itself in the initial document corresponding to the target document content lacks contextual text description. Therefore, the image analysis model can be used to analyze the content in the image corresponding to the image link (i.e., image link information), and then the analysis text result (i.e., image description information) is associated with the image link and stored to form complete knowledge. Among them, the image analysis model can be a large language model, that is, image analysis can be understood as recognition, and the image description information can be understood as the text content obtained by semantically recognizing the image corresponding to the image link information.

[0123] Optionally, when adding image introduction information to the target document content, the image introduction information can be added above the image link information corresponding to the image introduction information or below the image link information in the target document content, and the image introduction information is adjacent to the image link information corresponding to the image introduction information.

[0124] For example, send an image link to the large language model and ask the large language model what the image in the image link means. If the image in the image link sent to the large language model is Figure 4 As shown, the image introduction information fed back by the large language model can be: "message = ChatCompletionMessage (content = 'The picture shows two people working in front of a computer, and through a graphical interface, various symbols and icons related to technology and network security are displayed. These icons include cloud computing, global, coding symbols (such as CSS, etc.), security locks, Wi-Fi signals, and other elements related to information technology. The whole scene shows a modern digital working environment and emphasizes the importance of Internet technology in business operations.', role = 'assistant', function_call = None, tool_calls = None))". Afterwards, the target processing system can add the above image introduction information to the top of the corresponding image link in the target document content to store it as a whole knowledge, for example, to obtain the following document content fragment:

[0125] “The image shows two people working at a computer, with a graphic interface displaying various symbols and icons related to technology and cybersecurity. These icons include cloud computing, a globe, coding symbols (such as CSS, etc.), a security lock, a Wi-Fi signal, and other elements related to information technology. The whole scene shows a modern digital working environment and emphasizes the importance of Internet technology in business operations.

[0126] https: / / www.abc.com.cn / wp-content / uploads / 2023 / navi.jpg”.

[0127] Optionally, if the target document content contains text data, the target processing system may not update the target document content. Alternatively, if the target document content contains text data, the target processing system may also perform content analysis on it and add the obtained image introduction information to the target document content to update the target document content.

[0128] In some embodiments, in the process of generating multiple document blocks based on target document contents corresponding to multiple initial documents, the target processing system may generate multiple document blocks based on the latest target document contents corresponding to the multiple initial documents, wherein the latest target document contents corresponding to the initial documents are the target document contents of the initial documents, or the updated target document contents of the initial documents.

[0129] It should be noted that by adding the image introduction information to the target document content, the text information related to the image in the target document content is supplemented, thereby improving the richness of the document content.

[0130] In an optional embodiment, when the document content of the initial document includes an image, in the process of generating image link information corresponding to the image, the target processing system can convert the data type of the document content in the initial document to obtain a dictionary corresponding to the initial document, and when the dictionary includes image data, generate image link information of the image corresponding to the image data based on the image data.

[0131] Optionally, dictionary can be a Python dictionary (Dict).

[0132] In this embodiment, the target processing system can use the Fitz library to convert the data type of the document content in the PDF document (i.e., the initial document) to parse its document content into a Python dictionary (Dict), which is an easy-to-process data structure. This conversion enables the target processing system to perform in-depth analysis and operation on the document content of the PDF document in a programmatic manner.

[0133] Optionally, in the case of obtaining a dictionary corresponding to the initial document, the target processing system may divide the dictionary into at least one target sub-data. For example, the data corresponding to each page of the initial document in the dictionary is used as a target sub-data to obtain at least one target sub-data. For another example, the dictionary is divided according to the image link information to obtain at least one target sub-data. For example, each image data in the dictionary is used as a target sub-data, and the image link information is used as a separation unit, and each continuous text data in the dictionary is used as a target sub-data. If the image data does not exist in the target document content, the dictionary is the same as the target sub-data. In this way, the target processing system can more accurately identify the image data in the dictionary, and effectively divide the image data and text data in the dictionary, so as to facilitate the subsequent image link information generation process.

[0134] Optionally, after determining at least one target sub-data corresponding to a certain dictionary, the target processing system may identify whether each target sub-data includes image data, so that if image data does not exist in at least one target sub-data corresponding to the dictionary, it is determined that the dictionary does not include image data. If target sub-data including image data exists in at least one target sub-data corresponding to the dictionary, it is determined that the dictionary includes image data.

[0135] Optionally, when the dictionary includes image data, the target processing system may generate image link information of the image corresponding to the image data based on the image data. For example, when the dictionary includes image data, target sub-data including the image data is determined from the dictionary to obtain the target sub-data to be processed, and image link information of the image corresponding to the image data is generated based on the image data in the target sub-data to be processed.

[0136] In some embodiments, the image data is binary data corresponding to the image, which is composed of multiple hexadecimal bytes and can be directly decoded and saved. The target processing system can restore the image according to the image data and generate corresponding image link information according to the restored image.

[0137] In some embodiments, after determining at least one target sub-data corresponding to a certain dictionary, the position information of the image in the initial document can be determined.

[0138] In some embodiments, since the header of the initial document may also have a title and the footer may also have a page number, when the text paragraph or graphic context information is broken at the page break, the complete context graphic information may also be destroyed by the header title and the footer page number, losing context relevance, causing the slicing quality and parsing quality to decline. Therefore, after determining at least one target sub-data corresponding to a certain dictionary, for each target sub-data including text content, the target processing system can determine the position information of the text content in the target sub-data in the initial document, and obtain the first position corresponding to the target sub-data, so that when it is determined that the first position corresponding to the target sub-data is within the header position reference range or the footer position reference range, it is determined that the text content of the target sub-data is in the header or footer, and it is marked, so that when generating a document block, this part of the marked target sub-data is filtered out. Optionally, when it is determined that the first position corresponding to the target sub-data is outside the header position reference range and the footer position reference range, it is determined that the text content of the target sub-data is not in the header or footer, and it is not marked. This can effectively filter out the content in the header and footer, ensure the coherence of the document body content and the consistency of the context, and improve the accuracy and reliability of document processing.

[0139] It should be noted that by converting the document content into an operational dictionary and then processing the operational dictionary, it is helpful for the target processing system to more accurately identify the image of the document content and more effectively generate image link information.

[0140] In an optional embodiment, in the process of generating image link information of an image corresponding to the image data based on image data, the target processing system can decode the image data to obtain the image corresponding to the image data, and then store the image corresponding to the image data in a target storage area, and generate image link information based on the storage position of the image in the target storage area.

[0141] Optionally, the target processing system can extract the image data from the dictionary and then decode the image data to obtain the original picture file (ie, image) corresponding to the image data. For example, after determining the target sub-data including the image data, the target processing system extracts the image data from the target sub-data.

[0142] Afterwards, the target processing system may store the decoded image in a target storage area, for example, the target storage area may be an HTTP server or other medium with storage function. After storing the image in the target storage area, the target processing system may generate image link information according to the storage location of the image in the target storage area. For example, if the target storage area is an HTTP server, the image link information may be an HTTP link.

[0143] It should be noted that, through the above process, accurate generation of image link information is achieved, so that the large model can accurately find the corresponding image according to the image link information.

[0144] In an optional embodiment, in the process of replacing the image in the document content according to the image link information to obtain the target document content, the target processing system can replace the image data in the dictionary according to the image link information to obtain an updated dictionary, and determine the data content in the updated dictionary as the target document content.

[0145] Optionally, the data content in the dictionary can be understood as document content represented by an operable data structure. Therefore, after obtaining the image link information, the target processing system can replace the image data in the dictionary according to the image link information to obtain an updated dictionary, and determine the data content in the updated dictionary as the target document content.

[0146] It should be noted that, since the dictionary is an operable data structure, by updating the dictionary using the image link information, the document content is effectively updated, thereby improving the effectiveness of the target document content.

[0147] In some embodiments, Figure 6 This is a flow chart of an optional problem solving method according to an embodiment of the present invention. Figure 2 ,according to Figure 6 An optional application process of this embodiment is described. Figure 6 As shown, the user can first determine the documents that need to be injected into the target large model knowledge base, and then convert the format of these documents to obtain the initial document in PDF format. After that, the PDF document is uploaded to the large model platform (i.e., the target processing system), and the PDF is parsed by the large model platform. During the parsing process, the large model platform can extract text and pictures in the order of the original document to obtain a dictionary, and then replace the image data in the dictionary with image link information to obtain the target document content. Then, the target processing system can import the parsed data (i.e., the target document content) into the knowledge base. For example, the parsed data is imported into the MySQL database and Qdrant vector database running in the Docker container according to certain splitting rules (i.e., split into document blocks).

[0148] Furthermore, if Figure 6As shown, when the big model platform obtains the user question (i.e., the target question), the big model platform can match the relevant knowledge fragments (i.e., at least one target document block) from the knowledge base according to the user question, and the knowledge fragments include text, or text and pictures. Afterwards, the big model platform can combine the matched relevant knowledge fragments with the prompt words to obtain a prompt sentence, and then send the prompt sentence to the big model (i.e., the target big model), so as to answer the user question according to the output result of the target big model.

[0149] For example, Figure 7 is a schematic diagram of an optional target large model answering question according to an embodiment of the present invention, such as Figure 7 As shown in the figure, when the target question raised by the user is “how to add a dictionary”, the information output by the target model is as follows: Figure 7 As shown in the left window, the effect of combining text and pictures to answer user questions is achieved.

[0150] In some embodiments, the prompt statement is also used to guide the target large model to output the content of the target document block where the text answer is located, for example, Figure 7 The [reference content 1] in the text is used to indicate the target document block where the text answer is located. Figure 7 After [Reference 1] in ( Figure 7 (not shown) can be used to display the content of the target document block.

[0151] In some embodiments, in the above process of obtaining the target document content, the large model platform can also identify and remove interfering images, and in the scenario where there is no contextual word description of the image in the document, call the relevant model to obtain the image description text information, and add the extracted text information before or after the location of the image.

[0152] In an optional embodiment, in the process of replacing an image in a document content based on image link information to obtain target document content, the target processing system may generate a number for the image corresponding to the image link information based on the position information of the image corresponding to the image link information in the initial document, and then combine the image link information and the number of the image corresponding to the image link information to obtain combined information, thereby replacing the image in the document content based on the combined information to obtain the target document content, wherein the prompt statement is also used to guide the target large model to determine the processing order of the image link information based on the numbers corresponding to the image link information when there are multiple image link information associated with the text answer.

[0153] In some embodiments, when processing a document containing multiple consecutive images, if these image links are sent to a large language model (LLM) for analysis and understanding, the model may not process the images in the original order. This situation will cause the content generated by the model to be inconsistent with the order of the images in the original document, thereby affecting understanding and application. In order to solve this problem, when converting the image to a picture link, a number can be added before the picture link to clarify the position of the picture in the original document. In addition, it can also be guided in the prompt statement so that the model can process the pictures in the correct order to avoid the problem of multiple pictures being out of order.

[0154] For example, in the process of replacing an image in the document content according to the image link information to obtain the target document content, the unique number corresponding to the image can be determined first. The number format is "

Page X Picture Y

Page 2 Figure 4

[0155] It should be noted that, through the above method, the model can process relevant reference images in the correct order when answering user questions, avoiding the situation where the model does not process images in the original order, thereby effectively improving the accuracy of the answer, that is, improving the quality of the answer.

[0156] It can be seen that this application can cover almost all scenarios of document parsing with images to ensure the integrity and relevance of knowledge injection. It can especially highlight the solution of the following scenarios: (1) Scenarios where there is no text in the picture. Such as real firewall pictures, landscape pictures, portraits, etc. It is impossible to capture text information based on the picture through OCR. (2) The picture is closely related to the context words. For example, the firewall UI operation manual. Each operation step text description corresponds to a picture description of the operation. It avoids the problem that the related technology only supports the parsing of pure text content in the document, does not support the processing of pictures in the document, resulting in missing picture information, and also avoids the problem that the related technology cannot process wordless pictures and cannot retain the connection between text and pictures. For example, when the user expects to ask about the model of the B2000 model, the model picture cannot be displayed. This application provides solutions for the following 6 special scenarios:

[0157] (1) In the case of repeated logo images, we count the images through OCR recognition and remove redundant interference images.

[0158] (2) Scenarios where headers and footers interfere with paging: For headers and footers, remove all content in the headers and footers, including logo images, header titles, and footer page numbers, to maintain the coherence of the context and the main text;

[0159] (3) Scenarios where there are a large number of meaningless images in the document: When processing the document, we identify and ignore the meaningless images by judging the image size and comparing the similarity of the sample library, and retain and link the meaningful images.

[0160] (4) Scenario of image description without context: When a user uploads a single image, the large model extracts the image description text and adds it above the image. Finally, the image information and HTTP link are stored as a slice in a vector.

[0161] (5) Scenarios where multiple consecutive images are out of order: To prevent the large model from processing multiple consecutive images out of order, you can add a number before the image link (such as "

Page 2 Figure 4

[0162] (6) Slicing scenarios where the original document body information is missing: The document name and summary information of the entire document are concatenated at the beginning of each fragment of the document as a whole as a slice.

[0163] The solution provided in this application achieves the purpose of answering user questions based on a combination of pictures and texts, thereby achieving the technical effect of improving the accuracy of answers, and further solving the technical problem in related technologies that large models cannot answer user questions in combination with pictures, resulting in low answer accuracy.

[0164] Example 2

[0165] According to an embodiment of the present invention, an embodiment of a problem handling device is provided, wherein: Figure 8 is a schematic diagram of an optional problem handling device according to an embodiment of the present invention, such as Figure 8 As shown, the device comprises:

[0166] A first determination module 801 is used to determine at least one target document block required to answer the target question from a plurality of document blocks included in a pre-built database when receiving a target question from a user, wherein document contents of at least some of the plurality of document blocks include image link information and text content associated with the image link information;

[0167] A first generating module 802 is used to generate a prompt sentence based on at least one target document block and a target question, wherein the prompt sentence is used to guide the target macromodel to determine a text answer corresponding to the target question according to the at least one target document block, and guide the target macromodel to output the text answer and an image corresponding to the image link information when there is image link information associated with the text answer in the at least one target document block;

[0168] The first processing module 803 is used to answer the target question according to the prompt sentence through the target large model.

[0169] It should be noted that the above-mentioned first determination module 801, first generation module 802 and first processing module 803 correspond to steps S101 to S103 in the above-mentioned embodiment, and the examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment 1.

[0170] Optionally, the multiple document blocks include a first type of document block and a second type of document block, the document content of the first type of document block includes image link information and text content associated with the image link information, and the second type of document block does not include image link information; the problem processing device also includes: a second generation module, for each initial document among the multiple initial documents, when the document content of the initial document includes an image, generating image link information corresponding to the image; a second processing module, for replacing the image in the document content according to the image link information to obtain the target document content; a third generation module, for generating at least the first type of document block based on the target document content.

[0171] Optionally, the second generation module also includes: an acquisition submodule, used to acquire image information of the image in the initial document, wherein the image information includes at least one of the following: text information in the image, position information of the image in the initial document, and image size of the image; an identification submodule, used to identify interference images from the images in the initial document based on the image information; and a first generation submodule, used to generate image link information for images other than the interference images in the initial document.

[0172] Optionally, the identification submodule also includes: a statistical unit, used to count the total number of times that images containing text information in the image appear in the initial document for each image in the initial document, and to count the total number of pages of the initial document; a first determination unit, used to determine whether the image is an interference image based on the numerical relationship between the total number of times corresponding to the image and the total number of pages.

[0173] Optionally, the identification submodule also includes: a second determination unit, used to determine, for each image in the initial document, that the image is an interference image when the image size of the image is less than or equal to a first preset value; a third determination unit, used to determine that the image is not an interference image when the image size of the image is greater than or equal to a second preset value, wherein the second preset value is greater than the first preset value; and a fourth determination unit, used to determine whether the image is an interference image based on the similarity between the image and a preset sample interference image when the image size of the image is greater than the first preset value and less than the second preset value.

[0174] Optionally, the problem handling device also includes: a judgment module, used to judge whether the target document content includes text data for each target document content including image link information; a third processing module, used to perform content analysis on the image corresponding to the image link information in the target document content to obtain image introduction information if the target document content does not contain text data; and a fourth processing module, used to add the image introduction information as text content associated with the image link information to the target document content to obtain updated target document content.

[0175] Optionally, the second generation module also includes: a first processing submodule, used to convert the data type of the document content in the initial document to obtain a dictionary corresponding to the initial document; a second generation submodule, used to generate image link information of the image corresponding to the image data based on the image data when the dictionary includes image data.

[0176] Optionally, the second generation submodule also includes: a processing unit, used to decode the image data to obtain an image corresponding to the image data; a generation unit, used to store the image corresponding to the image data in a target storage area, and generate image link information based on the storage position of the image in the target storage area.

[0177] Optionally, the second processing module also includes: a third generating submodule, used to generate the number of the image corresponding to the image link information according to the position information of the image corresponding to the image link information in the initial document; a second processing submodule, used to combine the image link information and the number of the image corresponding to the image link information to obtain combined information; a third processing submodule, used to replace the image in the document content according to the combined information to obtain the target document content, wherein the prompt statement is also used to guide the target large model to determine the processing order of the image link information according to the number corresponding to the image link information when there are multiple image link information associated with the text answer.

[0178] Optionally, the third generation module also includes: a fourth processing sub-module, used to slice the target document content to obtain multiple slices; an extraction sub-module, used to extract the document name and summary information of the initial document from the initial document to which the slices belong; a splicing sub-module, used to splice the slices, document name and summary information to obtain spliced ​​slices; and a fourth generation sub-module, used to generate at least a first type of document block based on the spliced ​​slices.

[0179] Example 3

[0180] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned problem handling method when running.

[0181] Example 4

[0182] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, wherein: Fig. 9 is a schematic diagram of an optional electronic device according to an embodiment of the present invention, such as Fig. 9 As shown, the electronic device includes one or more processors; a memory for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the above-mentioned problem handling method when running.

[0183] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0184] The embodiments or examples of the present disclosure are not exhaustive, but are only illustrative of some embodiments or examples, and are not intended to be specific limitations on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment or example can be implemented as an independent example, and the steps can be combined arbitrarily. For example, the scheme after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be combined arbitrarily; in addition, the various embodiments or examples can be combined arbitrarily, for example, some or all steps of different embodiments or examples can be combined arbitrarily, and a certain embodiment or example can be combined arbitrarily with the optional methods or optional examples of other embodiments or examples.

[0185] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0186] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0187] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0188] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0189] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A problem solving method, characterized in that: include: In the case of receiving a target question from a user, determining at least one target document block required to answer the target question from a plurality of document blocks included in a pre-built database, wherein document contents of at least some of the plurality of document blocks include image link information and text content associated with the image link information; Generate a prompt sentence based on the at least one target document block and the target question, wherein the prompt sentence is used to guide the target macromodel to determine a text answer corresponding to the target question according to the at least one target document block, and guide the target macromodel to output the text answer and an image corresponding to the image link information when there is image link information associated with the text answer in the at least one target document block; The target question is answered according to the prompt sentence by the target big model.

2. The method according to claim 1, characterized in that The multiple document blocks include a first type of document block and a second type of document block, the document content of the first type of document block includes the image link information and text content associated with the image link information, and the second type of document block does not include the image link information, and the generation process of the first type of document block includes: For each of the multiple initial documents, if the document content of the initial document includes an image, generating image link information corresponding to the image; Replacing the image in the document content according to the image link information to obtain the target document content; At least the first type of document blocks are generated according to the target document content.

3. The method according to claim 2, characterized in that Generate image link information corresponding to the image, including: Acquire image information of the image in the initial document, wherein the image information includes at least one of the following: text information in the image, position information of the image in the initial document, and image size of the image; identifying an interfering image from the image in the initial document according to the image information; Image link information is generated for images other than the interfering image in the initial document.

4. The method according to claim 3, characterized in that In a case where the image information includes text information in the image, identifying an interference image from the image in the initial document according to the image information includes: For each image in the initial document, counting the total number of occurrences of the image containing the text information in the image in the initial document, and counting the total number of pages of the initial document; Whether the image is the interference image is determined according to the numerical relationship between the total number of times corresponding to the image and the total number of pages.

5. The method according to claim 3, characterized in that: In a case where the image information includes the image size of the image, identifying an interfering image from the image in the initial document according to the image information includes: For each image in the initial document, when the image size of the image is less than or equal to a first preset value, determining the image as the interference image; When the image size of the image is greater than or equal to a second preset value, determining that the image is not the interference image, wherein the second preset value is greater than the first preset value; When the image size of the image is larger than the first preset value and smaller than the second preset value, it is determined whether the image is the interference image according to the similarity between the image and a preset sample interference image.

6. The method according to claim 2, characterized in that Before generating at least the first type of document blocks according to the target document content, the method further includes: For each target document content including image link information, determining whether the target document content contains text data; If the target document content does not contain text data, then performing content analysis on the image corresponding to the image link information in the target document content to obtain image introduction information; The image introduction information is added to the target document content as text content associated with the image link information to obtain updated target document content.

7. The method according to any one of claims 2 to 6, characterized in that In the case where the document content of the initial document includes an image, generating image link information corresponding to the image includes: Performing data type conversion on the document content in the initial document to obtain a dictionary corresponding to the initial document; When image data is included in the dictionary, image link information of an image corresponding to the image data is generated based on the image data.

8. The method according to claim 7, characterized in that Generating image link information of an image corresponding to the image data according to the image data, comprising: Decoding the image data to obtain an image corresponding to the image data; The image corresponding to the image data is stored in a target storage area, and the image link information is generated according to the storage position of the image in the target storage area.

9. The method according to claim 2, characterized in that: Replacing the image in the document content according to the image link information to obtain the target document content includes: generating a serial number of the image corresponding to the image link information according to the position information of the image corresponding to the image link information in the initial document; Combining the image link information and the serial number of the image corresponding to the image link information to obtain combined information; The image in the document content is replaced according to the combination information to obtain the target document content, wherein the prompt statement is also used to guide the target large model to determine the processing order of the image link information according to the numbers corresponding to the image link information when there are multiple image link information associated with the text answer.

10. The method according to claim 2, characterized in that Generating at least the first type of document block according to the target document content includes: Slicing the target document content to obtain multiple slices; Extracting the document name and summary information of the initial document from the initial document to which the slice belongs; Concatenating the slice, the document name, and the summary information to obtain a concatenated slice; At least the first type of document blocks are generated based on the spliced ​​slices.

11. A problem handling device, characterized in that: include: A first determination module is used to determine, upon receiving a target question from a user, at least one target document block required to answer the target question from a plurality of document blocks included in a pre-built database, wherein document contents of at least some of the plurality of document blocks include image link information and text content associated with the image link information; A first generating module is used to generate a prompt sentence based on the at least one target document block and the target question, wherein the prompt sentence is used to guide the target macromodel to determine a text answer corresponding to the target question according to the at least one target document block, and guide the target macromodel to output the text answer and an image corresponding to the image link information when there is image link information associated with the text answer in the at least one target document block; The first processing module is used to answer the target question according to the prompt sentence through the target large model.

12. An electronic device, characterized in that: The electronic device includes one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the problem handling method described in any one of claims 1 to 10 when run.

Images