Evaluation data set generation method and device, equipment, medium and product

Abstract

The invention provides an evaluation data set generation method and device, equipment, a medium and a product. The method comprises the steps of obtaining a first document; determining at least one first information block and at least one second information block of the first document; for each first information block in the at least one first information block, executing the following steps: determining a question text associated with the first information block according to the first information block or according to the first information block and at least one second information block; and generating an evaluation data set according to each first information block and the question text associated with the first information block. According to the method, the type of the first document is not limited, the evaluation data set corresponding to the document of any type can be generated, and the applicability of cross-modal information retrieval evaluation in the RAG technology is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, computer-readable storage medium, and computer program product for generating an evaluation data set. Background Art

[0002] With the rapid development of computer technology, language models have emerged. Language models typically possess natural language processing capabilities and are capable of handling various natural language tasks. To improve the reasoning performance of language models, the industry typically uses retrieval-augmented generation (RAG) technology to assist language models in task processing.

[0003] Specifically, RAG technology is divided into two phases: information retrieval and response generation. The information retrieval phase is used to retrieve information related to the question text from the knowledge base, while the response generation phase is used to combine the information related to the question text to generate the response text.

[0004] Typically, the industry evaluates the effectiveness of the information retrieval phase of RAG technology using evaluation datasets. These datasets typically include information indicating the retrieval task and information indicating the retrieval results. For example, in a scenario where news articles are retrieved based on their headlines, the information indicating the retrieval task might be a specific headline, and the information indicating the retrieval results might be news articles with that headline.

[0005] In related technologies, evaluation datasets can be used to evaluate both text and image retrieval tasks. However, these image-based information retrieval evaluation datasets are typically only used to evaluate the retrieval performance of photographic images. The industry urgently needs an image-based information retrieval evaluation dataset that can evaluate document image retrieval from documents. Summary of the Invention

[0006] This application provides a method for generating an evaluation dataset. This method can generate evaluation datasets corresponding to any type of document, improving the applicability of cross-modal information retrieval evaluation in RAG technology. This application also provides a device, electronic device, computer-readable storage medium, and computer program product corresponding to the above method.

[0007] In a first aspect, the present application provides a method for generating an evaluation dataset, the method comprising:

[0008] Get the first document;

[0009] Determining at least one first information block and at least one second information block of the first document; wherein the first information block is composed of an image and the second information block is composed of text;

[0010] For each first information block of the at least one first information block, performing the following steps: determining a question text associated with the first information block based on the first information block, or based on the first information block and at least one second information block; wherein the question text is used to retrieve the first information block from the first document;

[0011] An evaluation dataset is generated based on each first information block and the question text associated with the first information block; wherein the evaluation dataset is used to evaluate the accuracy of the image retrieval task, the question text in the evaluation dataset indicates the image retrieval task, and the first information block in the evaluation dataset indicates the image retrieval result.

[0012] In some possible implementations, for each first information block in the at least one first information block, the following steps are further performed:

[0013] Constructing a question-answer pair associated with the first information block; wherein the question-answer pair includes the question text and the answer text, and the answer text is used to answer the question text;

[0014] Generating an evaluation data set according to each first information block and the question text associated with the first information block includes:

[0015] An evaluation dataset is generated based on each first information block and the question-answer pair associated with the first information block; wherein the evaluation dataset is used to evaluate the accuracy of the image retrieval task and the response generation task, the question text in the evaluation dataset indicates the image retrieval task, the first information block in the evaluation dataset indicates the image retrieval result, and the answer text in the evaluation dataset indicates the response generation result.

[0016] In some possible implementations, determining at least one first information block and at least one second information block of the first document includes:

[0017] extracting at least one image from the first document and determining at least one first information block; and extracting at least one text paragraph from the first document and determining at least one second information block;

[0018] An information block sequence consisting of the at least one first information block and the at least one second information block is determined according to an order of the at least one first information block and the at least one second information block in the first document.

[0019] In some possible implementations, the method further includes:

[0020] determining an image category of the at least one first information block;

[0021] The first information block that does not match the key image category in the at least one first information block is filtered out.

[0022] In some possible implementations, determining the image category of the at least one first information block includes:

[0023] Generate a first prompt word; wherein the first prompt word includes the at least one first information block, descriptive information of at least one image category, and information indicating the image category of the first information block;

[0024] The first prompt word is sent to a multimodal model, and the image category of the at least one first information block returned by the multimodal model is received.

[0025] In some possible implementations, determining, based on the first information block, or based on the first information block and at least one of the second information blocks, a question text associated with the first information block includes:

[0026] Obtaining an image category of the first information block;

[0027] In response to the image category of the first information block being the first key image category, determining, based on the first information block, a question text associated with the first information block;

[0028] In response to the image category of the first information block being the second key image category, a question text associated with the first information block is determined based on the first information block and at least one second information block of contextual information belonging to the first information block in the first document.

[0029] In some possible implementations, determining, based on the first information block, or based on the first information block and at least one of the second information blocks, a question text associated with the first information block includes:

[0030] Generate a second prompt word; wherein the second prompt word includes: the first information block or the information block and at least one of the second information blocks, and information indicating the generation of a question text associated with the first information block;

[0031] The second prompt word is sent to the multimodal model, and the question text associated with the first information block returned by the multimodal model is received.

[0032] In some possible implementations, after constructing the question-answer pair associated with the first information block, the method further includes:

[0033] In response to the question-answer pair associated with the first information block satisfying a set condition, the question-answer pair associated with the first information block is filtered out, or the question-answer pair associated with the first information block is updated; wherein the set condition includes at least one of the following: for the question text in the question-answer pair, the answer text in the question-answer pair can be generated based only on at least one of the second information blocks; and the degree of association between the question text in the question-answer pair and other first information blocks in the first document other than the first information block satisfies an association degree condition.

[0034] In some possible implementations, in response to the question-answer pair associated with the first information block satisfying a set condition, filtering out the question-answer pair associated with the first information block, or updating the question-answer pair associated with the first information block, includes:

[0035] Generate a third prompt word; wherein the third prompt word includes the question-answer pair associated with the first information block, description information of the set condition, and information indicating whether the question-answer pair associated with the first information block meets the set condition;

[0036] Sending the third prompt word to the language model, and receiving a judgment result returned by the language model;

[0037] In response to the judgment result indicating that the question-answer pair associated with the first information block meets the set condition, the question-answer pair associated with the first information block is filtered out, or, based on the modified question-answer pair carried in the judgment result, the question-answer pair associated with the first information block is updated.

[0038] In a second aspect, the present application provides a device for generating an evaluation dataset, the device comprising:

[0039] An acquisition module, configured to acquire a first document;

[0040] a determination module, configured to determine at least one first information block and at least one second information block of the first document; wherein the first information block is composed of an image and the second information block is composed of text;

[0041] an execution module, configured to perform the following steps for each first information block of the at least one first information block: determining a question text associated with the first information block based on the first information block, or based on the first information block and at least one second information block; wherein the question text is used to retrieve the first information block from the first document;

[0042] A generation module is used to generate an evaluation data set based on each first information block and the question text associated with the first information block; wherein the evaluation data set is used to evaluate the accuracy of the image retrieval task, the question text in the evaluation data set indicates the image retrieval task, and the first information block in the evaluation data set indicates the image retrieval result.

[0043] In some possible implementations, the execution module is further configured to:

[0044] Constructing a question-answer pair associated with the first information block; wherein the question-answer pair includes the question text and the answer text, and the answer text is used to answer the question text;

[0045] The generation module is specifically used for:

[0046] An evaluation dataset is generated based on each first information block and the question-answer pair associated with the first information block; wherein the evaluation dataset is used to evaluate the accuracy of the image retrieval task and the response generation task, the question text in the evaluation dataset indicates the image retrieval task, the first information block in the evaluation dataset indicates the image retrieval result, and the answer text in the evaluation dataset indicates the response generation result.

[0047] In some possible implementations, the determining module is specifically configured to:

[0048] extracting at least one image from the first document and determining at least one first information block; and extracting at least one text paragraph from the first document and determining at least one second information block;

[0049] An information block sequence consisting of the at least one first information block and the at least one second information block is determined according to an order of the at least one first information block and the at least one second information block in the first document.

[0050] In some possible implementations, the apparatus further includes a filtering module, wherein the filtering module is configured to:

[0051] determining an image category of the at least one first information block;

[0052] The first information block that does not match the key image category in the at least one first information block is filtered out.

[0053] In some possible implementations, the filtering module is specifically configured to:

[0054] Generate a first prompt word; wherein the first prompt word includes the at least one first information block, descriptive information of at least one image category, and information indicating the image category of the first information block;

[0055] The first prompt word is sent to a multimodal model, and the image category of the at least one first information block returned by the multimodal model is received.

[0056] In some possible implementations, the execution module is specifically configured to:

[0057] Obtaining an image category of the first information block;

[0058] In response to the image category of the first information block being the first key image category, determining, based on the first information block, a question text associated with the first information block;

[0059] In response to the image category of the first information block being the second key image category, a question text associated with the first information block is determined based on the first information block and at least one second information block of contextual information belonging to the first information block in the first document.

[0060] In some possible implementations, the execution module is specifically configured to:

[0061] Generate a second prompt word; wherein the second prompt word includes: the first information block or the information block and at least one of the second information blocks, and information indicating the generation of a question text associated with the first information block;

[0062] The second prompt word is sent to the multimodal model, and the question text associated with the first information block returned by the multimodal model is received.

[0063] In some possible implementations, the filtering module is further configured to:

[0064] In response to the question-answer pair associated with the first information block satisfying a set condition, the question-answer pair associated with the first information block is filtered out, or the question-answer pair associated with the first information block is updated; wherein the set condition includes at least one of the following: for the question text in the question-answer pair, the answer text in the question-answer pair can be generated based only on at least one of the second information blocks; and the degree of association between the question text in the question-answer pair and other first information blocks in the first document other than the first information block satisfies an association degree condition.

[0065] In some possible implementations, the filtering module is specifically configured to:

[0066] Generate a third prompt word; wherein the third prompt word includes the question-answer pair associated with the first information block, description information of the set condition, and information indicating whether the question-answer pair associated with the first information block meets the set condition;

[0067] Sending the third prompt word to the language model, and receiving a judgment result returned by the language model;

[0068] In response to the judgment result indicating that the question-answer pair associated with the first information block meets the set condition, the question-answer pair associated with the first information block is filtered out, or, based on the modified question-answer pair carried in the judgment result, the question-answer pair associated with the first information block is updated.

[0069] In a third aspect, the present application provides an electronic device comprising a processor and a memory. The processor and the memory communicate with each other. The processor is configured to execute instructions stored in the memory to cause the electronic device to perform the method for generating an evaluation dataset as described in the first aspect or any implementation of the first aspect.

[0070] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct an electronic device to execute the method for generating an evaluation data set described in the first aspect or any one of the implementations of the first aspect.

[0071] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method for generating an evaluation data set as described in the first aspect or any one of the implementations of the first aspect.

[0072] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods.

[0073] It can be seen from the above technical solutions that this application has the following advantages:

[0074] The present application provides a method for generating an evaluation dataset, which first obtains a first document, then determines at least one first information block and at least one second information block of the first document, wherein the first information block consists of an image and the second information block consists of text, and for each first information block in the at least one first information block, performs the following steps: based on the first information block, or based on the first information block and at least one second information block, determines a question text associated with the first information block, wherein the question text is used to retrieve the first information block from the first document, and generates an evaluation dataset based on each first information block and the question text associated with the first information block, wherein the evaluation dataset is used to evaluate the accuracy of an image retrieval task, the question text in the evaluation dataset indicates the image retrieval task, and the first information block in the evaluation dataset indicates the image retrieval result.

[0075] In this method, image blocks (i.e., first information blocks) and text blocks (i.e., second information blocks) are extracted from a first document. For each image block, a question text that can retrieve the image block is determined. The image blocks and question text are combined to generate an evaluation dataset. The question text in the evaluation dataset is equivalent to the image retrieval task of retrieving images from documents, and the image blocks in the evaluation dataset are equivalent to the image retrieval results. Using the evaluation dataset, the effect of image information retrieval of document images from documents can be effectively evaluated. In this way, there is no restriction on the type of the first document, and an evaluation dataset corresponding to any type of document can be generated, thereby improving the applicability of cross-modal information retrieval evaluation in RAG technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0077] Figure 1 A flowchart of a method for generating an evaluation data set provided in an embodiment of the present application;

[0078] Figure 2 A flowchart of a method for generating an evaluation data set provided in an embodiment of the present application;

[0079] Figure 3 A schematic diagram of the structure of a device for generating an evaluation data set provided in an embodiment of the present application;

[0080] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0082] First, some technical terms and application scenarios involved in the embodiments of this application are introduced.

[0083] With the rapid development of computer technology, language models have emerged. They possess powerful expressive and learning capabilities, enabling them to handle a wide variety of natural language tasks, such as conversation, translation, speech recognition, text generation, and semantic analysis.

[0084] Typically, a language model performs natural language processing on a user's input text to generate a corresponding response text. The input text may indicate a question, an instruction, etc.

[0085] To improve the reasoning performance of language models, the industry typically uses retrieval-augmented generation (RAG) technology to assist language models in task processing. Specifically, RAG technology is divided into two phases: information retrieval and response generation. In the information retrieval phase, the recall model is invoked to retrieve information related to the question text from the knowledge base based on the user's input question text. Typically, knowledge data is vectorized and stored in the knowledge base. In other words, the knowledge base stores information in the form of vector representations. Therefore, in the information retrieval phase, the vector representation of the question text is determined, and the similarity between the vector representation of the question text and the vector representation of information in the knowledge base is calculated. Information with a similarity greater than a similarity threshold is then identified as relevant to the question text. In the response generation phase, the information related to the question text retrieved in the information retrieval phase is sent to the language model along with the question text, allowing the language model to analyze the question text in combination with the relevant information and generate a response text.

[0086] Typically, the industry evaluates the effectiveness of the information retrieval stage in RAG technology through evaluation datasets. The evaluation dataset usually includes information indicating the retrieval task and information indicating the retrieval results. In some examples, the information retrieval stage searches for text information. In this case, the evaluation dataset includes information indicating the text retrieval task and information indicating the text retrieval results. For example, the information indicating the text retrieval task in the evaluation dataset may be a news headline, and the information indicating the text retrieval results may be a news article with the news headline. For another example, the information indicating the text retrieval task in the evaluation dataset may be a question text, and the information indicating the text retrieval results may be a web page information corresponding to the question text.

[0087] In other embodiments, considering the need for cross-modal retrieval, the information retrieval stage may also target image-related information. In this case, the evaluation dataset includes information indicating the image retrieval task and information indicating the image retrieval results. For example, the information indicating the image retrieval task in the evaluation dataset may be image description information, and the information indicating the image retrieval results may be images that meet the description information.

[0088] However, the above-mentioned evaluation datasets for image information retrieval are usually only used to evaluate the retrieval effect of photographed pictures. For image information retrieval that retrieves document images from documents, since the images in the documents are usually closely related to the document context information, the document context information also needs to be considered in the image information retrieval that retrieves document images from documents. Therefore, the above-mentioned evaluation datasets are not suitable for evaluating the retrieval effect of image information retrieval.

[0089] In view of this, the present application provides a method for generating an evaluation dataset, which first obtains a first document, then determines at least one first information block and at least one second information block of the first document, wherein the first information block is composed of an image and the second information block is composed of text, and for each first information block in the at least one first information block, performs the following steps: based on the first information block, or based on the first information block and at least one second information block, determines a question text associated with the first information block, wherein the question text is used to retrieve the first information block from the first document, and generates an evaluation dataset based on each first information block and the question text associated with the first information block, wherein the evaluation dataset is used to evaluate the accuracy of the image retrieval task, the question text in the evaluation dataset indicates the image retrieval task, and the first information block in the evaluation dataset indicates the image retrieval result.

[0090] In this method, image blocks (i.e., first information blocks) and text blocks (i.e., second information blocks) are extracted from a first document. For each image block, a question text that can retrieve the image block is determined. The image blocks and question text are combined to generate an evaluation dataset. The question text in the evaluation dataset is equivalent to the image retrieval task of retrieving images from documents, and the image blocks in the evaluation dataset are equivalent to the image retrieval results. Using the evaluation dataset, the effect of image information retrieval of document images from documents can be effectively evaluated. In this way, there is no restriction on the type of the first document, and an evaluation dataset corresponding to any type of document can be generated, thereby improving the applicability of cross-modal information retrieval evaluation in RAG technology.

[0091] To facilitate understanding of the technical solutions provided in the embodiments of the present application, the following description will be made with reference to the accompanying drawings. Figure 1 FIG. 1 is a flow chart of a method for generating an evaluation dataset, wherein the method specifically includes:

[0092] S101: Acquire a first document.

[0093] The first document can be understood as any document that has an image retrieval task evaluation requirement. The first document includes an image and text. In other words, the first document contains at least one image, and the image retrieval task can be understood as the task of retrieving an image from the first document.

[0094] In the embodiment of the present application, there is no limitation on the type of the first document. For example, the first document can be a document of any format, any field, and any content.

[0095] In specific implementations, the first document can be obtained from different channels based on the actual evaluation requirements of the image retrieval task. In some embodiments, the first document can be a document in a knowledge base, in which case the first document can be obtained from the knowledge base. In other embodiments, the first document can be a local document, in which case the first document can be obtained locally.

[0096] S102: Determine at least one first information block and at least one second information block of a first document.

[0097] In this embodiment of the present application, the first information block consists of an image, and the second information block consists of text. In other words, the first information block can be understood as an image block, and the second information block can be understood as a text block. Considering the evaluation accuracy of subsequent image retrieval tasks, the first information block typically contains only one image, and the second information block contains only one text paragraph.

[0098] The first information block can take various forms. For example, the first information block can be an image formed by a chart, an image formed by a table, an image formed by a screenshot, an image obtained by photography, an image obtained by a document illustration, an image obtained by a company or organization logo, etc. In other words, all content in the first document other than plain text can be considered a first information block.

[0099] In some possible implementations, since the images and texts in the first document have a certain order, that is, for each image, there is text in the first document that is adjacent to the image and located before or after the image (that is, the context information of the image), in the task of retrieving the image from the first document, the influence of the context information needs to be considered. Therefore, the sequential relationship between at least one first information block and at least one second information block needs to be retained.

[0100] In a specific implementation, at least one image is extracted from a first document to determine at least one first information block, and at least one text paragraph is extracted from the first document to determine at least one second information block. Then, based on the order of the at least one first information block and the at least one second information block in the first document, an information block sequence consisting of the at least one first information block and the at least one second information block is determined.

[0101] That is, after extracting at least one first information block and at least one second information block from the first document, a sequence of information blocks consisting of at least one first information block and at least one second information block is formed according to the order in which the at least one first information block and at least one second information block are arranged in the first document. This, on the one hand, breaks the rich content contained in the first document into multiple information blocks, enabling finer-grained processing based on the information block dimension. On the other hand, the sequential relationship between the information blocks in the first document is preserved, allowing subsequent image retrieval tasks to be combined with the richer information in the first document to match the image retrieval task.

[0102] The embodiment of the present application does not limit the method of extracting the first information block and the second information block from the first document. For example, the first document can be parsed using an existing document parsing tool to obtain the first information block and the second information block of the first document.

[0103] It should be noted that, because the images and text in the first document can have different layouts, in some layouts, the arrangement order of the first information block and the second information block may not be fixed. For example, for a document page of the first document, the upper left area is a chart, the upper right area is a text paragraph, and the lower area is another text paragraph. In this case, the chart in the upper left area is determined to be the first information block A, the text paragraph in the upper right area is determined to be the second information block A, and the text paragraph in the lower area is determined to be the second information block B. In the information block sequence consisting of the first information block A, the second information block A, and the second information block B, the arrangement order can be [first information block A, second information block A, second information block B] or [first information block A, second information block B, second information block A].

[0104] In some embodiments, the first information blocks may be classified and filtered. Specifically, the image category of at least one first information block is determined, and first information blocks that do not match a key image category are filtered out.

[0105] The image category can be understood as a classification result after classifying at least one first information block. In an embodiment of the present application, the image category can be predefined. For example, the image category can include charts, tables, screenshots, photos, illustrations, logos, icons, QR codes, and others.

[0106] The key image category can be understood as the category of first information blocks that need to be retained and not filtered out. In other words, first information blocks belonging to the key image category are retained, while first information blocks that do not belong to the key image category are discarded. Subsequently, only first information blocks belonging to the key image category are used to generate the evaluation dataset.

[0107] Key image categories can be predefined. In some embodiments, based on the predefined image categories, image categories that contain rich information and have a high probability of appearing in image retrieval tasks are determined as key image categories. For example, key image categories may include charts, tables, screenshots, and photos, while illustrations, logos, icons, QR codes, and others are not considered key image categories.

[0108] On the one hand, first information blocks incorrectly extracted from the first document typically fall into the "other" image category. For example, first information blocks containing incomplete images or images with meaningless content can all fall into the "other" image category and are considered noise information blocks with low information content. By classifying the first information blocks, meaningless noise information blocks can be filtered out, ensuring the quality of the subsequent evaluation dataset construction.

[0109] On the other hand, considering that the first information blocks of different image categories have different probabilities of being retrieved in the image retrieval task, for example, the probability that a user retrieves a chart in the first document is much higher than the probability that a user retrieves an illustration in the first document, therefore, illustrations, logos, icons and QR codes, which are information blocks of image categories with a low probability of being used in image retrieval tasks, are filtered out to improve the efficiency of constructing subsequent evaluation datasets and the effectiveness of the evaluation datasets.

[0110] In some possible implementations, the at least one first information block is classified using a multimodal model. Specifically, a first prompt word is generated, the first prompt word is sent to the multimodal model, and the image category of the at least one first information block is received back from the multimodal model.

[0111] A multimodal model is an artificial intelligence (AI) model capable of processing multimodal information (e.g., text, image, audio, video, etc.). In this embodiment, since the first information block consists of an image, a multimodal model capable of processing image information is used to classify the first information block.

[0112] Multimodal models usually have natural language processing capabilities, can understand the meaning of natural language, and handle different types of natural language tasks. In an embodiment of the present application, the multimodal model determines the image category of the first information block based on prompt learning. Among them, prompt words can be used to guide the multimodal model to perform specific outputs in generative tasks (such as text generation tasks, question-answering tasks, and dialogue tasks). By configuring prompt words, the multimodal model is helped to understand the background and requirements of the task, so that the multimodal model can handle different types of natural language processing tasks without retraining the multimodal model, thereby increasing the scalability and flexibility of the language model.

[0113] In this embodiment of the present application, the first prompt word may include at least one first information block, at least one image category description information, and information indicating the image category of the first information block. The at least one image category description information may be understood as information describing the at least one image category in natural language content.

[0114] Because the first prompt word includes the aforementioned information, the multimodal model can analyze the first information block based on the prompt capability of the first prompt word and the descriptive information of at least one image category, and output the image category of the first information block. In this way, the first information block extracted from the first document is classified, allowing the first information block to be filtered based on image category.

[0115] The embodiment of the present application does not limit the type and version of the multimodal model. For example, the multimodal model may be a MiniCPM-V 2.6 model.

[0116] S103: For each first information block in the at least one first information block, perform the following steps: determine a question text associated with the first information block based on the first information block, or based on the first information block and at least one second information block.

[0117] The question text can be used to retrieve the first information block from the first document. In other words, the question text can be understood as text extracted from the first information block and used to perform the image retrieval task.

[0118] In an embodiment of the present application, two different methods are provided to determine the question text associated with the first information block: determining the question text using only the first information block, or determining the question text using the first information block and context information of the first information block.

[0119] In some possible implementations, the question text is determined in a manner corresponding to the image category of the first information block. Specifically, the image category of the first information block is obtained, and in response to the image category of the first information block being a first key image category, the question text associated with the first information block is determined based on the first information block. In response to the image category of the first information block being a second key image category, the question text associated with the first information block is determined based on the first information block and at least one second information block of contextual information belonging to the first information block in the first document.

[0120] That is to say, in an embodiment of the present application, considering that in the process of users retrieving images from the first document, different questioning methods can be used for different types of images, therefore, based on the image category of the first information block, the question text is determined by using the first information block or the first information block and contextual information to improve the matching degree between the question text and the first information block.

[0121] For example, for the first information block with the image category of "chart," the information density of the first information block is relatively high. When performing image retrieval, users typically ask questions about the content of the image itself in the first information block. Therefore, the image category of "chart" belongs to the first key image category, and the question text associated with the first information block can be determined based solely on the first information block. On the other hand, for the first information block with the image category of "screenshot," the first information block is typically used as a schematic diagram in the first document to explain a specific operating process or display a specific software interface. Therefore, the detailed content of the image in the first information block is less important. When performing image retrieval, users typically ask questions based on the contextual information of the first information block. Therefore, the image category of "screenshot" belongs to the second key image category, and the question text associated with the first information block can be determined based on the first information block and its contextual information.

[0122] In some embodiments, the first key image category may include charts and tables, and the second key image category may include screenshots and photos. In actual applications, the first key image category and the second key image category may also be flexibly configured based on the evaluation requirements of the actual image task retrieval, and this embodiment of the application does not impose any restrictions on this.

[0123] In some possible implementations, an information block sequence consisting of at least one first information block and at least one second information block is formed according to the arrangement order of at least one first information block and at least one second information block in the first document. In this case, the context information of the first information block may be composed of a set number of information blocks located before and after the first information block in the information block sequence. For example, when the set number is 5, the first information block is recorded as C i , the context information of the first information block may be [C i-5 ,…,C i-1 ,C i ,C i+1 ,…C i+5] spliced ​​together. For the second information block in the context information of the first information block, text information corresponding to the second information block is used as the context information, and for other first information blocks in the context information of the first information block, text placeholders corresponding to the other first information blocks (for example, text placeholders representing the positions of the other information blocks in the information block sequence) are used as the context information.

[0124] In practical applications, a larger set number can ensure complete coverage of the contextual information associated with the first information block, preventing the layout of images and text in the first document from causing the order of the information blocks in the information block sequence to differ, resulting in the contextual information of the first information block in the information block sequence being far away from the position of the first information block. On the other hand, a larger set number may interfere with the determination of the question text, reducing the degree of relevance between the question text and the first information block and making it more closely related to the contextual information of the first information block. Therefore, when setting the set number, comprehensive considerations should be made based on the actual situation of the first document.

[0125] In some possible implementations, the question text for each first information block is determined using a multimodal model. Specifically, a second prompt word is generated, the second prompt word is sent to the multimodal model, and the question text associated with the first information block is received back from the multimodal model.

[0126] In an embodiment of the present application, the multimodal model determines the question text for each first information block based on prompt learning. The second prompt word may include: the first information block or the information block and at least one second information block, and information indicating the generation of the question text associated with the first information block.

[0127] Because the second prompt word includes the aforementioned information, the multimodal model can analyze the first information block based on the second prompt word's prompting capabilities, or analyze the first information block in conjunction with the first information block's contextual information, and output the question text associated with the first information block. In this way, the question text associated with each first information block is determined.

[0128] In some embodiments, for each first information block, a question-answer pair associated with the first information block may be constructed, wherein the question-answer pair includes a question text and an answer text, and the answer text is used to answer the question text.

[0129] That is to say, for each first information block, in addition to generating a question text associated with the first information block, an answer text corresponding to the question text is also generated to form a question-answer pair of "question text-answer text".

[0130] In a specific implementation, a multimodal model can similarly be used to construct a question-answer pair for each first information block. In this case, the second prompt word may include: the first information block, or the information block and at least one second information block, and information indicating the construction of a question-answer pair associated with the first information block.

[0131] By constructing a question-answer pair associated with each first information block, it is possible to verify whether the question text is relevant to the first information block based on the answer text, and then update the question-answer pair to improve the quality of subsequent evaluation data sets.

[0132] Specifically, in response to the question-answer pair associated with the first information block satisfying a set condition, the question-answer pair associated with the first information block is filtered out, or the question-answer pair associated with the first information block is updated.

[0133] Among them, the set conditions include at least one of the following: for the question text in the question and answer pair, the answer text in the question and answer pair can be generated based only on at least one second information block, and the degree of association between the question text in the question and answer pair and other first information blocks in the first document except the first information block meets the association degree condition.

[0134] In other words, the question-answer pairs associated with the first information block are screened and modified to improve their quality, thereby improving the quality of the subsequently generated evaluation dataset. The fact that the answer text in a question-answer pair can be generated based solely on at least one second information block can be understood as meaning that the question text can be answered solely based on contextual information, and that the question text in the question-answer pair associated with the first information block is unrelated to the first information block itself.

[0135] The correlation degree condition can be understood as the correlation degree being greater than the correlation degree threshold value. The correlation degree between the question text in the question and answer pair and other first information blocks in the first document except the first information block satisfies the correlation degree condition. It can be understood as: the question text in the question and answer pair is too general and applicable to multiple first information blocks in the first document, and is not suitable as a question and answer pair associated with the first information block alone.

[0136] In this way, question-answer pairs that are irrelevant to the image content and can be answered only by relying on contextual information, as well as question-answer pairs that are too general and applicable to multiple images, are screened or modified to obtain a batch of high-quality question-answer pairs.

[0137] In some possible implementations, the question-answer pairs are filtered or updated with the aid of a language model. Specifically, a third prompt word is generated, the third prompt word is sent to the language model, and a judgment result returned by the language model is received. In response to the judgment result indicating that the question-answer pair associated with the first information block meets a set condition, the question-answer pair associated with the first information block is filtered out, or the question-answer pair associated with the first information block is updated based on the modified question-answer pair carried in the judgment result.

[0138] The language model has natural language processing capabilities, can understand the meaning of natural language, and handle different types of natural language tasks. For example, the language model can be a deep learning model trained using text data. In the embodiment of the present application, since the question-answer pairs associated with the first information block all represent natural language content and do not include other modal information such as image information, the language model can be used to filter or update the question-answer pairs.

[0139] The language model can judge whether the question-answer pair meets the set conditions based on prompt learning. The third prompt word can include the question-answer pair associated with the first information block, descriptive information of the set conditions, and information indicating whether the question-answer pair associated with the first information block meets the set conditions.

[0140] Since the third prompt word includes the above information, the language model can analyze whether the question and answer pair associated with the first information block meets the set conditions based on the prompt ability of the third prompt word and the descriptive information of the set conditions, and output a judgment result. When the judgment result represents that the question and answer pair associated with the first information block meets the set conditions, the judgment result can also include a modified question and answer pair, that is, the judgment result returned by the language model can also include a modified question and answer pair after modifying the question and answer pair associated with the first information block. In this way, for the question and answer pairs associated with each first information block, by judging whether the set conditions are met, the question and answer pairs are filtered out, or the question and answer pairs are modified to obtain question and answer pairs that do not meet the set conditions, so that the question and answer pairs are closer to the first information block, more specific, and higher in quality.

[0141] S104: Generate an evaluation data set according to each first information block and the question text associated with the first information block.

[0142] In an embodiment of the present application, the evaluation data set is used to evaluate the accuracy of the image retrieval task, the question text in the evaluation data set indicates the image retrieval task, and the first information block in the evaluation data set indicates the image retrieval result.

[0143] In other words, the evaluation dataset includes multiple "first information blocks and question texts associated with the first information blocks." When the evaluation dataset is applied to the evaluation of the information retrieval stage in the RAG technology, the question text associated with the first information block can be understood as an image retrieval task, such as "What is the distribution of the number of business processes?", and the first information block can be understood as an image retrieval result, such as the first information block A. In this way, the multiple question texts associated with the first information block in the evaluation dataset are used as multiple image retrieval tasks to evaluate whether the information retrieval stage in the RAG technology can retrieve the corresponding multiple first information blocks in the evaluation dataset, thereby implementing tests for image retrieval tasks for various types of documents.

[0144] In some embodiments, for each first information block, a question-answer pair associated with the first information block is constructed. In this case, an evaluation dataset is generated based on each first information block and the question-answer pair associated with the first information block.

[0145] Among them, the evaluation dataset is used to evaluate the accuracy of the image retrieval task and the response generation task, the question text in the evaluation dataset indicates the image retrieval task, the first information block in the evaluation dataset indicates the image retrieval result, and the answer text in the evaluation dataset indicates the response generation result.

[0146] That is to say, the evaluation dataset includes multiple "first information blocks and question-answer pairs associated with the first information blocks". When the evaluation dataset is applied to the evaluation of the information retrieval stage in the RAG technology, the question text in the question-answer pair can be understood as an image retrieval task, and the first information block can be understood as an image retrieval result. When the evaluation dataset is applied to the evaluation of the response generation stage in the RAG technology, the answer text in the question-answer pair can be understood as a response generation result. In the evaluation of the information retrieval stage and the response generation stage in the RAG technology, the corresponding multiple first information blocks in the evaluation dataset can be retrieved, and the retrieved first information blocks can be combined to generate accurate response generation results, thereby enhancing the scope of use and versatility of the evaluation dataset.

[0147] In this method, image blocks (i.e., first information blocks) and text blocks (i.e., second information blocks) are extracted from a first document. For each image block, a question text that can retrieve the image block is determined. The image blocks and question text are combined to generate an evaluation dataset. The question text in the evaluation dataset is equivalent to the image retrieval task of retrieving images from documents, and the image blocks in the evaluation dataset are equivalent to the image retrieval results. Using the evaluation dataset, the effect of image information retrieval of document images from documents can be effectively evaluated. In this way, there is no restriction on the type of the first document, and an evaluation dataset corresponding to any type of document can be generated, thereby improving the applicability of cross-modal information retrieval evaluation in RAG technology.

[0148] Next, the method for generating the evaluation data set provided in the embodiment of the present application is described with reference to a specific example. Figure 2 As shown, the first document is parsed, at least one first information block and at least one second information block are determined from the first document, the at least one first information block is classified using a multimodal model, and the image category of the at least one first information block is determined.

[0149] The first prompt word input to the multimodal model can be as follows:

[0150] Please classify the input image. The categories are defined as follows. Please select an appropriate category name and fill it in the "type" field:

[0151] -chart: charts, such as line charts, bar charts, pie charts, organizational charts, flow charts, etc.

[0152] -table: table.

[0153] -screenshot: A screenshot of a mobile phone or computer screen, such as a webpage or application. If the image is mostly text or has no other information, please classify it as else.

[0154] -photo: real photos taken

[0155] -background: Some illustrations used to beautify the document, which have little to do with the main content and are not real photos.

[0156] -logo: The logo of a company or organization.

[0157] -icon: Icon, which is small in size and has no practical meaning.

[0158] -qrcode: QR code.

[0159] -else: Other images, such as scanned pages, incomplete images, images containing only text information, etc.

[0160] Please output the results in the following JSON format without any additional content.

[0161] {

[0162] "type":"...",

[0163] }”

[0164] Among them, chart, table, screenshot, photo, background, logo, icon, qrcode, and else are image categories, and the content after each image category is the description information of the image category.

[0165] In combination with the image category of at least one first information block, at least one first information block is filtered, and for each filtered first information block, a multimodal model is used to construct a question-answer pair associated with the first information block based on the first information block, or based on the first information block and contextual information of the first information block.

[0166] The second prompt word input to the multimodal model can be as follows:

[0167] "You are an assistant specializing in multimodal RAG tasks.

[0168] The tasks are as follows:

[0169] Given an image from a document and the context of the image in the document, simulate the query questions that a user may ask to retrieve the image from a large document library and give the answer to the question.

[0170] Please note:

[0171] 1. If the image itself contains a lot of information, construct a query about the image's content. You can appropriately incorporate relevant content from the image's context to make the query more specific, but ensure that the question requires the image to answer. Conversely, if the image is merely a schematic, don't ask about the image's content. Instead, construct a query about the image's function in the context based on relevant information from the image's context.

[0172] 2. The user does not have the image yet when querying, so avoid generating queries like "What is the function of xxx in the picture?"

[0173] 3. The question should be related to the image, and the answer to the question should be found within the image or in the context of the image. The answer is output in a list format, where each element in the list is a possible answer to the original question.

[0174] 4. Do not include the name of the image in context in the question.

[0175] Generate up to three pairs of questions and answers in the dictionary, in the following format. Please only output this dictionary and do not answer other content:

[0176]

[0177] Where XXXXXX is the question and ['YYYYYY'] is the corresponding list of answers, which can be made longer as needed.

[0178] If there are no suitable questions, for example, the image itself has no information content or the user is unlikely to need to retrieve the image, an empty list is returned, such as {"questions":[]}

[0179] The following is the input image and its context in the document. The image's name in the context is {{image_name}}

[0180] {{image}}

[0181] {{context}}”

[0182] Among them, {{image}} is the first information block, and {{context}} is the context information of the first information block.

[0183] Furthermore, after constructing the question-answer pairs associated with each first information block, the language model can be used to filter and update the question-answer pairs. In some embodiments, the language model is used to determine whether the question-answer pair meets the set condition of "the answer text in the question-answer pair can be generated based solely on at least one second information block, with respect to the question text in the question-answer pair." The third prompt word input into the language model can be as follows:

[0184] You will be given a question and its answer, and you will be asked to determine whether the answer to the question can be found from the context. If it can be found, please output the relevant content in the context that is related to the given answer and answer "yes"; otherwise, no reason is required (return an empty string) and reply "no".

[0185] Please output in the following JSON format:

[0186]

[0187] - Question: {{question}}

[0188] -Answer: {{answer}}

[0189] -Context:{{context}}"

[0190] Among them, {{question}} is the question text, {{answer}} is the answer text, {{context}} is the context information of the first information block, and preds is the judgment result.

[0191] In other embodiments, a language model is used to determine whether a question-answer pair satisfies a set condition, "the degree of relevance between the question text in the question-answer pair and other first information blocks in the first document other than the first information block satisfies a relevance condition." The third prompt word input into the language model may be as follows:

[0192] You will get a question and its answer for an image. Please determine whether the question is too general. General questions are questions with low information content. Such questions can also be used to ask other images, rather than questions about a specific image. If the question is too general, try to generate a more specific and targeted question based on the question and answer. Otherwise, return an empty string in "question_new".

[0193] Here are some examples of overly general questions:

[0194] -"What does the red line in the picture represent?"

[0195] - "What does the picture show?"

[0196] - "Which option has the highest percentage?"

[0197] -"What is the value of XX?"

[0198] - "What are the main elements in the diagram?"

[0199] - "What does this chart illustrate?"

[0200] - "What are the objects in the diagram doing?"

[0201] - "What shape is the figure in the picture?"

[0202] -"What does this graphic mean?"

[0203] Please output in the following JSON format:

[0204] {

[0205] "preds":"The result of the judgment [yes / no]",

[0206] "question_new":"More specific questions"

[0207] }

[0208] Question: {{question}}

[0209] Answer: {{answer}}"

[0210] Among them, {{question}} is the question text, {{answer}} is the answer text, preds is the judgment result, and question_new is the modified question and answer pair.

[0211] After filtering or updating the question-answer pairs associated with each first information block, a high-quality evaluation dataset is generated for testing image retrieval tasks in various types of documents.

[0212] Combined with the above Figure 1 and Figure 2 The method for generating the evaluation data set provided in the embodiment of the present application is introduced in detail. The apparatus and device provided in the embodiment of the present application will be introduced below with reference to the accompanying drawings.

[0213] See also Figure 3 The schematic diagram of the structure of the device for generating the evaluation data set shown in FIG. 30 includes:

[0214] An acquisition module 301 is configured to acquire a first document;

[0215] A determination module 302 is configured to determine at least one first information block and at least one second information block of the first document; wherein the first information block is composed of an image and the second information block is composed of text;

[0216] The execution module 303 is configured to perform the following steps for each first information block in the at least one first information block: determining a question text associated with the first information block based on the first information block, or based on the first information block and at least one second information block; wherein the question text is used to retrieve the first information block from the first document;

[0217] The generation module 304 is used to generate an evaluation data set based on each first information block and the question text associated with the first information block; wherein the evaluation data set is used to evaluate the accuracy of the image retrieval task, the question text in the evaluation data set indicates the image retrieval task, and the first information block in the evaluation data set indicates the image retrieval result.

[0218] In some possible implementations, the execution module 303 is further configured to:

[0219] Constructing a question-answer pair associated with the first information block; wherein the question-answer pair includes the question text and the answer text, and the answer text is used to answer the question text;

[0220] The generating module 304 is specifically used for:

[0221] An evaluation dataset is generated based on each first information block and the question-answer pair associated with the first information block; wherein the evaluation dataset is used to evaluate the accuracy of the image retrieval task and the response generation task, the question text in the evaluation dataset indicates the image retrieval task, the first information block in the evaluation dataset indicates the image retrieval result, and the answer text in the evaluation dataset indicates the response generation result.

[0222] In some possible implementations, the determining module 302 is specifically configured to:

[0223] extracting at least one image from the first document and determining at least one first information block; and extracting at least one text paragraph from the first document and determining at least one second information block;

[0224] An information block sequence consisting of the at least one first information block and the at least one second information block is determined according to an order of the at least one first information block and the at least one second information block in the first document.

[0225] In some possible implementations, the apparatus further includes a filtering module, wherein the filtering module is configured to:

[0226] determining an image category of the at least one first information block;

[0227] The first information block that does not match the key image category in the at least one first information block is filtered out.

[0228] In some possible implementations, the filtering module is specifically configured to:

[0229] Generate a first prompt word; wherein the first prompt word includes the at least one first information block, descriptive information of at least one image category, and information indicating the image category of the first information block;

[0230] The first prompt word is sent to a multimodal model, and the image category of the at least one first information block returned by the multimodal model is received.

[0231] In some possible implementations, the execution module 303 is specifically configured to:

[0232] Obtaining an image category of the first information block;

[0233] In response to the image category of the first information block being the first key image category, determining, based on the first information block, a question text associated with the first information block;

[0234] In response to the image category of the first information block being the second key image category, a question text associated with the first information block is determined based on the first information block and at least one second information block of contextual information belonging to the first information block in the first document.

[0235] In some possible implementations, the execution module 303 is specifically configured to:

[0236] Generate a second prompt word; wherein the second prompt word includes: the first information block or the information block and at least one of the second information blocks, and information indicating the generation of a question text associated with the first information block;

[0237] The second prompt word is sent to the multimodal model, and the question text associated with the first information block returned by the multimodal model is received.

[0238] In some possible implementations, the filtering module is further configured to:

[0239] In response to the question-answer pair associated with the first information block satisfying a set condition, the question-answer pair associated with the first information block is filtered out, or the question-answer pair associated with the first information block is updated; wherein the set condition includes at least one of the following: for the question text in the question-answer pair, the answer text in the question-answer pair can be generated based only on at least one of the second information blocks; and the degree of association between the question text in the question-answer pair and other first information blocks in the first document other than the first information block satisfies an association degree condition.

[0240] In some possible implementations, the filtering module is specifically configured to:

[0241] Generate a third prompt word; wherein the third prompt word includes the question-answer pair associated with the first information block, description information of the set condition, and information indicating whether the question-answer pair associated with the first information block meets the set condition;

[0242] Sending the third prompt word to the language model, and receiving a judgment result returned by the language model;

[0243] In response to the judgment result indicating that the question-answer pair associated with the first information block meets the set condition, the question-answer pair associated with the first information block is filtered out, or, based on the modified question-answer pair carried in the judgment result, the question-answer pair associated with the first information block is updated.

[0244] The evaluation data set generation device 30 according to the embodiment of the present application may correspond to the method described in the embodiment of the present application, and the above and other operations and / or functions of each module / unit of the evaluation data set generation device 30 are respectively to achieve Figure 1 For the sake of brevity, the corresponding processes of the various methods in the illustrated embodiments are not described here in detail.

[0245] The embodiment of the present application also provides an electronic device. The electronic device is specifically used to implement Figure 3 The functionality of the device 30 for generating the evaluation data set in the illustrated embodiment.

[0246] Figure 4 A structural diagram of an electronic device 400 is provided. Figure 4 As shown, electronic device 400 includes bus 401, processor 402, communication interface 403 and memory 404. Processor 402, memory 404 and communication interface 403 communicate with each other via bus 401.

[0247] The bus 401 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0248] The processor 402 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0249] The communication interface 403 is used for communicating with the outside, for example, the communication interface 403 can be used for communicating with a terminal.

[0250] The memory 404 may include volatile memory, such as random access memory (RAM), or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0251] The memory 404 stores executable codes, and the processor 402 executes the executable codes to perform the aforementioned method for generating an evaluation data set.

[0252] Specifically, in the implementation Figure 3 In the case of the embodiment shown, and Figure 3 When each module or unit of the evaluation data set generation device 30 described in the embodiment is implemented by software, the execution Figure 3 The software or program code required for the functions of each module / unit in the system may be partially or completely stored in the memory 404. The processor 402 executes the program code corresponding to each unit stored in the memory 404 to perform the aforementioned method for generating the evaluation data set.

[0253] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned method for generating an evaluation data set applied to the generation device 30 of the evaluation data set.

[0254] The present application also provides a computer program product comprising one or more computer instructions that, when loaded and executed on a computing device, fully or partially generate the process or function described in the present application.

[0255] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0256] When the computer program product is executed by a computer, the computer performs any of the aforementioned methods for generating an evaluation dataset. The computer program product may be a software installation package, which can be downloaded and executed on a computer when any of the aforementioned methods for generating an evaluation dataset is required.

[0257] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to the various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the prescribed logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0258] The units involved in the embodiments described in this application may be implemented in software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0259] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0260] In the context of the present application embodiment, machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include but is not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0261] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0262] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0263] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0264] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0265] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating an evaluation data set, characterized in that: The method comprises: Get a first document; Determine at least one first information block and at least one second information block of the first document; wherein the first information block is composed of an image and the second information block is composed of text; For each first information block in the at least one first information block, the following steps are performed: determining a question text associated with the first information block according to the first information block, or according to the first information block and at least one of the second information blocks; wherein the question text is used to retrieve the first information block from the first document; An evaluation data set is generated according to each of the first information blocks and the question text associated with the first information block; wherein the evaluation data set is used to evaluate the accuracy of the image retrieval task, the question text in the evaluation data set indicates the image retrieval task, and the first information block in the evaluation data set indicates the image retrieval result.

2. The method according to claim 1, characterized in that For each first information block in the at least one first information block, the following steps are further performed: Constructing a question-answer pair associated with the first information block; wherein the question-answer pair includes the question text and the answer text, and the answer text is used to answer the question text; The step of generating an evaluation data set according to each of the first information blocks and the question text associated with the first information block comprises: An evaluation dataset is generated based on each of the first information blocks and the question-answer pairs associated with the first information blocks; wherein the evaluation dataset is used to evaluate the accuracy of an image retrieval task and a response generation task, the question text in the evaluation dataset indicates an image retrieval task, the first information block in the evaluation dataset indicates an image retrieval result, and the answer text in the evaluation dataset indicates a response generation result.

3. The method according to claim 1, characterized in that The determining of at least one first information block and at least one second information block of the first document comprises: extracting at least one image from the first document to determine at least one first information block; and extracting at least one text paragraph from the first document to determine at least one second information block; According to the order of the at least one first information block and the at least one second information block in the first document, an information block sequence consisting of the at least one first information block and the at least one second information block is determined.

4. The method according to claim 1, characterized in that The method further comprises: determining an image category of the at least one first information block; The first information block that does not match the key image category in the at least one first information block is filtered out.

5. The method according to claim 5, characterized in that: The determining the image category of the at least one first information block comprises: Generate a first prompt word; wherein the first prompt word includes the at least one first information block, at least one image category description information and information indicating the image category of the first information block; The first prompt word is sent to a multimodal model, and the image category of the at least one first information block returned by the multimodal model is received.

6. The method according to claim 1, characterized in that The determining, based on the first information block, or based on the first information block and at least one of the second information blocks, a question text associated with the first information block comprises: Acquire the image category of the first information block; In response to the image category of the first information block being a first key image category, determining a question text associated with the first information block according to the first information block; In response to the image category of the first information block being the second key image category, a question text associated with the first information block is determined based on the first information block and at least one second information block of context information belonging to the first information block in the first document.

7. The method according to claim 1, characterized in that The determining, based on the first information block, or based on the first information block and at least one of the second information blocks, a question text associated with the first information block comprises: Generate a second prompt word; wherein the second prompt word includes: the first information block or the information block and at least one of the second information blocks, and information indicating the generation of a question text associated with the first information block; The second prompt word is sent to the multimodal model, and the question text associated with the first information block returned by the multimodal model is received.

8. The method according to claim 2, characterized in that: After constructing the question-answer pair associated with the first information block, the method further includes: In response to the question-answer pair associated with the first information block satisfying a set condition, the question-answer pair associated with the first information block is filtered out, or the question-answer pair associated with the first information block is updated; wherein the set condition includes at least one of the following: for the question text in the question-answer pair, the answer text in the question-answer pair can be generated based only on at least one of the second information blocks; and the degree of association between the question text in the question-answer pair and other first information blocks in the first document except the first information block satisfies an association degree condition.

9. The method according to claim 8, characterized in that In response to the question-answer pair associated with the first information block satisfying a set condition, filtering out the question-answer pair associated with the first information block, or updating the question-answer pair associated with the first information block, comprises: Generate a third prompt word; wherein the third prompt word includes the question-answer pair associated with the first information block, description information of the set condition, and information indicating whether the question-answer pair associated with the first information block satisfies the set condition; Sending the third prompt word to the language model, and receiving the judgment result returned by the language model; In response to the judgment result indicating that the question-answer pair associated with the first information block satisfies the set condition, the question-answer pair associated with the first information block is filtered out, or, based on the modified question-answer pair carried in the judgment result, the question-answer pair associated with the first information block is updated.

10. A device for generating an evaluation data set, characterized in that: The device comprises: An acquisition module, used for acquiring a first document; A determination module, configured to determine at least one first information block and at least one second information block of the first document; wherein the first information block is composed of an image, and the second information block is composed of text; An execution module is used to perform the following steps for each first information block in the at least one first information block: determining a question text associated with the first information block according to the first information block, or according to the first information block and at least one of the second information blocks; wherein the question text is used to retrieve the first information block from the first document; A generation module is used to generate an evaluation data set based on each of the first information blocks and the question text associated with the first information block; wherein the evaluation data set is used to evaluate the accuracy of the image retrieval task, the question text in the evaluation data set indicates the image retrieval task, and the first information block in the evaluation data set indicates the image retrieval result.

11. An electronic device, characterized in that: The electronic device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The method comprises instructions, wherein the instructions instruct an electronic device to execute the method as claimed in any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises computer readable instructions for implementing the method according to any one of claims 1 to 9.

Images