Database construction method, information retrieval method and electronic equipment

Abstract

The invention provides a database construction method, an information retrieval method and electronic equipment. The database construction method comprises the following steps: acquiring a plurality of images corresponding to a plurality of local contents in a to-be-processed document; dividing the plurality of images into at least one image group; performing joint processing on the plurality of images in any image group to obtain a plurality of image feature vectors including context information corresponding to the plurality of images in the image group; and constructing a target database based on the plurality of image feature vectors.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, in particular to the technical field of big data, artificial intelligence and the like, and more particularly to a database construction method, an information retrieval method and an electronic device. BACKGROUND

[0002] In information retrieval, parsing documents and building databases are important preconditions. The construction of high-quality databases highly depends on accurate and comprehensive document parsing. However, in practical applications, document formats are diverse and complex, which brings great difficulties to document parsing. SUMMARY

[0003] Therefore, the present disclosure provides a database construction method, an information retrieval method and an electronic device.

[0004] According to a first aspect of the present disclosure, a database construction method is provided, comprising: obtaining a plurality of images corresponding to a plurality of local contents in a to-be-processed document; dividing the plurality of images into at least one image group; jointly processing the plurality of images in any image group to obtain a plurality of image feature vectors corresponding to the plurality of images in the image group, the image feature vectors comprising context information; and constructing a target database based on the plurality of image feature vectors.

[0005] According to an embodiment of the present disclosure, the plurality of images are divided into at least one image group, comprising: dividing the plurality of images into at least one image group according to the related information of the plurality of local contents; wherein the context information is determined based on at least the related information of the plurality of images in the image group.

[0006] According to an embodiment of the present disclosure, the plurality of images in any image group are jointly processed to obtain a plurality of image feature vectors corresponding to the plurality of images in the image group, the image feature vectors comprising context information, comprising: obtaining a plurality of text features corresponding to the plurality of images in any image group; embedding and encoding the plurality of text features based on a large language model to obtain a plurality of image feature vectors corresponding to the plurality of text features, the image feature vectors comprising context information.

[0007] According to an embodiment of the present disclosure, the plurality of text features corresponding to the plurality of images in any image group are obtained, comprising: obtaining a first image feature corresponding to the plurality of images in any image group; mapping the first image feature to a first text feature; embedding and encoding the plurality of text features based on a large language model to obtain a plurality of image feature vectors corresponding to the plurality of text features, the image feature vectors comprising context information, comprising: embedding and encoding the plurality of first text features based on a large language model to obtain a plurality of second text features corresponding to the plurality of first text features, the second text features comprising context information; mapping the plurality of second text features to a second image feature in a target dimensional space to obtain a plurality of image feature vectors, the target dimensional space corresponding to a task requirement of a target database.

[0008] According to an embodiment of the present disclosure, the obtaining the plurality of images corresponding to the plurality of local contents in the to-be-processed document comprises: forming an image corresponding to each local content according to the plurality of local contents in the to-be-processed document; or obtaining related information of the plurality of local contents in the to-be-processed document, the related information comprising at least one of the following: abstract information, theme information, title information, semantic information, keywords, or visual features; and forming an image corresponding to each local content according to the related information.

[0009] According to an embodiment of the present disclosure, the dividing the plurality of images into at least one image group according to the related information of the plurality of local contents comprises: determining a correlation degree between the plurality of local contents according to the related information of the local contents, the related information comprising at least one of the following: abstract information, theme information, title information, semantic information, keywords, or visual features; and in a case where the correlation degree between a first local content and a second local content is greater than a first threshold, dividing an image corresponding to the first local content and an image corresponding to the second local content into one image group.

[0010] The second aspect of the present disclosure provides an information retrieval method, comprising: obtaining at least one of a query text feature vector and a query image feature vector corresponding to query information; and determining at least one target image feature vector corresponding to the query information in a target database according to at least one of a first similarity and a second similarity, wherein the first similarity is a similarity between the query text feature vector and a plurality of image feature vectors of a plurality of images in the target database, and the second similarity is a similarity between the query image feature vector and the plurality of image feature vectors of the plurality of images in the target database; and the target database comprises image feature vectors corresponding to a plurality of local contents in a document and comprising context information.

[0011] According to an embodiment of the present disclosure, in a case where the target database comprises text feature vectors corresponding to the plurality of local contents in the document, the method further comprises: determining a third similarity between the query text feature vector and a plurality of text feature vectors in the target database, wherein the text feature vectors correspond to the image feature vectors; and determining at least one text feature vector corresponding to the query information according to the first similarity, the second similarity, and the third similarity.

[0012] According to an embodiment of the present disclosure, in a case where the query information is query text, the query text feature vector corresponding to the query information is obtained, including: embedding and encoding the query text based on a large language model to obtain intermediate text features; and mapping the intermediate text features to the query text feature vector in a target dimensional space; in a case where the query information is a query image, the query text feature vector corresponding to the query information is obtained, including: extracting text information in the query image; embedding and encoding the text information based on a large language model to obtain intermediate text features; and mapping the intermediate text features to the query text feature vector in a target dimensional space; and the query image feature vector corresponding to the query information is obtained, including: determining the query image feature vector according to the query image; wherein the target dimensional space corresponds to the task requirements of the target database.

[0013] According to an embodiment of the present disclosure, the query image feature vector is determined according to the query image, including: extracting a third image feature of the query image; mapping the third image feature to a third text feature; embedding and encoding the third text feature based on a large language model to obtain a fourth text feature; and mapping the fourth text feature to a fourth image feature in a target dimensional space to obtain the query image feature vector, wherein the target dimensional space corresponds to the task requirements of the target database.

[0014] According to an embodiment of the present disclosure, the at least one target image feature vector corresponding to the query information in the target database is determined according to the first similarity and the second similarity, including: determining a first intermediate value according to the first weight value and the first similarity, the first weight value representing the importance of the query text feature vector to the query information; determining a second intermediate value according to the second weight value and the second similarity, the second weight value representing the importance of the query image feature vector to the query information; determining a target similarity between the plurality of image feature vectors and the query information according to the first intermediate value and the second intermediate value; and in a case where the target similarity is greater than a second threshold value, determining the image feature vector corresponding to the target similarity as the target image feature vector.

[0015] A third aspect of the present disclosure provides a database construction apparatus, including: a first acquisition module configured to acquire a plurality of images corresponding to a plurality of local contents in a to-be-processed document; a division module configured to divide the plurality of images into at least one image group; a processing module configured to jointly process the plurality of images in any image group to obtain a plurality of image feature vectors corresponding to the plurality of images in the image group and including context information; and a construction module configured to construct a target database based on the plurality of image feature vectors.

[0016] The fourth aspect of the present disclosure provides an information retrieval device, comprising: a second acquisition module configured to acquire at least one of a query text feature vector and a query image feature vector corresponding to query information; a first determination module configured to determine at least one target image feature vector corresponding to the query information in a target database according to at least one of a first similarity and a second similarity; wherein the first similarity is a similarity between the query text feature vector and a plurality of image feature vectors of a plurality of images in the target database, and the second similarity is a similarity between the query image feature vector and the plurality of image feature vectors of the plurality of images in the target database; and the target database comprises image feature vectors corresponding to a plurality of local contents in a document and comprising context information.

[0017] The fifth aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method described above.

[0018] The sixth aspect of the present disclosure further provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method described above.

[0019] The seventh aspect of the present disclosure further provides a computer program product comprising a computer program, which, when executed by a processor, implements the method described above.

[0020] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0021] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A system architecture schematic diagram of a database construction method and an information retrieval method according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 2 A flowchart of a database construction method according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 3 A schematic diagram of a database construction process according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 4 A schematic diagram of a joint processing process according to an embodiment of the present disclosure is schematically shown;

[0026] Figure 5 A schematic diagram of a target database according to an embodiment of the present disclosure is shown schematically;

[0027] Figure 6 A flowchart of a method of information retrieval according to an embodiment of the present disclosure is shown schematically;

[0028] Figure 7A A schematic diagram of forming a query text feature vector and a query image feature vector from query information according to an embodiment of the present disclosure is shown schematically, where the query information is a query text;

[0029] Figure 7B A schematic diagram of forming a query text feature vector and a query image feature vector from query information according to an embodiment of the present disclosure is shown schematically, where the query information is a query image;

[0030] Figure 7C A schematic diagram of forming a query image feature vector from a query image according to an embodiment of the present disclosure is shown schematically;

[0031] Figure 8A One of the schematic diagrams of a process of determining a target similarity between a query text feature vector and a query image feature vector corresponding to query information and image feature vectors in a target database according to an embodiment of the present disclosure is shown schematically;

[0032] Figure 8B The second of the schematic diagrams of a process of determining a target similarity between a query text feature vector and a query image feature vector corresponding to query information and image feature vectors in a target database according to an embodiment of the present disclosure is shown schematically;

[0033] Figure 9A A block diagram of a database construction apparatus according to an embodiment of the present disclosure is shown schematically;

[0034] Figure 9B A block diagram of an information retrieval apparatus according to an embodiment of the present disclosure is shown schematically; and

[0035] Figure 10 A block diagram of an electronic device suitable for implementing the database construction method and the information retrieval method according to an embodiment of the present disclosure is shown schematically. DETAILED DESCRIPTION

[0036] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it would be apparent to those skilled in the art that the embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known structures and methods are not described in detail in order to avoid obscuring the concepts of the present disclosure.

[0037] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so forth, mean the term "comprises," as long as the above terms do not themselves appear herein.

[0038] All terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the specification, and should not be interpreted in an idealized or overly formal manner.

[0039] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should be generally interpreted as including one or more of the items enumerated in the list (e.g., "a system having at least one of A, B, and C" should include, but not be limited to, a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.).

[0040] The embodiments of the present disclosure provide a database construction method and an information retrieval method. Before introducing the technical solutions provided by the embodiments of the present disclosure, the related technologies involved in the present disclosure are described.

[0041] In information retrieval, parsing documents and building databases are important preconditions. The construction of high-quality databases highly depends on accurate and comprehensive document parsing work. However, in practical applications, document formats are various and complex, which brings great difficulties to document parsing.

[0042] In one example, with the rapid development of artificial intelligence technology and natural language processing technology, large language models (LLMs) have been widely applied in various fields, especially in the field of question answering (QA). LLMs can provide high-quality answers in various scenarios through their powerful language generation capabilities. However, relying solely on LLMs for question answering still faces the technical challenge of inaccurate answers. Based on the above technical challenges, the method of retrieval augmented generation (RAG) has emerged. RAG systems combine the advantages of information retrieval and generation models, relying not only on the generation capabilities of LLMs in question answering tasks, but also on relevant text fragments from external retrieval corpora as context input to the generation model, thereby generating more accurate and real-time answers.

[0043] In RAG systems, parsing documents and building knowledge bases are important preliminary steps. The construction of high-quality knowledge bases highly depends on accurate and comprehensive document parsing. However, in actual scenarios, document layouts are diverse and complex, posing great difficulties for document parsing. Multi-modal RAG, which directly stores image modalities, can preserve the layout structure information of source documents and is a promising solution. However, multi-modal RAG still has certain limitations: on the one hand, multi-modal RAG usually operates on each page of the document independently, making it difficult to capture cross-page context information. Cross-page context information is crucial for understanding the overall semantics and logical relationships of the document, and the lack of this information can significantly affect the accuracy and comprehensiveness of retrieval. On the other hand, in some graphic-text content scenarios, the pre-training of graphic-text alignment is not satisfactory. The deviation caused by graphic-text alignment can interfere with the model's accurate understanding of graphic-text related information, affecting retrieval accuracy.

[0044] Currently, the problem of missing cross-page context information is still being explored. As for the optimization of graphic-text alignment pre-training, only by fine-tuning model parameters can the graphic-text alignment accuracy in a specific scenario be improved to some extent, thereby improving retrieval accuracy. Fine-tuning the model can achieve more accurate graphic-text alignment in a specific scenario and improve retrieval accuracy. However, constructing model parameters for fine-tuning requires a lot of time and effort. Moreover, once the application scenario changes, the fine-tuned model may no longer be applicable, making it difficult to guarantee the graphic-text alignment accuracy and retrieval accuracy in the new scenario, resulting in time-consuming and labor-intensive fine-tuning.

[0045] Figure 1 The system architecture of the database construction method and the information retrieval method according to the embodiments of the present disclosure is schematically shown. It should be noted that, Figure 1The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0046] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0047] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as information retrieval applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0048] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0049] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0050] For example, server 105 can obtain multiple images corresponding to multiple local contents in a document to be processed from terminal devices 101, 102, and 103 via network 104; server 105 can divide the multiple images into at least one image group; and perform joint processing on the multiple images in any image group to obtain multiple image feature vectors for constructing a target database, wherein the image feature vectors include context information of the multiple images in the image group to which they belong.

[0051] In some embodiments, the server 105 can also use the target database described above to perform information retrieval. After the server 105 builds the target database, the server 105 can also use the target database to perform information retrieval on query information. In some examples, the server 105 can also send the target database to the terminal devices 101, 102, 103. In this way, a user can use the target object detection model in the terminal device to perform information retrieval.

[0052] It should be noted that the database construction method provided in the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the database construction apparatus provided in the embodiments of the present disclosure can generally be arranged in the server 105. The database construction method provided in the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Accordingly, the database construction apparatus provided in the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0053] It should be noted that the information retrieval method provided in the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the information retrieval apparatus provided in the embodiments of the present disclosure can generally be arranged in the server 105. The information retrieval method provided in the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Accordingly, the information retrieval apparatus provided in the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0054] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the system 100 is merely illustrative. Any number of terminal devices, networks, and servers can be provided according to implementation needs.

[0055] It should be noted that the serial numbers of the various operations in the following methods are merely used to represent the operations for description, and should not be regarded as representing the execution sequence of the various operations. The method does not need to be executed in the order shown unless explicitly indicated.

[0056] The database construction method provided in the embodiments of the present disclosure will be described in detail below based on the scenario described above. Figure 1 The database construction method provided in the embodiments of the present disclosure will be described in detail below based on the scenario described above. Figures 2-5 The database construction method provided in the embodiments of the present disclosure will be described in detail below based on the scenario described above.

[0057] Figure 2 A flowchart of a database construction method according to an embodiment of the present disclosure is shown schematically.

[0058] As shown in Figure 2 , the database construction method of this embodiment includes operations S210-S240.

[0059] In operation S210, a plurality of images corresponding to a plurality of local contents in a to-be-processed document are acquired.

[0060] In operation S220, the plurality of images are divided into at least one image group.

[0061] In operation S230, the plurality of images in any image group are jointly processed to obtain a plurality of image feature vectors including context information corresponding to the plurality of images in the image group.

[0062] In operation S240, a target database is constructed based on the plurality of image feature vectors.

[0063] By way of example, the to-be-processed document can be an original document having specific information for constructing the target database. By way of example, the to-be-processed document can record specific information in the form of a web page, a Word document, a PDF document, a PPT document, an image, etc.

[0064] The local content can be part of the specific information in the to-be-processed document. By way of example, in the case where the to-be-processed document is a PDF document of a certain paper, the local content can be a page, a natural paragraph, a sentence, a table, a figure, etc. in the PDF document. In the case where the to-be-processed document is a PPT document of a certain courseware, the local content can be a page, at least one sentence in a page, etc. in the PPT document. The present disclosure does not make specific limitations on the division of the local content, and the division rule of the local content in the document can be set according to actual application.

[0065] It should be noted that the local content can be obtained by dividing the entire content in the to-be-processed document. For example, a plurality of local contents of an article are obtained by dividing all natural paragraphs of a PDF document of the article. The local content can also be obtained by dividing part of the content in the to-be-processed document. For example, a plurality of local contents of a courseware are obtained by dividing the text pages (excluding the table of contents page and the acknowledgments page) of a PPT document of the courseware, and the plurality of local contents of the courseware do not include the entire content of the courseware.

[0066] In the embodiments of the present disclosure, the images corresponding to the plurality of local contents in the to-be-processed document are obtained by screenshot and saving. For example, a screenshot of a certain natural paragraph of a PDF document is taken to obtain an image of the natural paragraph (local content). The image contains at least part of the information of the local content. For example, the image can contain all the text content of a certain natural paragraph of the PDF document, or part of the text content of a certain natural paragraph of the PDF document. Here, it needs to be distinguished from the division rule of the local content described above. The image can have its own formation rule. In the case where the local content contains part of the text content of a certain natural paragraph of the PDF document, the division rule of the local content can be to divide according to the natural paragraph of the PDF document, and the image can be formed according to the key content in the local content after processing.

[0067] The image group can be obtained by dividing the images associated with each other in the plurality of images of the same to-be-processed document. The plurality of images in the same image group can be associated with each other in terms of local content, or in terms of the positions of the local content in the to-be-processed document, or in terms of the importance of the local content in the to-be-processed document. The division rule of the image group is not specifically limited in the embodiments of the present disclosure, and can be set according to actual needs.

[0068] The joint processing can be to interactively encode the information of the plurality of images in the image group to form an image feature vector of each image, wherein the image feature vector of each image is a fusion feature vector containing the context information of the plurality of images in the image group to which the image belongs. In other words, the image feature vector of each image contains not only the features corresponding to the image content of the image itself, but also the features associated with the image content of other images in the image group to which the image belongs.

[0069] ‌The context information refers to the environment, background and related information used to understand the target object or event in a specific scenario. The context information of the image can be the spatial layout relationship, semantic logical relationship and time sequence dependency of the visual features between the plurality of images in the image group; or can represent the association information between a certain image in the image group and other images in the image group. For the convenience of subsequent understanding, the context information of the text in natural language processing is also simply explained here, which refers to the text environment around the target word or sentence, including the semantic association of the preceding text (preceding text) and the following text (following text).

[0070] The multiple image groups in the same to-be-processed document can be divided according to the same division rule, and different image groups have different meanings for subsequent construction of the database. For example, after the image groups are divided according to the degree of association between the local content corresponding to the images, the multiple image groups formed by the same to-be-processed document can be jointly processed. After the image groups are divided according to the importance of the local content corresponding to the images in the to-be-processed document, the image groups with an importance greater than a preset threshold can be jointly processed, and the image groups with an importance less than or equal to the preset threshold are not jointly processed. That is, the multiple image groups formed by the same to-be-processed document are selectively jointly processed.

[0071] The target database can be a knowledge base composed of multiple image feature vectors. The target database can also be a knowledge base composed of multiple image feature vectors-local content, where the image feature vector-local content indicates that the image feature vector has the local content corresponding thereto, and the local content is the local content represented by the image corresponding to the image feature vector.

[0072] The target database can be used for information retrieval. Of course, the use of the target database is not limited thereto. For the sake of simplicity, the following examples are described by taking the target database used for information retrieval as an example.

[0073] In information retrieval, the image feature vector corresponding to the information can be retrieved from the target database for subsequent information processing. The image feature vector corresponding to the information can also be retrieved from the target database first, and then the corresponding local content is determined according to the correspondence between the image feature vector and the local content, for subsequent information processing. The present disclosure does not limit this, and additional information (for example, local content, images, etc.) about the image feature vector in the target database can be added or reduced according to actual application requirements.

[0074] It can be understood that, on the one hand, according to the association between multiple local contents in the same document, the images corresponding to the multiple local contents are grouped, the logically associated local contents are associated, the local semantic context environment of the document is constructed, the information fragmentation between the local contents of the document is not easy to occur, and the processing efficiency of information interaction in the subsequent joint processing of images is improved. On the other hand, the multiple images in the same group are jointly processed, the joint understanding between different local contents in the document is unified, the multi-modal fusion processing is realized, the multiple images in the group interact with each other, the context information between the images is captured, the implicit association between the local contents is captured, the implicit association between the local contents is in the form of context information in the image feature vector, and the subsequent application process of the document content is more complete and accurate.

[0075] Figure 3A schematic diagram of a database construction process according to an embodiment of the present disclosure is shown.

[0076] In one example, referring to Figure 3 Taking the article in the document 1: PDF version as an example. The partial content of the document 1 is divided according to the natural paragraph (the picture and the table are also a natural paragraph), and the document 1 includes partial content 1, partial content 2, partial content 3, partial content 4 and partial content 5. The different partial content in the document 1 can be screenshot to obtain the corresponding image, the partial content 1 is screenshot to obtain the image 1, the partial content 2 is screenshot to obtain the image 2, the partial content 3 is screenshot to obtain the image 3, the partial content 4 is screenshot to obtain the image 4, and the partial content 5 is screenshot to obtain the image 5. According to the correlation degree between the partial content corresponding to the five images, the five images are divided into two image groups, wherein the image 1, the image 2 and the image 5 are divided into the image group 1, and the image 3 and the image 4 are divided into the image group 2. The image 1, the image 2 and the image 5 in the image group 1 are processed jointly to make the information interaction between the image 1, the image 2 and the image 5, and the image feature vector 1 corresponding to the image 1, the image feature vector 2 corresponding to the image 2 and the image feature vector 5 corresponding to the image 5 are obtained. The image feature vector 1 contains the correlation information (context information) between the image 1, the image 2 and the image 5, the image feature vector 2 contains the correlation information (context information) between the image 2, the image 1 and the image 5, and the image feature vector 5 contains the correlation information (context information) between the image 5, the image 1 and the image 2. The image 3 and the image 4 in the image group 2 are processed jointly to make the information interaction between the image 3 and the image 4, and the image feature vector 3 corresponding to the image 3 and the image feature vector 4 corresponding to the image 4 are obtained. The image feature vector 3 contains the correlation information (context information) between the image 3 and the image 4, and the image feature vector 4 contains the correlation information (context information) between the image 3 and the image 4. The image feature vectors (1-5) corresponding to the multiple partial content in the document 1 are stored in the target database.

[0077] As described above, in operation S220, the multiple images are divided into at least one image group. In one implementable manner, the operation can further include: dividing the multiple images into at least one image group according to the correlation information of the multiple partial content; wherein the context information is determined based on at least the correlation information of the multiple images in the image group.

[0078] Exemplarily, the correlation information of the partial content can be the content information, the position in the document, the logical order, etc. of the partial content.

[0079] The contextual information can reflect the association between the multiple local content-related information corresponding to the multiple images in the same image group. The contextual information can also reflect the association between the implicit information of the multiple local content-related information corresponding to the multiple images in the same image group.

[0080] For example, in a legal document, images corresponding to different local contents including the “breach of contract” clause can be grouped together. In an article, images corresponding to different local contents including “experimental data” and “analysis results” can be grouped together because the experimental data is usually used together with the analysis results.

[0081] It can be understood that grouping the images corresponding to the local contents into image groups according to the related information of the local contents can ensure the semantic and logical integrity between the multiple local contents in the same group.

[0082] In some embodiments, grouping the multiple images into at least one image group according to the related information of the multiple local contents can further include: determining the association degree between the multiple local contents according to the related information of the local contents, the related information including at least one of the following: summary information, theme information, title information, semantic information, keywords, or visual features; and grouping the images corresponding to the first local content and the images corresponding to the second local content into one image group if the association degree between the first local content and the second local content is greater than a first threshold.

[0083] For example, in the case of text content, the summary information can be a compressed restatement of the text content of the local content; the theme information can be the core topic (for example, environmental protection, economy, etc.) of the text content of the local content; the title information can be the label (for example, the subheading in the article, etc.) of the text content of the local content; the keywords can be the core words representing the text content of the local content; and the semantic information can be the implicit information expressed by the local content (for example, the implicit information between the sentences of the text content). In the case of images, the semantic information can be the abstract information related to the meaning of the content in the image; and the visual features can be the bottom representation of the image.

[0084] The summary information, the theme information, the title information, the semantic information, the keywords, or the visual features can be obtained after extracting the local content. The title information can be directly obtained from the local content. For example, in a PDF document of an article, each part has a subheading, which can be directly used as the title information of the local content.

[0085] In one example, in the case of local content being text, the title 1 of the local content 1 is: related drugs for treating high blood pressure, and the title 2 of the local content 2 is: side effects of related drugs for high blood pressure. The correlation degree (80%) between the title 1 and the title 2 can be calculated by means of the BGE-M3 model, and the correlation degree (80%) between the local content 1 and the local content 2 is greater than the first threshold (60%), so the image 1 corresponding to the local content 1 and the image 2 corresponding to the local content 2 can be divided into the same image group.

[0086] The disclosure embodiments do not make specific limitations on the size of the first threshold, and the size of the first threshold can be adjusted according to actual application.

[0087] It can be understood that by calculating the correlation degree between the related information of the local content, the images corresponding to the local content are divided into image groups, which can ensure the semantic and logical integrity between multiple local contents in the same group.

[0088] As described above, in operation S230, the multiple images in any image group are jointly processed to obtain multiple image feature vectors corresponding to the multiple images in the image group, including context information. In one implementable manner, the operation can further include: obtaining multiple text features corresponding to the multiple images in any image group; embedding and encoding the multiple text features based on a large language model to obtain corresponding multiple image feature vectors including context information.

[0089] In one example, taking the image group 2 obtained by dividing the local content of the above document 1 as an example, the text feature 3 and the text feature 4 corresponding to the image 3 and the image 4 in the image group 2 are obtained. The text feature 3 and the text feature 4 are input into a large language model (LLM) for embedding and encoding at the same time, so as to capture the context information between the text feature 3 and the text feature 4, and obtain the image feature vector 3 of the image 3 and the image feature vector 4 of the image 4. The image feature vector 3 contains not only the basic information of the image 3 itself but also the association information (context information) between the image 3 and the image 4. The image feature vector 4 contains not only the basic information of the image 4 itself but also the association information (context information) between the image 3 and the image 4.

[0090] It can be understood that mapping multiple images into text features can provide more semantic information for the images, enhance the context perception ability between the images, and facilitate the large language model to process in the text feature space, so as to better capture the context information between the images.

[0091] Figure 4 An illustrative diagram of a joint processing process according to the embodiments of the disclosure is schematically shown.

[0092] As described above, the plurality of text features corresponding to the plurality of images in any image group are obtained. In an implementable manner, the plurality of first image features corresponding to the plurality of images in any image group can be further obtained; and the first image features are mapped to the first text features.

[0093] Exemplarily, the first image features can include visual features and semantic features of the images. The first text features can be representations of the image features in a text feature space.

[0094] In an example, referring to Figure 4 The plurality of images in the same image group are collectively input into a visual encoder to obtain the first image features. The visual encoder converts the original image data (pixels) of the images into structured feature representations (global feature vectors), i.e., maps the images from a "pixel space" to a "feature space". The first image features are input into a first mapping layer to obtain the first text features. The first mapping layer maps the first image features from image space vectors to text space vectors to achieve image-text alignment.

[0095] It can be understood that mapping the image features of the images to the text features can break the modal barrier and achieve semantic alignment of the images and the text; at the same time, the text features facilitate subsequent understanding of the image features of the images by the large language model.

[0096] As described above, the plurality of text features corresponding to the plurality of images in any image group are obtained. In an implementable manner, the plurality of first image features corresponding to the plurality of images in any image group can be further obtained; and the first image features are mapped to the first text features.

[0097] Exemplarily, the second text features can be results of interactions of the plurality of first text features, and the second text features not only contain basic information of the first text features, but also contain context information between the plurality of first text features.

[0098] The second image features can be representations of the second text features in an image feature space.

[0099] The target dimension space can be a dimension of a vector space that the image feature vector corresponding to the task requirement of the target database needs to satisfy. For example, the task requirement of the target database can be a classification task, a retrieval task, etc. In the case where the task requirement of the target database is a classification task, the target dimension space can be composed of several dimensions, each of which represents the features of a certain specific category. Mapping the text to these dimensions can determine which category it belongs to. For example, in the case where the task requirement of the target database is a retrieval task, the target dimension space can be a high-dimensional space, and each dimension can represent a certain feature of the image (such as color, shape, texture, etc.).

[0100] In one example, continuing to refer to Figure 4 The plurality of first text features are simultaneously input into the large language model for embedding coding to obtain a plurality of second text features. The large language model can capture context information, thereby generating more detailed and deep text representations. The plurality of second text features are input into the second mapping layer to obtain second image features, i.e., the image feature vectors corresponding to the images.

[0101] It can be understood that the large language model can capture the context information between the images by embedding coding the text features, thereby capturing the implicit associations between the local contents and existing the implicit associations between the local contents in the form of context information in the image feature vectors.

[0102] It can be understood that storing the converted text features into the target database as image features facilitates subsequent unified image feature retrieval of the contents in the document.

[0103] In some embodiments, the first image features obtained by inputting the plurality of images in the same image group into the encoder for coding can also be a group of feature vectors. Therefore, the second image features output after the first mapping layer, the large language model embedding coding, and the second mapping layer are also a group of feature vectors. Splitting the group of feature vectors can obtain the image feature vectors corresponding to each image.

[0104] As described above, in operation S210, a plurality of images corresponding to a plurality of local contents of a to-be-processed document are obtained. In one implementable manner, the operation can further include: forming an image corresponding to each local content according to the plurality of local contents in the to-be-processed document.

[0105] In one example, the image can be directly formed according to all the contents of the local content. For example, a screenshot of a certain paragraph in an article is obtained to obtain an image corresponding to the paragraph.

[0106] As described above, in operation S210, a plurality of images corresponding to a plurality of local contents of a document to be processed are acquired. In another implementation, the operation can further include: acquiring related information of the plurality of local contents of the document to be processed, the related information including at least one of: abstract information, theme information, title information, semantic information, keywords, or visual features; and forming an image corresponding to each local content according to the related information.

[0107] For example, for the related information being abstract information, theme information, title information, semantic information, keywords, or visual features of the local content, reference can be made to the description of the abstract information, theme information, title information, semantic information, keywords, or visual features above, which will not be repeated here.

[0108] The image corresponding to the local content can be formed according to the abstract information, theme information, title information, semantic information, keywords, or visual features of the local content. The abstract information, theme information, title information, semantic information, keywords, or visual features of the local content can be a summary expression of the local content, and not the entire content of the local content.

[0109] In one example, taking the document to be processed as a medical article, the first local content 1 is the first paragraph in the article, the keywords of the first paragraph are extracted as: medical technology, development, and the image 1 corresponding to the first local content 1 is formed according to the keywords: medical technology, development. The second local content 2 is the second paragraph in the article, the keywords of the second paragraph are extracted as: hypertension, treatment process, and the image 2 corresponding to the second local content 2 is formed according to the keywords: hypertension, treatment process.

[0110] It can be understood that forming the image corresponding to the local content according to the related information of the local content not only represents the local content, but also reduces the image information and saves the storage space of the database.

[0111] Figure 5 A schematic diagram of a target database according to an embodiment of the present disclosure is schematically shown.

[0112] In some embodiments, the target database can only store image feature vectors of a plurality of local contents in different documents. In other embodiments, referring to Figure 5 , the target database can store image feature vectors of a plurality of local contents in different documents and the local contents corresponding to the image feature vectors. In still other embodiments, if the image feature vectors are related information (such as keywords, etc.) of the local contents, the target database can store image feature vectors of the local contents (related information), image feature vectors of the local contents (entire content), and the local contents (entire content) corresponding to the image feature vectors.

[0113] The following will be based on Figure 1 The described scenario, by Figures 6-8B The information retrieval method of the embodiments of the present disclosure is described in detail.

[0114] Figure 6 The flowchart of the information retrieval method according to the embodiments of the present disclosure is schematically shown.

[0115] As Figure 6 shown, the information retrieval method of the embodiments includes operation S310~operation S320.

[0116] In operation S310, at least one of the query text feature vector and the query image feature vector corresponding to the query information is acquired.

[0117] In operation S320, at least one target image feature vector corresponding to the query information in the target database is determined according to at least one of the first similarity and the second similarity.

[0118] The first similarity is the similarity between the query text feature vector and the plurality of image feature vectors of the plurality of images in the target database, and the second similarity is the similarity between the query image feature vector and the plurality of image feature vectors of the plurality of images in the target database.

[0119] The target database includes image feature vectors corresponding to a plurality of local contents in a document and including context information.

[0120] Exemplarily, after operation S320, the query information can also be fed back and answered based on the local content corresponding to the target image feature vector.

[0121] Exemplarily, the query information can be of a text type or of an image type. For example, the user inputs the query information of the text type: What are the drugs for treating high blood pressure? The query information of the text type can be directly input in words, or can be converted from voice information. The user inputs the query information of the image type: uploaded CT image.

[0122] The query text feature vector can be a feature vector corresponding to the query information in a text space. The query image feature vector can be a feature vector corresponding to the query information in an image space. For the query information of the text type, the query text feature vector and the query image feature vector can be corresponded; for the query information of the image type, the query text feature vector and the query image feature vector can be corresponded.

[0123] It should be noted that the target database is used for information retrieval, and the target database stores image feature vectors corresponding to a plurality of local contents in different documents. For details, please refer to the above Figure 2The description of the construction of the target database will not be repeated.

[0124] The first similarity can be the similarity between the query text feature vector and the plurality of image feature vectors in the target database. That is, the similarity between the query text feature vector and the n image feature vectors in the target database is calculated respectively, and n first similarities are obtained.

[0125] The second similarity can be the similarity between the query image feature vector and the plurality of image feature vectors in the target database. That is, the similarity between the query image feature vector and the n image feature vectors in the target database is calculated respectively, and n second similarities are obtained.

[0126] The target image feature vector can be the image feature vector corresponding to the similarity (the first similarity and / or the second similarity) between the image feature vector in the target database and the feature vector of the query information (the query text feature vector and / or the query image feature vector) that satisfies the target condition.

[0127] The target image feature vector corresponding to the query information can be determined according to the first similarity. The target image feature vector corresponding to the query information can also be determined according to the second similarity. The target image feature vector corresponding to the query information can also be determined according to the first similarity and the second similarity.

[0128] The target condition can be whether the similarity is greater than a second threshold. For example, in the case where the first similarity is greater than the second threshold, the image feature vector corresponding to the first similarity is determined as the target image feature vector. The target condition can also be that after sorting the plurality of similarities from large to small, the target feature vector located in the first m positions, where m is less than n. For example, the n first similarities are sorted in a manner from large to small, and the image feature vectors located in the first m positions are determined as the target image feature vectors.

[0129] It can be understood that the context information between the local contents in the document is combined into the image feature vectors corresponding to the local contents for information retrieval, which provides more information for the image feature vectors, helps to understand the relationship between different image feature vectors, and identifies relevant image feature vectors, thereby significantly improving the accuracy, robustness and flexibility of information retrieval.

[0130] As described above, in the case where the target database includes the text feature vectors corresponding to the plurality of local contents in the document, the information retrieval method of the embodiment can further include: determining a third similarity between the query text feature vector and the plurality of text feature vectors in the target database; and determining at least one text feature vector corresponding to the query information according to the first similarity, the second similarity and the third similarity.

[0131] Exemplarily, the text feature vector can be a feature vector corresponding to a plurality of local contents in a document in a text space, and the text feature vector corresponds to the image feature vector of the local content.

[0132] The third similarity can be a similarity degree between the query text feature vector and a plurality of text feature vectors in the target database respectively. That is, the similarity degrees between the query text feature vector and n text feature vectors in the target database are calculated respectively, and n third similarities are obtained.

[0133] In one example, the text feature vector corresponding to the query information can be determined according to the first similarity, the second similarity and the third similarity at the same time.

[0134] It should be noted here that when the text feature vector is added to the calculation, since the text feature vector and the image feature vector are in a one-to-one correspondence, the text feature vector can be directly determined according to the similarity.

[0135] It can be understood that the target image feature vector corresponding to the query information is determined through multi-dimensional similarity, which can improve the accuracy of the information retrieval result.

[0136] In other embodiments, at least one text feature vector corresponding to the query information can be determined according to the first similarity and the third similarity. Alternatively, at least one text feature vector corresponding to the query information can be determined according to the second similarity and the third similarity.

[0137] Figure 7A An illustrative diagram of forming a query text feature vector and a query image feature vector according to the query information of the embodiment of the present disclosure is shown in the case where the query information is query text; Figure 7B An illustrative diagram of forming a query text feature vector and a query image feature vector according to the query information of the embodiment of the present disclosure is shown in the case where the query information is query image; Figure 7C An illustrative diagram of forming a query image feature vector according to the query image of the embodiment of the present disclosure is shown.

[0138] As described above, in the case where the query information is query text, the query text feature vector corresponding to the query information is obtained in operation S310. In one implementable manner, the operation can further include: embedding and encoding the query text based on a large language model to obtain an intermediate text feature; and mapping the intermediate text feature to the query text feature vector in the target dimensional space.

[0139] In one example, referring to Figure 7AThe query information (query text) of the text type is input into the large language model, and the large language model performs embedding coding on the query text to obtain intermediate text features. The large language model can convert the text into a vector while capturing the semantic information of the query text. The intermediate text features are input into the second mapping layer to obtain the query text feature vector. The second mapping layer maps the intermediate text features from the text space vector to the image space vector corresponding to the retrieval task of the target database, so as to facilitate subsequent calculation of the similarity between the query text feature vector and the plurality of image feature vectors in the target database.

[0140] As described above, in the case where the query information is the query text, the query image feature vector corresponding to the query information is obtained in operation S310. In an implementable manner, the operation can further include: writing the query text into a blank image to obtain a query image; and determining the query image feature vector according to the query image.

[0141] In one example, continuing to refer to Figure 7A , a blank image is created, a text box is created in the blank image, the textual content of the query text is pasted in the text box to obtain a query image. The query image is encoded to obtain the query image feature vector.

[0142] As described above, in the case where the query information is the query image, the query text feature vector corresponding to the query information is obtained in operation S310. In an implementable manner, the operation can further include: extracting the textual information in the query image; embedding and coding the textual information based on the large language model to obtain intermediate text features; and mapping the intermediate text features into the query text feature vector of the target dimension space.

[0143] In one example, referring to Figure 7B , the textual information in the query information (query image) of the image type is extracted, the textual information is input into the large language model, and the large language model performs embedding coding on the textual information to obtain intermediate text features. The large language model can convert the textual information into a vector while capturing the semantic information of the textual information. The intermediate text features are input into the second mapping layer to obtain the query text feature vector. The second mapping layer maps the intermediate text features from the text space vector to the image space vector corresponding to the retrieval task of the target database, so as to facilitate subsequent calculation of the similarity between the query text feature vector and the plurality of image feature vectors in the target database.

[0144] As described above, in the case where the query information is the query image, the query image feature vector corresponding to the query information is obtained in operation S310. In an implementable manner, the operation can further include: determining the query image feature vector according to the query image.

[0145] In one example, continuing to refer toFigure 7B The query information of the image type (query image) is encoded to obtain a query image feature vector.

[0146] It can be understood that different processing methods can be used for different types of query information to convert the query information into a query text feature vector and a query image feature vector.

[0147] As described above, the query image feature vector is determined according to the query image. In an implementable manner, the operation can further include: extracting a third image feature of the query image; mapping the third image feature into a third text feature; embedding and encoding the third text feature based on the large language model to obtain a fourth text feature; mapping the fourth text feature into a fourth image feature in a target dimension space corresponding to the task requirement of the target database to obtain the query image feature vector.

[0148] In one example, with reference to Figure 7C The query image is input into a visual encoder to obtain a third image feature. The visual encoder converts the original image data (pixels) of the query image into a structured feature representation (global feature vector), that is, maps the image from the "pixel space" to the "feature space". The third image feature is input into a first mapping layer to obtain a third text feature. The first mapping layer maps the third image feature from an image space vector to a text space vector to realize image-text alignment.

[0149] The third text feature is simultaneously input into a large language model for embedding and encoding to obtain a fourth text feature. The large language model can capture the semantic information between text features. The fourth text feature is input into a second mapping layer to obtain a fourth image feature, that is, an image feature vector corresponding to the query image. The second mapping layer maps the fourth text feature from a text space vector to an image space vector corresponding to the retrieval task of the target database.

[0150] It can be understood that the large language model embedding and encoding of the third text feature can capture the implicit information in the third text feature to obtain more content-rich query information, improving the accuracy and comprehensiveness of the retrieval result.

[0151] It should be noted that Figure 7C The scheme shown is applicable to Figure 7A and Figure 7B are applicable.

[0152] Further, in order to facilitate the understanding of the above operation S320: determining at least one target image feature vector corresponding to the query information in the target database according to one of the first similarity and the second similarity, the following will be further described with the query information as query text, combined with Figure 8A further description.

[0153] Figure 8A Fig. 1 shows a schematic diagram of a process of determining a target similarity between a query text feature vector and a query image feature vector corresponding to query information and image feature vectors in a target database according to an embodiment of the present disclosure.

[0154] For example, according to the first similarity and the second similarity, at least one target image feature vector corresponding to the query information in the target database is determined.

[0155] According to the first similarity and the second similarity, at least one target image feature vector corresponding to the query information in the target database is determined, including: determining a first intermediate value according to a first weight value and the first similarity, the first weight value representing an importance degree of the query text feature vector to the query information; determining a second intermediate value according to a second weight value and the second similarity, the second weight value representing an importance degree of the query image feature vector to the query information; determining a target similarity between the plurality of image feature vectors and the query information according to the first intermediate value and the second intermediate value; and determining the image feature vector corresponding to the target similarity as the target image feature vector in the case that the target similarity is greater than a second threshold value.

[0156] Referring to Figure 8A After obtaining the query text feature vector and the query image feature vector corresponding to the query information respectively, a first similarity between the query text feature vector and image feature vectors of a plurality of local contents of different documents stored in a target database is calculated, wherein n image feature vectors are stored in the target database. For example, a similarity between the query text feature vector and the image feature vector 1 is calculated to obtain a first similarity 1 (80%); a similarity between the query text feature vector and the image feature vector 2 is calculated to obtain a first similarity 2 (60%); and a similarity between the query text feature vector and the image feature vector n is calculated to obtain a first similarity n (30%).

[0157] At the same time, a second similarity between the query image feature vector and the image feature vectors of the plurality of local contents of different documents stored in the target database is calculated, wherein n image feature vectors are stored in the target database. For example, a similarity between the query image feature vector and the image feature vector 1 is calculated to obtain a second similarity 1 (70%); a similarity between the query image feature vector and the image feature vector 2 is calculated to obtain a second similarity 2 (60%); and a similarity between the query image feature vector and the image feature vector n is calculated to obtain a second similarity n (40%).

[0158] The first weight value (0.3) corresponding to the influence degree of the query text feature vector on the query information and the second weight value (0.7) corresponding to the influence degree of the query image feature vector on the query information are obtained. According to the first weight value and the first similarity 1, the first intermediate value 1 (24%) of the query text feature vector and the image feature vector 1 is calculated; according to the first weight value and the first similarity 2, the first intermediate value 2 (18%) of the query text feature vector and the image feature vector 1 is calculated; and according to the first weight value and the first similarity n, the first intermediate value n (9%) of the query text feature vector and the image feature vector n is calculated.

[0159] According to the second weight value and the second similarity 1, the second intermediate value 1 (49%) of the query image feature vector and the image feature vector 1 is calculated; according to the second weight value and the second similarity 2, the second intermediate value 2 (42%) of the query image feature vector and the image feature vector 1 is calculated; and according to the second weight value and the second similarity n, the second intermediate value n (28%) between the query image feature vector and the image feature vector n is calculated.

[0160] The first intermediate value 1 and the second intermediate value 2 of the image feature vector 1 are added to obtain the target similarity 1 of the image feature vector 1 and the query information, which is 73%. The first intermediate value 1 and the second intermediate value 2 of the image feature vector 2 are added to obtain the target similarity 2 of the image feature vector 2 and the query information, which is 60%. The first intermediate value n and the second intermediate value n of the image feature vector n are added to obtain the target similarity n of the image feature vector n and the query information, which is 37%.

[0161] The second threshold value is 50%. The target similarity 1 (73%) is greater than the second threshold value, and the image feature vector 1 is determined as the target feature vector corresponding to the query information. The target similarity 2 (60%) is greater than the second threshold value, and the image feature vector 2 is determined as the target feature vector corresponding to the query information. The target similarity n (37%) is less than the second threshold value, and the image feature vector n is not the target feature vector corresponding to the query information.

[0162] The size of the second threshold value in the embodiments of the present disclosure is not specifically limited, and the size of the second threshold value can be adjusted according to actual application.

[0163] In order to facilitate the understanding of the above operation: determining at least one text feature vector corresponding to the query information according to the first similarity, the second similarity and the third similarity, the query information is taken as a query text, and the operation is further described in combination with the following. Figure 8B .

[0164] Figure 8BFig. 2 is a schematic diagram illustrating a process of determining a target similarity between a query text feature vector and a query image feature vector corresponding to query information and an image feature vector in a target database according to an embodiment of the present disclosure.

[0165] For example, the target database is determined according to the first similarity, the second similarity and the third similarity.

[0166] On the basis of the above Figure 8A After obtaining the query text feature vector and the query image feature vector corresponding to the query information, the third similarity between the query text feature vector and the image feature vectors of different local contents stored in the target database is calculated, where the target database stores n text feature vectors. For example, the similarity between the query text feature vector and the text feature vector 1 is calculated to obtain the third similarity 1 (80%); the similarity between the query text feature vector and the text feature vector 2 is calculated to obtain the third similarity 2 (60%); and the similarity between the query text feature vector and the text feature vector n is calculated to obtain the third similarity n (30%).

[0167] The first weight value (0.2) corresponding to the influence degree of the query text feature vector (on the image feature vector) on the query information, the third weight value (0.2) corresponding to the influence degree of the query text feature vector (on the text feature vector) on the query information, and the second weight value (0.6) corresponding to the influence degree of the query image feature vector on the query information are obtained. According to the third weight value and the third similarity 1, the third intermediate value 1 (16%) between the query text feature vector and the text feature vector 1 is calculated; according to the third weight value and the third similarity 2, the third intermediate value 2 (12%) between the query text feature vector and the text feature vector 1 is calculated; and according to the third weight value and the third similarity n, the third intermediate value n (6%) between the query text feature vector and the text feature vector n is calculated.

[0168] The first intermediate values 1-n (16%, 12%, …, 6%) are calculated according to the new first weight value (0.2). The second intermediate values 1-n (42%, 36%, …, 24%) are calculated according to the new second weight value (0.6).

[0169] The first intermediate value 1, the second intermediate value 2 and the third intermediate value of the image feature vector 1 are added to obtain a target similarity 1 between the image feature vector 1 and the query information, which is 74%.

[0170] The second threshold is 50%. The target similarity 1 (74%) is greater than the second threshold, and the text feature vector 1 (the image feature vector 1) is determined as the target feature vector corresponding to the query information. The target similarity 2 (60%) is greater than the second threshold, and the text feature vector 2 (the image feature vector 2) is determined as the target feature vector corresponding to the query information. … The target similarity n (36%) is less than the second threshold, and the text feature vector n (the image feature vector n) is not the target feature vector corresponding to the query information.

[0171] In the case that the target database stores the image feature vector and the text feature vector corresponding to the image feature vector, the target text feature vector can be directly determined according to the finally calculated target similarity, which is used for generating the subsequent query result.

[0172] In other embodiments, the target database can store the image feature vector and the local content corresponding to the image feature vector. In information retrieval, if it is determined that the image feature vector 1 is the target feature vector, the corresponding local content 1 can be found according to the image feature vector 1, and the local content 1 is taken as the return result of information retrieval.

[0173] Based on the above database construction method, the present disclosure further provides a database construction device. The following will be described in detail in combination with Figure 9A The device is described in detail.

[0174] Figure 9A The structure block diagram of the database construction device according to the embodiment of the present disclosure is schematically shown.

[0175] As Figure 9A shown, the database construction device 400 of the embodiment includes a first acquisition module 410, a division module 420, a processing module 430 and a construction module 430.

[0176] The first acquisition module 410 is configured to acquire a plurality of images corresponding to a plurality of local contents in a to-be-processed document. In an embodiment, the first acquisition module 410 can be configured to perform the operation S210 described above, and details are not described herein.

[0177] The dividing module 420 is configured to divide the plurality of images into at least one image group. In an embodiment, the dividing module 420 can be configured to perform the operation S220 described above, and details are not repeated here.

[0178] The processing module 430 is configured to jointly process the plurality of images in any image group to obtain a plurality of image feature vectors corresponding to the plurality of images in the image group and including context information. In an embodiment, the processing module 430 can be configured to perform the operation S230 described above, and details are not repeated here.

[0179] The constructing module 440 is configured to construct a target database based on the plurality of image feature vectors. In an embodiment, the constructing module 440 can be configured to perform the operation S240 described above, and details are not repeated here.

[0180] In some embodiments, the dividing module includes a dividing sub-module.

[0181] The dividing sub-module is configured to divide the plurality of images into at least one image group according to the related information of the plurality of local contents; and wherein the context information is determined based on at least the related information of the plurality of images in the image group.

[0182] In some embodiments, the processing module includes a first obtaining sub-module and a first encoding sub-module.

[0183] The first obtaining sub-module is configured to obtain a plurality of text features corresponding to the plurality of images in any image group; and the first encoding sub-module is configured to embed and encode the plurality of text features based on a large language model to obtain a plurality of image feature vectors including context information.

[0184] In some embodiments, the first obtaining sub-module includes an obtaining unit and a first mapping unit.

[0185] The obtaining unit is configured to obtain first image features corresponding to the plurality of images in any image group; and the first mapping unit is configured to map the first image features to first text features.

[0186] In some embodiments, the first encoding sub-module includes a first encoding unit and a second mapping unit.

[0187] The first encoding unit is configured to embed and encode the plurality of first text features based on a large language model to obtain a plurality of second text features including context information; and the second mapping unit is configured to map the plurality of second text features to second image features in a target dimensional space to obtain a plurality of image feature vectors, the target dimensional space corresponding to a task requirement corresponding to the target database.

[0188] In some embodiments, the first obtaining module includes a first forming sub-module.

[0189] The first forming sub-module is configured to form an image corresponding to each of the local contents according to the local contents in the document to be processed.

[0190] In some embodiments, the first obtaining module comprises a second obtaining sub-module and a second forming sub-module.

[0191] The second obtaining sub-module is configured to obtain related information of the local contents in the document to be processed, and the related information comprises at least one of the following: abstract information, theme information, title information, semantic information, keywords or visual features; and the second forming sub-module is configured to form an image corresponding to each of the local contents according to the related information.

[0192] In some embodiments, the dividing sub-module comprises a first determining unit and a dividing unit.

[0193] The first determining unit is configured to determine the correlation degree between the local contents according to the related information of the local contents, and the related information comprises at least one of the following: abstract information, theme information, title information, semantic information, keywords or visual features; and the dividing unit is configured to divide the image corresponding to the first local content and the image corresponding to the second local content into one image group in the case that the correlation degree between the first local content and the second local content is greater than a first threshold.

[0194] Based on the above information retrieval method, the present disclosure further provides an information retrieval device. The following will be described in detail Figure 9B The device.

[0195] Figure 9B The structure block diagram of the information retrieval device according to the embodiment of the present disclosure is schematically shown.

[0196] As Figure 9B shown, the information retrieval device 500 of the embodiment comprises a second obtaining module 510 and a first determining module 520.

[0197] The second obtaining module 510 is configured to obtain at least one of the following: a query text feature vector corresponding to the query information and a query image feature vector. In an embodiment, the second obtaining module 510 can be configured to perform the operation S310 described above, and details are not repeated here.

[0198] The first determining module 520 is configured to determine at least one target image feature vector corresponding to the query information in the target database according to at least one of the first similarity and the second similarity. In an embodiment, the first determining module 520 can be configured to perform the operation S320 described above, and details are not repeated here.

[0199] The first similarity is a similarity between the query text feature vector and a plurality of image feature vectors of a plurality of images in the target database, and the second similarity is a similarity between the query image feature vector and the plurality of image feature vectors of the plurality of images in the target database. The target database includes image feature vectors corresponding to a plurality of local contents in the document and including context information.

[0200] In some embodiments, when the target database includes text feature vectors corresponding to a plurality of local contents in the document, the information retrieval device of the embodiment further includes a second determination module.

[0201] The second determination module is configured to determine a third similarity between the query text feature vector and a plurality of text feature vectors in the target database, wherein the text feature vectors correspond to the image feature vectors. The first determination module is further configured to determine at least one text feature vector corresponding to the query information according to the first similarity, the second similarity, and the third similarity.

[0202] In some embodiments, when the query information is a query text, the second acquisition module includes a second encoding submodule and a first mapping submodule.

[0203] The second encoding submodule is configured to embed and encode the query text based on the large language model to obtain an intermediate text feature. The first mapping submodule is configured to map the intermediate text feature to a query text feature vector in a target dimensional space.

[0204] The second acquisition module further includes a writing submodule and a first determination submodule.

[0205] The writing submodule is configured to write the query text into a blank image to obtain a query image. The first determination submodule is configured to determine a query image feature vector according to the query image.

[0206] In some embodiments, when the query information is a query image, the second acquisition module includes an extraction submodule, a third encoding submodule, and a second mapping submodule.

[0207] The extraction submodule is configured to extract text information in the query image. The third encoding submodule is configured to embed and encode the text information based on the large language model to obtain an intermediate text feature. The second mapping submodule is configured to map the intermediate text feature to a query text feature vector in a target dimensional space.

[0208] The second acquisition module further includes a second determination submodule.

[0209] The second determination submodule is configured to determine a query image feature vector according to the query image.

[0210] In some embodiments, the determining submodule comprises an extraction unit, a third mapping unit, a second encoding unit, and a fourth mapping unit.

[0211] The extraction unit is configured to extract a third image feature of the query image; the third mapping unit is configured to map the third image feature into a third text feature; the second encoding unit is configured to perform embedding coding on the third text feature based on a large language model to obtain a fourth text feature; and the fourth mapping unit is configured to map the fourth text feature into a fourth image feature in a target dimensional space corresponding to a task requirement of the target database, to obtain a query image feature vector.

[0212] In some embodiments, the first determining module comprises a second determining unit, a third determining unit, a fourth determining unit, and a fifth determining unit.

[0213] The second determining unit is configured to determine a first intermediate value according to the first weight value and the first similarity, the first weight value representing an importance degree of the query text feature vector to the query information; the third determining unit is configured to determine a second intermediate value according to the second weight value and the second similarity, the second weight value representing an importance degree of the query image feature vector to the query information; the fourth determining unit is configured to determine a target similarity between the plurality of image feature vectors and the query information according to the first intermediate value and the second intermediate value; and the fifth determining unit is configured to determine, in a case where the target similarity is greater than a second threshold value, an image feature vector corresponding to the target similarity as the target image feature vector.

[0214] According to an embodiment of the present disclosure, any of the first obtaining module 410, the dividing module 420, the processing module 430 and the constructing module 430 (the second obtaining module 510 and the first determining module 520) can be combined in one module, or any of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of other modules, and implemented in one module. According to an embodiment of the present disclosure, at least one of the first obtaining module 410, the dividing module 420, the processing module 430 and the constructing module 430 (the second obtaining module 510 and the first determining module 520) can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware that can be integrated or packaged with a circuit, or implemented in any one of software, hardware and firmware or in a proper combination of any of them. Alternatively, at least one of the first obtaining module 410, the dividing module 420, the processing module 430 and the constructing module 430 (the second obtaining module 510 and the first determining module 520) can be at least partially implemented as a computer program module that can perform corresponding functions when it is run.

[0215] Figure 10 A block diagram of an electronic device suitable for implementing the database construction method and the information retrieval method according to an embodiment of the present disclosure is schematically shown.

[0216] As shown in Figure 10 The electronic device 600 according to an embodiment of the present disclosure includes a processor 601 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 can include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset, and / or a special-purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), and the like. The processor 601 can also include an on-board memory for cache use. The processor 601 can include a single processing unit or multiple processing units for performing different actions of the method processes according to embodiments of the present disclosure.

[0217] In the RAM 603, various programs and data required for the operation of the electronic device 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via the bus 604. The processor 601 performs various operations of the method flow according to the embodiments of the present disclosure by executing the programs in the ROM 602 and / or the RAM 603. It should be noted that the programs can also be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 can also perform various operations of the method flow according to the embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0218] According to the embodiments of the present disclosure, the electronic device 600 can further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 can further include one or more of the following components connected to the I / O interface 605: an input part 606 including a keyboard, a mouse, and the like; an output part 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage part 608 including a hard disk, and the like; and a communication part 609 including a network interface card such as a LAN card, a modem, and the like. The communication part 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as necessary. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted on the drive 610 as necessary, so that a computer program read therefrom is installed in the storage part 608 as necessary.

[0219] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, when the one or more programs are executed, the method according to the embodiments of the present disclosure is implemented.

[0220] According to an embodiment of the present disclosure, the computer readable storage medium can be a nonvolatile computer readable storage medium, for example, can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer readable storage medium can include the ROM 602 and / or the RAM 603 described above and / or one or more memory other than the ROM 602 and the RAM 603.

[0221] Embodiments of the present disclosure also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the above-mentioned methods provided by the embodiments of the present disclosure.

[0222] The above-mentioned functions defined in the system / device of the embodiments of the present disclosure are performed when the computer program is executed by the processor 601. According to an embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by computer program modules.

[0223] In one embodiment, the computer program can rely on tangible storage media such as optical storage media, magnetic storage media, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of signals on network media and installed and downloaded through the communication part 609 and / or installed from the detachable medium 611. The program codes contained in the computer program can be transmitted by any appropriate network media, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0224] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609 and / or installed from the detachable medium 611. When the computer program is executed by the processor 601, the above-mentioned functions defined in the system of the embodiments of the present disclosure are performed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0225] According to embodiments of the present disclosure, program code of the computer programs provided by embodiments of the present disclosure can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. Programming languages include, but are not limited to, such as Java, C++, python, “C” language or similar programming languages. Program code can be executed entirely on a user computing device, partially on a user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected to the Internet through an Internet service provider).

[0226] The flow diagrams and the block diagrams in the drawings are illustrations of possible architectures, functions, and operations for systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0227] Those skilled in the art can understand that the features described in various embodiments of the present disclosure and / or claims can be combined or / and integrated, even if such combinations or integrations are not explicitly described in the present disclosure. In particular, the features described in various embodiments of the present disclosure and / or claims can be combined and / or integrated in various combinations, without departing from the spirit and teachings of the present disclosure. All such combinations and / or integrations are within the scope of the present disclosure.

[0228] The above describes embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims

1. A database construction method, comprising: obtaining a plurality of images corresponding to a plurality of local contents in a to-be-processed document; dividing the plurality of images into at least one image group; jointly processing the plurality of images in any of the image groups to obtain a plurality of image feature vectors corresponding to the plurality of images in the image group, the plurality of image feature vectors comprising context information; constructing a target database based on the plurality of image feature vectors.

2. The method of claim 1, wherein the dividing the plurality of images into at least one image group comprises: dividing the plurality of images into at least one image group according to related information of the plurality of local contents; wherein the context information is determined based on at least the related information of the plurality of images in the image group.

3. The method of claim 1, wherein the jointly processing the plurality of images in any of the image groups to obtain a plurality of image feature vectors corresponding to the plurality of images in the image group, the plurality of image feature vectors comprising context information comprises: obtaining a plurality of text features corresponding to the plurality of images in any of the image groups; embedding and encoding the plurality of text features based on a large language model to obtain a plurality of image feature vectors corresponding to the plurality of text features, the plurality of image feature vectors comprising context information.

4. The method of claim 3, wherein the obtaining a plurality of text features corresponding to the plurality of images in any of the image groups comprises: obtaining first image features corresponding to the plurality of images in any of the image groups; mapping the first image features into first text features; wherein the embedding and encoding the plurality of text features based on a large language model to obtain a plurality of image feature vectors corresponding to the plurality of text features, the plurality of image feature vectors comprising context information comprises: embedding and encoding the plurality of first text features based on a large language model to obtain a plurality of second text features corresponding to the plurality of first text features, the plurality of second text features comprising context information; mapping the plurality of second text features into second image features in a target dimensional space to obtain a plurality of image feature vectors, the target dimensional space corresponding to a task requirement corresponding to the target database.

5. The method of claim 1, wherein the obtaining a plurality of images corresponding to a plurality of local contents in a to-be-processed document comprises: forming an image corresponding to each local content according to the plurality of local contents in the to-be-processed document; or obtaining related information of the plurality of local contents in the to-be-processed document, the related information comprising at least one of the following: summary information, theme information, title information, semantic information, keywords, or visual features; forming an image corresponding to each local content according to the related information.

6. An information retrieval method, comprising: obtaining at least one of a query text feature vector and a query image feature vector corresponding to query information; determining at least one target image feature vector corresponding to the query information in a target database based on at least one of a first similarity and a second similarity; wherein the first similarity is a similarity degree between the query text feature vector and a plurality of image feature vectors of a plurality of images in the target database, and the second similarity is a similarity degree between the query image feature vector and the plurality of image feature vectors of the plurality of images in the target database. ​ The target database includes image feature vectors corresponding to a plurality of local contents in a document, the image feature vectors including context information.

7. The method of claim 6, in the case where the target database includes text feature vectors corresponding to a plurality of local contents in the document, the method further comprising: determining a third similarity between the query text feature vector and a plurality of text feature vectors in the target database, wherein the text feature vectors correspond to the image feature vectors; determining at least one text feature vector corresponding to the query information according to the first similarity, the second similarity, and the third similarity.

8. The method of claim 7, in the case where the query information is a query text, the obtaining of the query text feature vector corresponding to the query information comprises: embedding and encoding the query text based on a large language model to obtain intermediate text features; mapping the intermediate text features to a query text feature vector in a target dimensional space; the obtaining of the query image feature vector corresponding to the query information comprises: writing the query text into a blank image to obtain a query image; determining a query image feature vector according to the query image; or, in the case where the query information is a query image, the obtaining of the query text feature vector corresponding to the query information comprises: extracting text information in the query image; embedding and encoding the text information based on a large language model to obtain intermediate text features; mapping the intermediate text features to a query text feature vector in a target dimensional space; the obtaining of the query image feature vector corresponding to the query information comprises: determining a query image feature vector according to the query image; wherein the target dimensional space corresponds to a task requirement corresponding to the target database.

9. The method of claim 8, the determining of the query image feature vector according to the query image comprises: extracting a third image feature of the query image; mapping the third image feature to a third text feature; embedding and encoding the third text feature based on a large language model to obtain a fourth text feature; mapping the fourth text feature to a fourth image feature in a target dimensional space to obtain the query image feature vector, the target dimensional space corresponding to a task requirement corresponding to the target database.

10. An electronic device comprising: a memory for storing one or more computer programs; at least one processor for executing the one or more computer programs to implement the steps of the method of any one of claims 1-5 or 6-9.

Images