Conversational image retrieval method and device based on large language model

Abstract

The present invention provides a conversational image retrieval method and device based on a large language model, belonging to the field of artificial intelligence technology, wherein the method comprises: obtaining multimodal conversation information input by a user, the multimodal conversation information comprising images and texts; inputting the multimodal conversation information into a pre-built large language model, obtaining answer text and / or image retrieval results corresponding to the multimodal conversation information output by the large language model; wherein the large language model is trained based on sample multimodal conversation information, and answer text labels and / or image retrieval result labels corresponding to the sample multimodal conversation information, and the sample multimodal conversation information comprises sample images and sample texts. The conversational image retrieval method and device based on a large language model provided by the present invention are applicable to a variety of application scenarios, thus expanding the scope of application.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a conversational image retrieval method and device based on a large language model. Background Art

[0002] Image retrieval is a task that focuses on searching for desired images that correspond to an abstract concept formed in the user's mind, and the user needs to convey this concept in some way through human-computer interaction. Traditional content-based image retrieval tasks use images as queries to identify the desired images, which has certain limitations and a limited scope of application. Summary of the invention

[0003] The present invention provides a conversational image retrieval method and device based on a large language model, which is used to solve the defect that the image retrieval task in the prior art uses an image as a query to identify the required image and has a limited application scope.

[0004] In a first aspect, the present invention provides a conversational image retrieval method based on a large language model, comprising:

[0005] Acquire multimodal dialogue information input by a user, wherein the multimodal dialogue information includes an image and a text;

[0006] Inputting the multimodal dialogue information into a pre-built large language model to obtain answer text and / or image retrieval results corresponding to the multimodal dialogue information output by the large language model;

[0007] The large language model is trained based on sample multimodal dialogue information and answer text labels and / or image retrieval result labels corresponding to the sample multimodal dialogue information, and the sample multimodal dialogue information includes sample images and sample texts.

[0008] In some embodiments, the large language model includes a visual feature encoding module, a perceptual compression module, a large language model decoder module, and an image retrieval module;

[0009] Correspondingly, the step of inputting the multimodal dialogue information into a pre-built large language model to obtain an answer text and / or image retrieval result corresponding to the multimodal dialogue information output by the large language model includes:

[0010] Inputting the image into the visual feature encoding module to obtain image features of the image output by the visual feature encoding module;

[0011] Inputting the image features into the perceptual compression module to obtain image compression features output by the perceptual compression module;

[0012] Input the image compression features and the text into the large language model decoder module to obtain the answer text output by the large language model decoder module, and / or input the image compression features and the text into the large language model decoder module to obtain the query information output by the large language model decoder module, input the query information into the image retrieval module to obtain the image retrieval result output by the image retrieval module.

[0013] In some embodiments, inputting the query information into the image retrieval module to obtain the image retrieval result output by the image retrieval module includes:

[0014] Based on the image retrieval module, the query information is compared with a plurality of conversation contexts in a pre-built index to determine a target conversation context that is most similar to the query information;

[0015] Based on the target conversation context and the index, determining an image corresponding to the target conversation context from a pre-built external image library to obtain an image retrieval result;

[0016] The index includes the multiple conversation contexts and mapping relationships between the multiple conversation contexts and the multiple images in the external image library;

[0017] Among them, the types of the conversation context include multimodal conversation context and plaintext conversation context, the multimodal conversation context includes a first multimodal conversation context, a second multimodal conversation context and a third multimodal conversation context, the first multimodal conversation context is constructed based on a reference image, the second multimodal conversation context is constructed based on a reference text, and the third multimodal conversation context is constructed after context fusion of the first multimodal conversation context and the second multimodal conversation context.

[0018] In some embodiments, the process of constructing the first multimodal conversation context includes:

[0019] Acquire a first image-text pair, wherein the first image-text pair includes a reference image and a reference image description;

[0020] Acquire reference image prompt information, wherein the reference image prompt information includes task prompt information for constructing a multimodal dialogue based on the reference image;

[0021] Based on the reference image prompt information, inputting the multimodal query information into an image retriever to obtain a first target image corresponding to the multimodal query information output by the image retriever, wherein the multimodal query information includes the reference image and an initial instruction text;

[0022] Inputting the first target image into an image captioning device to obtain a first target image description output by the image captioning device;

[0023] Inputting the reference image description and the first target image description into a first text generator to obtain a modified instruction text output by the first text generator;

[0024] The first multimodal dialogue context is constructed based on the reference image, the modified instruction text and the first target image.

[0025] In some embodiments, the process of constructing the second multimodal conversation context includes:

[0026] Acquire a second image-text pair, wherein the image-text pair includes a second target image and an original description of the second target image;

[0027] Acquiring reference text prompt information, wherein the reference text prompt information includes task prompt information for constructing a multimodal dialogue based on the reference text;

[0028] Based on the reference text prompt information, inputting the original description of the second target image into a second text generator to obtain two rounds of dialogue output by the second text generator;

[0029] Based on the two rounds of dialogue, obtaining the reference text;

[0030] Based on the reference text, retrieving the most relevant image corresponding to the reference text from the external image library;

[0031] The second multimodal dialogue context is constructed based on the two rounds of dialogue, the reference text and the most relevant image.

[0032] In some embodiments, the process of constructing the plaintext conversation context includes:

[0033] Acquire a third image-text pair, wherein the image-text pair includes a third target image and an original description of the third target image;

[0034] Obtaining plaintext dialogue prompt information, wherein the plaintext dialogue prompt information includes task prompt information for constructing a plaintext dialogue;

[0035] Based on the reference text prompt information, inputting the original description of the third target image into a third text generator to obtain multiple rounds of simulated query information output by the third text generator;

[0036] Based on the multiple rounds of simulated query information, a plaintext conversation context is constructed.

[0037] In some embodiments, the image retrieval result includes a plurality of candidate images, and the method further includes:

[0038] A selection instruction of a user is received, and in response to the selection instruction, a target candidate image is determined from the multiple candidate images, and the target candidate image is appended to the end of the historical multimodal dialogue sequence.

[0039] In some embodiments, the process of determining the large language model includes:

[0040] Acquiring sample multimodal dialogue information, wherein the sample multimodal dialogue information includes a sample image and a sample text;

[0041] Determining an answer text label and / or an image retrieval result label corresponding to the sample multimodal dialogue information;

[0042] Using the sample multimodal dialogue information as training samples and answer text labels and / or image retrieval result labels corresponding to the sample multimodal dialogue information as sample labels, training an initial large language model;

[0043] After the initial large language model training is completed, the large language model is obtained.

[0044] In some embodiments, the initial large language model includes an initial visual feature encoding module, an initial perceptual compression module, an initial large language model decoder module and an initial image retrieval module, and the training of the initial large language model includes:

[0045] Inputting the sample image into the initial visual feature encoding module to obtain the sample image features of the sample image output by the initial visual feature encoding module;

[0046] Inputting the sample image features into the initial perceptual compression module to obtain the sample image compression features output by the initial perceptual compression module;

[0047] Input the sample image compression feature and the sample text into the initial large language model decoder module to obtain the answer text prediction result output by the initial large language model decoder module, and / or input the sample image compression feature and the sample text into the initial large language model decoder module to obtain the sample query information output by the initial large language model decoder module, input the sample query information into the initial image retrieval module to obtain the image retrieval prediction result output by the initial image retrieval module;

[0048] Calculating a first loss function value based on the image retrieval prediction result and the image retrieval result label corresponding to the sample multimodal dialogue information;

[0049] Calculating a second loss function value based on the answer text prediction result and the answer text label corresponding to the sample multimodal dialogue information;

[0050] Based on the first loss function value and the second loss function value, the parameters of the initial large language model are updated.

[0051] In a second aspect, the present invention further provides a conversational image retrieval device based on a large language model, comprising:

[0052] An acquisition unit, configured to acquire multimodal dialogue information input by a user, wherein the multimodal dialogue information includes images and texts;

[0053] an output unit, configured to input the multimodal dialogue information into a pre-built large language model, and obtain an answer text and / or image retrieval result corresponding to the multimodal dialogue information output by the large language model;

[0054] The large language model is trained based on sample multimodal dialogue information and answer text labels and / or image retrieval result labels corresponding to the sample multimodal dialogue information, and the sample multimodal dialogue information includes sample images and sample texts.

[0055] The present invention provides a conversational image retrieval method and device based on a large language model. By acquiring multimodal conversation information input by a user, the multimodal conversation information includes images and texts, and inputting the multimodal conversation information into a pre-built large language model, the answer text and / or image retrieval results output by the large language model are obtained. The method and device are suitable for a variety of application scenarios and expand the scope of application. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 is a flow chart of a conversational image retrieval method based on a large language model provided by an embodiment of the present invention;

[0058] Figure 2 is a schematic diagram of a multimodal conversational search interaction interface provided by an embodiment of the present invention;

[0059] Figure 3 is an architecture diagram of a large language model provided by an embodiment of the present invention;

[0060] Figure 4is a flowchart of a process for constructing a multimodal conversation context provided by an embodiment of the present invention;

[0061] Figure 5 It is a flowchart of a process of constructing a plaintext conversation context provided by an embodiment of the present invention;

[0062] Figure 6 is a flowchart of a process for determining a large language model provided by an embodiment of the present invention;

[0063] Figure 7 is a flowchart of a training process of an initial large language model provided by an embodiment of the present invention;

[0064] Figure 8 It is a structural schematic diagram of the conversational image retrieval device based on a large language model provided by the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] The terms "first", "second", etc. in the specification of the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.

[0067] Figure 1 The flowchart of the conversational image retrieval method based on a large language model provided by an embodiment of the present invention is as follows. Figure 1 As shown, a conversational image retrieval method based on a large language model is provided, comprising the following steps: step 110 and step 120. The steps of the method flow are only a possible implementation of the present invention.

[0068] Step 110: Acquire multimodal dialogue information input by the user, where the multimodal dialogue information includes images and texts.

[0069] In addition to text information, multimodal dialogue information also includes information in various other forms, such as pictures, videos, voice, etc. In multimodal dialogue, different forms of information can complement and enrich each other, providing more comprehensive communication content. Multimodal dialogue provides a richer and more natural way for human-computer interaction, which can improve communication effects and user experience.

[0070] The text includes questions, instruction data and other texts.

[0071] It can be understood that by obtaining multimodal dialogue information input by the user, the image retrieval system can more accurately understand the user's intention and provide a more appropriate response.

[0072] Step 120: input the multimodal dialogue information into a pre-built large language model to obtain answer text and / or image retrieval results corresponding to the multimodal dialogue information output by the large language model;

[0073] The large language model is trained based on sample multimodal dialogue information and answer text labels and / or image retrieval result labels corresponding to the sample multimodal dialogue information, and the sample multimodal dialogue information includes sample images and sample texts.

[0074] It can be understood that by inputting multimodal dialogue information into a large language model and combining it with technologies such as text generation and image retrieval, answer text and image results that match the dialogue information can be obtained, providing a more comprehensive and accurate response.

[0075] In some embodiments, the image retrieval result includes a plurality of candidate images, and the method further includes:

[0076] A selection instruction from a user is received, and in response to the selection instruction, a target candidate image is determined from a plurality of candidate images, and the target candidate image is appended to the end of the historical multimodal dialogue sequence.

[0077] Optionally, the image retrieval result includes a plurality of candidate images sorted by priority.

[0078] Optionally, a user's selection instruction may be received through a multimodal conversational retrieval interaction interface, and in response to the selection instruction, a target candidate image may be determined from a plurality of candidate images, and the target candidate image may be appended to the end of the historical multimodal conversation sequence.

[0079] Figure 2 A schematic diagram of a multi-modal conversational search interaction interface provided by an embodiment of the present invention. Figure 2 As shown, in some embodiments, human-computer interaction can be performed through a multimodal conversational retrieval interaction interface. Based on the multimodal conversational retrieval interaction interface, multimodal conversation information can be obtained, and answer text and / or image retrieval results corresponding to the multimodal conversation information can be output.

[0080] For example, the multimodal information input by the user is “Who is this scientist?” and an image of a scientist, and the answer text is “This scientist is a man named Galileo Galilei, who is known for his contributions to…”.

[0081] It is understandable that the multimodal conversational search interface has the following advantages:

[0082] 1) Natural interaction: Users are able to interact with the system using their preferred communication style, resulting in a more natural and seamless retrieval process;

[0083] 2) Comprehensive context: Through multimodal dialogue, the system can understand and consider the overall context, thereby providing users with more relevant and accurate information;

[0084] 3) Interactive feedback: In multiple rounds of dialogue, the search results can be gradually refined based on user feedback;

[0085] 4) Multimodal experience: Both users and systems can express information through images, text, and other means to create an integrated multimodal interactive experience.

[0086] In an embodiment of the present invention, by obtaining multimodal dialogue information input by a user, inputting the multimodal dialogue information into a pre-built large language model, and obtaining answer text and / or image retrieval results output by the large language model, more accurate, rich and diverse response information can be provided in combination with information in different forms, which is suitable for a variety of application scenarios and expands the scope of application.

[0087] Figure 3 The architecture diagram of the large language model provided by the embodiment of the present invention is as follows: Figure 3 As shown, in some embodiments, the large language model includes a visual feature encoding module, a perceptual compression module, a large language model decoder module, and an image retrieval module;

[0088] Correspondingly, step 120 inputs the multimodal dialogue information into a pre-built large language model to obtain answer text and / or image retrieval results corresponding to the multimodal dialogue information output by the large language model, including:

[0089] Step 121, inputting the image into a visual feature encoding module, and obtaining image features of the image output by the visual feature encoding module;

[0090] Step 122: input the image features into the perceptual compression module to obtain the image compression features output by the perceptual compression module;

[0091] Step 123: input the image compression features and text into the large language model decoder module to obtain the answer text output by the large language model decoder module, and / or input the image compression features and text into the large language model decoder module to obtain the query information output by the large language model decoder module, input the query information into the image retrieval module to obtain the image retrieval result output by the image retrieval module.

[0092] The query information includes some keywords or phrases related to the user's question and the context information required to describe the user's question.

[0093] Optionally, the visual feature encoding module extracts key features in the image and generates visual encoding and global visual features for each input image, thereby converting each input image into a set of numerical visual feature representations.

[0094] Optionally, the perceptual compression module compresses image features through a dimensionality reduction algorithm, feature selection or other compression methods to reduce their dimensions and redundant information for subsequent processing and storage, which helps to improve the efficiency of large language models and save storage space.

[0095] Optionally, a perceptual compression module is adopted to compress the visual encoding into a fixed number of dense embeddings, and different linear projection layers are used to project the global visual features and the dense embeddings into the same dimension.

[0096] Optionally, the image compression features are marked, for example, by using IMG or / IMG to mark the image compression features to obtain marked image compression features; the marked image compression features are used as visual input of the large language model decoder module.

[0097] Optionally, the image retrieval module compares the query information with data in a pre-built external image library to obtain image retrieval results that match the query information.

[0098] In some embodiments, query information is input into an image retrieval module to obtain image retrieval results output by the image retrieval module, including:

[0099] Based on the image retrieval module, the query information is compared with multiple conversation contexts in the pre-built index to determine the target conversation context that is most similar to the query information;

[0100] Based on the target conversation context and index, the image corresponding to the target conversation context is determined from the pre-built external image library to obtain the image retrieval result;

[0101] The index includes a plurality of conversation contexts and a mapping relationship between the plurality of conversation contexts and a plurality of images in an external image library;

[0102] Among them, the types of conversation contexts include multimodal conversation contexts and plaintext conversation contexts. The multimodal conversation contexts include a first multimodal conversation context, a second multimodal conversation context and a third multimodal conversation context. The first multimodal conversation context is constructed based on a reference image, the second multimodal conversation context is constructed based on a reference text, and the third multimodal conversation context is constructed after context fusion of the first multimodal conversation context and the second multimodal conversation context.

[0103] Among them, the external image library is a database that stores a large number of images, each image is associated with one or more dialogue contexts.

[0104] Among them, the multimodal conversation context contains not only text information, but also information of other modalities (such as images, audio, video, etc.); the plaintext conversation context only contains text information and does not contain information of other modalities.

[0105] It should be noted that after the image retrieval module determines the target conversation context, it can quickly obtain images related to the target conversation context from the external image library according to the mapping relationship in the index, and these images will be returned to the user as image retrieval results.

[0106] Figure 4 A flowchart of the process of constructing a multimodal conversation context provided by an embodiment of the present invention. Figure 4 As shown, a first multimodal dialogue context is constructed based on a reference image, a second multimodal dialogue context is constructed based on a reference text, and a third multimodal dialogue context is constructed by context fusion based on the first multimodal dialogue context and the second multimodal dialogue context.

[0107] In some embodiments, the process of constructing the first multimodal conversation context includes:

[0108] Acquire a first image-text pair, the first image-text pair comprising a reference image and a reference image description;

[0109] Acquiring reference image prompt information, the reference image prompt information including task prompt information for constructing a multimodal dialogue based on the reference image;

[0110] Based on the reference image prompt information, the multimodal query information is input into the image retriever to obtain a first target image corresponding to the multimodal query information output by the image retriever, wherein the multimodal query information includes the reference image and the initial instruction text;

[0111] Inputting the first target image into the image captioner to obtain a first target image description output by the image captioner;

[0112] Inputting the reference image description and the first target image description into the first text generator to obtain a modified instruction text output by the first text generator;

[0113] A first multimodal dialogue context is constructed based on the reference image, the modified instruction text, and the first target image.

[0114] Among them, image description includes theme or scene description, object and character description, environment and background description, visual feature description, etc., for example: "a photo of a sunset on the beach."

[0115] Optionally, an image retriever is used to retrieve multiple similar images corresponding to the multimodal query information from an external image library, and the similarity between each similar image and the multimodal query information is calculated. The multiple similar images are screened according to the similarity between each similar image and the multimodal query information and a similarity threshold to obtain screened images, and one image is randomly selected from the screened images as the first target image.

[0116] In some embodiments, the process of constructing the second multimodal conversation context includes:

[0117] Acquire a second image-text pair, the image-text pair comprising a second target image and an original description of the second target image;

[0118] Acquiring reference text prompt information, the reference text prompt information including task prompt information for constructing a multimodal dialogue based on the reference text;

[0119] Based on the reference text prompt information, the original description of the second target image is input into the second text generator to obtain two rounds of dialogue output by the second text generator;

[0120] Based on two rounds of dialogue, a reference text is obtained;

[0121] Based on the reference text, the most relevant image corresponding to the reference text is retrieved from an external image library;

[0122] Based on the two-turn dialogue, reference text and the most relevant image, a second multimodal dialogue context is constructed.

[0123] It should be noted that the answer to the first round of dialogue is related to the corresponding reference image description, and the answer to the second round of dialogue is related to the second target image description. The corresponding reference image can be obtained based on the first round of dialogue, and the most relevant image can be obtained based on the second round of dialogue.

[0124] Figure 5 The following is a flow chart of the process of constructing a plain text conversation context provided by an embodiment of the present invention. Figure 5 As shown, in some embodiments, the process of constructing the plaintext conversation context includes:

[0125] Acquire a third image-text pair, where the image-text pair includes a third target image and an original description of the third target image;

[0126] Obtaining plaintext dialogue prompt information, the plaintext dialogue prompt information including task prompt information for constructing a plaintext dialogue;

[0127] Based on the reference text prompt information, the original description of the third target image is input into the third text generator to obtain multiple rounds of simulated query information output by the third text generator;

[0128] Based on multiple rounds of simulated query information, a plaintext conversation context is constructed.

[0129] Optionally, the constraint condition of the third text generator is that clues for image retrieval are implicit in multiple rounds of simulated query information.

[0130] It can be understood that by constructing multimodal conversation context and plaintext conversation context, the existing real-world image text retrieval dataset can be expanded, the image content can be enriched, and indexes can be established, thereby improving the efficiency and effectiveness of conversational image retrieval.

[0131] Figure 6 A flow chart of the process of determining a large language model provided by an embodiment of the present invention is shown as follows: Figure 6 As shown, in some embodiments, the process of determining the large language model includes the following steps:

[0132] Step 610: Obtain sample multimodal dialogue information, where the sample multimodal dialogue information includes a sample image and a sample text;

[0133] Step 620: Determine the answer text label and / or image retrieval result label corresponding to the sample multimodal dialogue information;

[0134] Step 630: Using the sample multimodal dialogue information as training samples and the answer text labels and / or image retrieval result labels corresponding to the sample multimodal dialogue information as sample labels, train an initial large language model;

[0135] Step 640: After the initial large language model training is completed, a large language model is obtained.

[0136] The sample multimodal dialogue information includes not only text information but also other forms of information, such as pictures, videos, and voice.

[0137] Optionally, after the initial large language model training is completed, parameter optimization iterations are performed on the initial large language model to obtain a large language model.

[0138] In some embodiments, the initial large language model includes an initial visual feature encoding module, an initial perceptual compression module, an initial large language model decoder module, and an initial image retrieval module, and training the initial large language model includes:

[0139] Inputting the sample image into the initial visual feature encoding module to obtain the sample image features of the sample image output by the initial visual feature encoding module;

[0140] Inputting the sample image features into the initial perceptual compression module to obtain the sample image compression features output by the initial perceptual compression module;

[0141] Inputting the sample image compression features and the sample text into the initial large language model decoder module to obtain the answer text prediction result output by the initial large language model decoder module, and / or inputting the sample image compression features and the sample text into the initial large language model decoder module to obtain the sample query information output by the initial large language model decoder module, and inputting the sample query information into the initial image retrieval module to obtain the image retrieval prediction result output by the initial image retrieval module;

[0142] Calculate a first loss function value based on the image retrieval prediction result and the image retrieval result label corresponding to the sample multimodal dialogue information;

[0143] Calculate the second loss function value based on the answer text prediction result and the answer text label corresponding to the sample multimodal dialogue information;

[0144] Based on the first loss function value and the second loss function value, the parameters of the initial large language model are updated.

[0145] It can be understood that by calculating the first loss function value and the second loss function value, a suitable optimization target can be defined for the model. When the parameters are updated, these loss function values ​​can be used as feedback signals to guide the model to optimize in a better direction. By updating the parameters of the initial large language model, the performance of the model can be continuously improved. After multiple iterative training, the model can gradually learn richer semantic and visual features, thereby improving prediction accuracy and generalization ability.

[0146] Figure 7 A schematic diagram of the process of training the initial large language model provided by an embodiment of the present invention is shown in FIG. Figure 7As shown, the initial large language model training includes two stages: using interlaced sample images and sample text data to establish bidirectional alignment and using different instruction data to adjust the conversation instructions. In the alignment stage, sample images and sample texts are used for training. In the instruction adjustment stage, the constructed instruction data allows the model to better follow the user's instruction request. After the above two stages, a multimodal token sequence is obtained, and the sample multimodal token sequence includes a sample image token sequence and a sample text token sequence. Based on the sample image token sequence and the pre-built dynamic visual dictionary, the image retrieval prediction result is obtained, and the generative loss corresponding to the image retrieval prediction result is calculated; based on the sample text token sequence and the pre-built language dictionary, the answer text prediction result is obtained, and the generative loss corresponding to the answer text prediction result is calculated.

[0147] The dynamic visual dictionary contains a rich set of sample features, namely the global visual features extracted from the frozen visual encoder. The dynamic visual dictionary contains a dynamically updated sample feature queue.

[0148] Optionally, the calculation formula of the first loss function value is:

[0149]

[0150] Where w represents the sample multimodal token sequence, t represents the position in the sequence, and w t represents the token at position t in the sample multimodal token sequence, w ＜t , represents the token before position t in the sample multimodal token sequence, θ,∈ represents the trainable parameters of the large language model, represents a dynamic visual dictionary, The set of tokens representing all positions in the sample multimodal token sequence.

[0151] Optionally, the sample query information includes a sample image query and a sample image query feature.

[0152] Optionally, the matching probability between each sample image query feature and the sample feature corresponding to the sample multimodal token sequence is calculated, and the calculation formula is as follows:

[0153]

[0154] Among them, I represents the location corresponding to the sample image query, w I represents the token corresponding to the sample image query in the sample multimodal token sequence, w ＜I represents the token before position I in the sample multimodal token sequence, h I represents the sample image query feature, Represents the matched sample features, represents the jth sample feature, and τ represents the temperature coefficient.

[0155] Optionally, the calculation formula of the normalized cosine similarity between the sample image query feature and the sample feature is:

[0156]

[0157] x=h I

[0158] or

[0159] in, represents the projection matrix corresponding to x, Represents the projection matrix corresponding to y.

[0160] In an embodiment of the present invention, sample multimodal dialogue information is obtained, answer text labels and / or image retrieval result labels corresponding to the sample multimodal dialogue information are determined, the sample multimodal dialogue information is used as a training sample, and the answer text labels and / or image retrieval result labels corresponding to the sample multimodal dialogue information are used as sample labels to train an initial large language model. After the initial large language model training is completed, a large language model is obtained, which improves the prediction accuracy and generalization ability of the large language model and expands the application scope of the large language model.

[0161] The following is a description of a conversational image retrieval device based on a large language model provided in an embodiment of the present invention. The conversational image retrieval device based on a large language model described below and the conversational image retrieval method based on a large language model described above can refer to each other.

[0162] Figure 8 A schematic diagram of the structure of a conversational image retrieval device based on a large language model provided by the present invention, such as Figure 8 As shown, the device 800 includes:

[0163] An acquisition unit 810 is used to acquire multimodal dialogue information input by a user, where the multimodal dialogue information includes images and texts;

[0164] The output unit 820 is used to input the multimodal dialogue information into a pre-built large language model to obtain the answer text and / or image retrieval result corresponding to the multimodal dialogue information output by the large language model;

[0165] The large language model is trained based on sample multimodal dialogue information and answer text labels and / or image retrieval result labels corresponding to the sample multimodal dialogue information, and the sample multimodal dialogue information includes sample images and sample texts.

[0166] Optionally, the large language model includes a visual feature encoding module, a perceptual compression module, a large language model decoder module and an image retrieval module;

[0167] Correspondingly, the multimodal dialogue information is input into a pre-built large language model to obtain answer text and / or image retrieval results corresponding to the multimodal dialogue information output by the large language model, including:

[0168] Inputting the image into the visual feature encoding module to obtain image features of the image output by the visual feature encoding module;

[0169] Inputting the image features into the perceptual compression module to obtain the image compression features output by the perceptual compression module;

[0170] Input the image compression features and text into the large language model decoder module to obtain the answer text output by the large language model decoder module, and / or input the image compression features and text into the large language model decoder module to obtain the query information output by the large language model decoder module, input the query information into the image retrieval module to obtain the image retrieval result output by the image retrieval module.

[0171] Optionally, the query information is input into an image retrieval module to obtain an image retrieval result output by the image retrieval module, including:

[0172] Based on the image retrieval module, the query information is compared with multiple conversation contexts in the pre-built index to determine the target conversation context that is most similar to the query information;

[0173] Based on the target conversation context and index, the image corresponding to the target conversation context is determined from the pre-built external image library to obtain the image retrieval result;

[0174] The index includes a plurality of conversation contexts and a mapping relationship between the plurality of conversation contexts and a plurality of images in an external image library;

[0175] Among them, the types of conversation contexts include multimodal conversation contexts and plaintext conversation contexts. The multimodal conversation contexts include a first multimodal conversation context, a second multimodal conversation context and a third multimodal conversation context. The first multimodal conversation context is constructed based on a reference image, the second multimodal conversation context is constructed based on a reference text, and the third multimodal conversation context is constructed after context fusion of the first multimodal conversation context and the second multimodal conversation context.

[0176] Optionally, the process of constructing the first multimodal dialogue context includes:

[0177] Acquire a first image-text pair, the first image-text pair comprising a reference image and a reference image description;

[0178] Acquiring reference image prompt information, the reference image prompt information including task prompt information for constructing a multimodal dialogue based on the reference image;

[0179] Based on the reference image prompt information, the multimodal query information is input into the image retriever to obtain a first target image corresponding to the multimodal query information output by the image retriever, wherein the multimodal query information includes the reference image and the initial instruction text;

[0180] Inputting the first target image into the image captioner to obtain a first target image description output by the image captioner;

[0181] Inputting the reference image description and the first target image description into the first text generator to obtain a modified instruction text output by the first text generator;

[0182] A first multimodal dialogue context is constructed based on the reference image, the modified instruction text, and the first target image.

[0183] Optionally, the process of constructing the second multimodal dialogue context includes:

[0184] Acquire a second image-text pair, the image-text pair comprising a second target image and an original description of the second target image;

[0185] Acquiring reference text prompt information, the reference text prompt information including task prompt information for constructing a multimodal dialogue based on the reference text;

[0186] Based on the reference text prompt information, the original description of the second target image is input into the second text generator to obtain two rounds of dialogue output by the second text generator;

[0187] Based on two rounds of dialogue, a reference text is obtained;

[0188] Based on the reference text, the most relevant image corresponding to the reference text is retrieved from an external image library;

[0189] Based on the two-turn dialogue, reference text and the most relevant image, a second multimodal dialogue context is constructed.

[0190] Optionally, the process of constructing the plaintext conversation context includes:

[0191] Acquire a third image-text pair, where the image-text pair includes a third target image and an original description of the third target image;

[0192] Obtaining plaintext dialogue prompt information, the plaintext dialogue prompt information including task prompt information for constructing a plaintext dialogue;

[0193] Based on the reference text prompt information, the original description of the third target image is input into the third text generator to obtain multiple rounds of simulated query information output by the third text generator;

[0194] Based on multiple rounds of simulated query information, a plaintext conversation context is constructed.

[0195] Optionally, the conversational image retrieval device 800 based on a large language model further includes:

[0196] The determination unit is used to receive a selection instruction from a user, determine a target candidate image from a plurality of candidate images in response to the selection instruction, and append the target candidate image to the end of the historical multimodal dialogue sequence.

[0197] Optionally, the process of determining the large language model includes:

[0198] Acquire sample multimodal dialogue information, where the sample multimodal dialogue information includes a sample image and a sample text;

[0199] Determine an answer text label and / or an image retrieval result label corresponding to the sample multimodal dialogue information;

[0200] Using the sample multimodal dialogue information as training samples and the answer text labels and / or image retrieval result labels corresponding to the sample multimodal dialogue information as sample labels, an initial large language model is trained;

[0201] After the initial large language model training is completed, a large language model is obtained.

[0202] Optionally, the initial large language model includes an initial visual feature encoding module, an initial perceptual compression module, an initial large language model decoder module and an initial image retrieval module, and training the initial large language model includes:

[0203] Inputting the sample image into the initial visual feature encoding module to obtain the sample image features of the sample image output by the initial visual feature encoding module;

[0204] Inputting the sample image features into the initial perceptual compression module to obtain the sample image compression features output by the initial perceptual compression module;

[0205] Inputting the sample image compression features and the sample text into the initial large language model decoder module to obtain the answer text prediction result output by the initial large language model decoder module, and / or inputting the sample image compression features and the sample text into the initial large language model decoder module to obtain the sample query information output by the initial large language model decoder module, and inputting the sample query information into the initial image retrieval module to obtain the image retrieval prediction result output by the initial image retrieval module;

[0206] Calculate a first loss function value based on the image retrieval prediction result and the image retrieval result label corresponding to the sample multimodal dialogue information;

[0207] Calculate the second loss function value based on the answer text prediction result and the answer text label corresponding to the sample multimodal dialogue information;

[0208] Based on the first loss function value and the second loss function value, the parameters of the initial large language model are updated.

[0209] The conversational image retrieval device based on a large language model provided by the present invention is described below. The conversational image retrieval device based on a large language model described below and the conversational image retrieval method based on a large language model described above can be referenced to each other.

[0210] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0211] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0212] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A conversational image retrieval method based on a large language model, characterized in that: include: Acquire multimodal dialogue information input by a user, wherein the multimodal dialogue information includes an image and a text; Inputting the multimodal dialogue information into a pre-built large language model to obtain answer text and / or image retrieval results corresponding to the multimodal dialogue information output by the large language model; The large language model is trained based on sample multimodal dialogue information and answer text labels and / or image retrieval result labels corresponding to the sample multimodal dialogue information, wherein the sample multimodal dialogue information includes sample images and sample texts; The process of determining the large language model includes: Acquiring sample multimodal dialogue information, wherein the sample multimodal dialogue information includes a sample image and a sample text; Determining an answer text label and / or an image retrieval result label corresponding to the sample multimodal dialogue information; Using the sample multimodal dialogue information as training samples and answer text labels and / or image retrieval result labels corresponding to the sample multimodal dialogue information as sample labels, training an initial large language model; After the initial large language model training is completed, obtaining the large language model; The large language model includes a visual feature encoding module, a perceptual compression module, a large language model decoder module and an image retrieval module; Correspondingly, the step of inputting the multimodal dialogue information into a pre-built large language model to obtain an answer text and / or image retrieval result corresponding to the multimodal dialogue information output by the large language model includes: Inputting the image into the visual feature encoding module to obtain image features of the image output by the visual feature encoding module; Inputting the image features into the perceptual compression module to obtain image compression features output by the perceptual compression module; Inputting the image compression feature and the text into the large language model decoder module to obtain the answer text output by the large language model decoder module, and / or inputting the image compression feature and the text into the large language model decoder module to obtain the query information output by the large language model decoder module, and inputting the query information into the image retrieval module to obtain the image retrieval result output by the image retrieval module; The step of inputting the query information into the image retrieval module to obtain the image retrieval result output by the image retrieval module includes: Based on the image retrieval module, the query information is compared with a plurality of conversation contexts in a pre-built index to determine a target conversation context that is most similar to the query information; Based on the target conversation context and the index, determining an image corresponding to the target conversation context from a pre-built external image library to obtain an image retrieval result; The index includes the multiple conversation contexts and mapping relationships between the multiple conversation contexts and the multiple images in the external image library; Among them, the types of the conversation context include multimodal conversation context and plaintext conversation context, the multimodal conversation context includes a first multimodal conversation context, a second multimodal conversation context and a third multimodal conversation context, the first multimodal conversation context is constructed based on a reference image, the second multimodal conversation context is constructed based on a reference text, and the third multimodal conversation context is constructed after context fusion of the first multimodal conversation context and the second multimodal conversation context.

2. The conversational image retrieval method based on a large language model according to claim 1, characterized in that: The process of constructing the first multimodal dialogue context includes: Acquire a first image-text pair, wherein the first image-text pair includes a reference image and a reference image description; Acquire reference image prompt information, wherein the reference image prompt information includes task prompt information for constructing a multimodal dialogue based on the reference image; Based on the reference image prompt information, inputting the multimodal query information into an image retriever to obtain a first target image corresponding to the multimodal query information output by the image retriever, wherein the multimodal query information includes the reference image and an initial instruction text; Inputting the first target image into an image captioning device to obtain a first target image description output by the image captioning device; Inputting the reference image description and the first target image description into a first text generator to obtain a modified instruction text output by the first text generator; The first multimodal dialogue context is constructed based on the reference image, the modified instruction text and the first target image.

3. The conversational image retrieval method based on a large language model according to claim 1, characterized in that: The construction process of the second multimodal dialogue context includes: Acquire a second image-text pair, wherein the image-text pair includes a second target image and an original description of the second target image; Acquiring reference text prompt information, wherein the reference text prompt information includes task prompt information for constructing a multimodal dialogue based on the reference text; Based on the reference text prompt information, inputting the original description of the second target image into a second text generator to obtain two rounds of dialogue output by the second text generator; Based on the two rounds of dialogue, obtaining the reference text; Based on the reference text, retrieving the most relevant image corresponding to the reference text from the external image library; The second multimodal dialogue context is constructed based on the two rounds of dialogue, the reference text and the most relevant image.

4. The conversational image retrieval method based on a large language model according to claim 1, characterized in that: The construction process of the plaintext conversation context includes: Acquire a third image-text pair, wherein the image-text pair includes a third target image and an original description of the third target image; Obtaining plaintext dialogue prompt information, wherein the plaintext dialogue prompt information includes task prompt information for constructing a plaintext dialogue; Based on the reference text prompt information, inputting the original description of the third target image into a third text generator to obtain multiple rounds of simulated query information output by the third text generator; Based on the multiple rounds of simulated query information, a plaintext conversation context is constructed.

5. The conversational image retrieval method based on a large language model according to any one of claims 1 to 4, characterized in that: The image retrieval result includes a plurality of candidate images, and the method further includes: A selection instruction of a user is received, and in response to the selection instruction, a target candidate image is determined from the multiple candidate images, and the target candidate image is appended to the end of the historical multimodal dialogue sequence.

6. The conversational image retrieval method based on a large language model according to claim 1, characterized in that: The initial large language model includes an initial visual feature encoding module, an initial perceptual compression module, an initial large language model decoder module and an initial image retrieval module. The training of the initial large language model includes: Inputting the sample image into the initial visual feature encoding module to obtain the sample image features of the sample image output by the initial visual feature encoding module; Inputting the sample image features into the initial perceptual compression module to obtain the sample image compression features output by the initial perceptual compression module; Input the sample image compression feature and the sample text into the initial large language model decoder module to obtain the answer text prediction result output by the initial large language model decoder module, and / or input the sample image compression feature and the sample text into the initial large language model decoder module to obtain the sample query information output by the initial large language model decoder module, input the sample query information into the initial image retrieval module to obtain the image retrieval prediction result output by the initial image retrieval module; Calculating a first loss function value based on the image retrieval prediction result and the image retrieval result label corresponding to the sample multimodal dialogue information; Calculating a second loss function value based on the answer text prediction result and the answer text label corresponding to the sample multimodal dialogue information; Based on the first loss function value and the second loss function value, the parameters of the initial large language model are updated.

7. A conversational image retrieval device based on a large language model, characterized in that: include: An acquisition unit, configured to acquire multimodal dialogue information input by a user, wherein the multimodal dialogue information includes images and texts; an output unit, configured to input the multimodal dialogue information into a pre-built large language model, and obtain an answer text and / or image retrieval result corresponding to the multimodal dialogue information output by the large language model; The large language model is trained based on sample multimodal dialogue information and answer text labels and / or image retrieval result labels corresponding to the sample multimodal dialogue information, wherein the sample multimodal dialogue information includes sample images and sample texts; The process of determining the large language model includes: Acquiring sample multimodal dialogue information, wherein the sample multimodal dialogue information includes a sample image and a sample text; Determining an answer text label and / or an image retrieval result label corresponding to the sample multimodal dialogue information; Using the sample multimodal dialogue information as training samples and answer text labels and / or image retrieval result labels corresponding to the sample multimodal dialogue information as sample labels, training an initial large language model; After the initial large language model training is completed, obtaining the large language model; The large language model includes a visual feature encoding module, a perceptual compression module, a large language model decoder module and an image retrieval module; Correspondingly, the step of inputting the multimodal dialogue information into a pre-built large language model to obtain an answer text and / or image retrieval result corresponding to the multimodal dialogue information output by the large language model includes: Inputting the image into the visual feature encoding module to obtain image features of the image output by the visual feature encoding module; Inputting the image features into the perceptual compression module to obtain image compression features output by the perceptual compression module; Inputting the image compression feature and the text into the large language model decoder module to obtain the answer text output by the large language model decoder module, and / or inputting the image compression feature and the text into the large language model decoder module to obtain the query information output by the large language model decoder module, and inputting the query information into the image retrieval module to obtain the image retrieval result output by the image retrieval module; The step of inputting the query information into the image retrieval module to obtain the image retrieval result output by the image retrieval module includes: Based on the image retrieval module, the query information is compared with a plurality of conversation contexts in a pre-built index to determine a target conversation context that is most similar to the query information; Based on the target conversation context and the index, determining an image corresponding to the target conversation context from a pre-built external image library to obtain an image retrieval result; The index includes the multiple conversation contexts and mapping relationships between the multiple conversation contexts and the multiple images in the external image library; Among them, the types of the conversation context include multimodal conversation context and plaintext conversation context, the multimodal conversation context includes a first multimodal conversation context, a second multimodal conversation context and a third multimodal conversation context, the first multimodal conversation context is constructed based on a reference image, the second multimodal conversation context is constructed based on a reference text, and the third multimodal conversation context is constructed after context fusion of the first multimodal conversation context and the second multimodal conversation context.

Images