Data processing method, computer equipment and readable storage medium

Abstract

The invention relates to a data processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps that an image-text knowledge data set is obtained, the image-text knowledge data set comprises multiple sets of data, and each set of data comprises an image, a knowledge text corresponding to the image and a query problem corresponding to the knowledge text; for each group of data, acquiring global description of a target image contained in the current data, performing target detection on the target image based on keywords contained in the global description to obtain a target detection frame, and acquiring a local image set based on the target detection frame; acquiring a local description of each local image in the local image set to obtain a local description set; forming an image-text matching pair by the global description, the local image set, the local description set and the current data; constructing positive and negative sample pairs based on the image-text matching pair corresponding to each group of data; and training the multi-modal model based on the positive and negative sample pairs. And the model output accuracy is improved.

Description

Technical Field

[0001] The present application relates to the technical field of machine learning, and in particular to a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Art

[0002] In recent years, with the development of deep learning technology, language models based on deep learning have gradually become mainstream. Among them, multimodal models allow users to ask questions in a combination of pictures and texts. However, the accuracy of the responses output by current multimodal models is not high. Summary of the invention

[0003] Based on this, it is necessary to provide a data processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of responses in response to the above technical problems.

[0004] In a first aspect, the present application provides a data processing method, comprising:

[0005] Obtaining a graphic and text knowledge dataset, the graphic and text knowledge dataset includes multiple groups of data, each group of data includes an image, a knowledge text corresponding to the image, and a query question corresponding to the knowledge text;

[0006] For each set of data, obtain the global description of the target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description, obtain the target detection frame, and obtain the local image set based on the target detection frame; obtain the local description of each local image in the local image set to obtain the local description set; the global description, the local image set, the local description set and the current data constitute an image-text matching pair;

[0007] Based on the image-text matching pairs corresponding to each set of data, positive and negative sample pairs are constructed; the positive and negative sample pairs are used to train the multimodal model.

[0008] In one of the embodiments, obtaining a local image set based on a target detection frame includes:

[0009] Perform cutting processing on the target image based on the target detection frame to obtain a local image of the target;

[0010] Obtaining the number of horizontal and vertical divisions based on a preset image segmentation rule, and dividing the target image into at least one regional local image based on the number of horizontal and vertical divisions;

[0011] Based on the target local image and the regional local image, a local image set is constructed.

[0012] In one embodiment, constructing positive and negative sample pairs based on the image-text matching pairs corresponding to each set of data includes:

[0013] For each image-text matching pair corresponding to each set of data, the target knowledge text and the target local description set contained in the current image-text matching pair are obtained; the target knowledge text is encoded to obtain a knowledge text vector; each local description in the target local description set is encoded respectively to obtain a local text vector for each local description;

[0014] Based on the knowledge text vector and the local text vector of each local description, the current image-text matching is split into positive and negative sample groups;

[0015] Based on the positive and negative sample groups obtained by splitting the image-text matching pairs corresponding to each group of data, positive and negative sample pairs are constructed.

[0016] In one embodiment, based on the knowledge text vector and the local text vector of each local description, the current image-text matching is split into positive and negative sample groups, including:

[0017] For each local description, the cosine distance between the knowledge text vector and the local text vector of the current local description is calculated. If the cosine distance is greater than or equal to the preset cosine distance threshold, the current local description is taken as a local description positive sample; if the cosine distance is less than the preset cosine distance threshold, the current local description is taken as a local description negative sample; all local description positive samples constitute a local description positive sample set, and all local description negative samples constitute a local description negative sample set;

[0018] Constructing a positive sample group based on the local description positive sample set, the local image corresponding to each local description in the local description positive sample set, the corresponding global description, and the corresponding data;

[0019] A negative sample group is constructed based on the local description negative sample set and the local image corresponding to each local description in the local description negative sample set.

[0020] In one embodiment, constructing positive and negative sample pairs based on the positive and negative sample groups obtained by splitting the image-text matching pairs corresponding to each group of data includes:

[0021] For each image-text matching pair corresponding to each group of data, obtain the target positive and negative sample groups obtained by splitting the current image-text matching pair, and use the positive sample group in the target positive and negative sample groups as a positive sample pair;

[0022] The target data is randomly selected from all data in the image-text knowledge dataset except the data corresponding to the current image-text matching pair, and a negative sample pair is constructed based on the negative sample group in the target positive and negative sample groups, the global description of the image contained in the target data, and the target data.

[0023] In one embodiment, training a multimodal model based on positive and negative sample pairs includes:

[0024] Based on the positive and negative sample pairs, the multimodal model to be trained is subjected to comparative learning to obtain a basic multimodal model.

[0025] Generate an instruction fine-tuning dataset based on the image-text matching pairs and preset instruction templates corresponding to each set of data;

[0026] Based on the instruction fine-tuning dataset, the basic multimodal model is trained by instruction fine-tuning to obtain a trained multimodal model.

[0027] In one embodiment, the method further comprises:

[0028] Acquire the data to be stored, the data to be stored including the stored image, the stored knowledge text corresponding to the stored image, and the stored query question corresponding to the stored knowledge text;

[0029] Obtain a global description of the stored image as the stored global description, perform target detection on the stored image based on the keywords contained in the stored global description, and obtain a stored local image set based on the detection result; obtain a local description of each local image in the stored local image set to obtain a stored local description set;

[0030] Encode the global description of the input data to obtain the global description vector of the input data; encode the knowledge text of the input data to obtain the knowledge text vector of the input data; respectively encode each local description in the local description set of the input data to obtain the local text vector of the input data;

[0031] Calculate the cosine distance between the stored knowledge text vector and each stored local text vector, and sort the local descriptions in the stored local description set in ascending order of the cosine distance to obtain the target local descriptions that are ranked in the front by a preset number;

[0032] Extract the local image embedding vector of each local image corresponding to the local description of the target; extract the global image embedding vector of the stored image;

[0033] Based on the stored global description vector, target local description, local image embedding vector, global image embedding vector and stored knowledge text vector, a knowledge group is constructed and stored in the knowledge base.

[0034] In one embodiment, the method further comprises:

[0035] receiving a search request, the search request including a query image and a query question;

[0036] Get a global description of the query image;

[0037] Based on the global description of the query image, at least one target knowledge group whose similarity meets a preset condition is retrieved from the knowledge base;

[0038] A local image description instruction is randomly selected from a preset local image description instruction set, and multiple local descriptions are generated based on the local image description instruction, the global description, the query question, and the knowledge text in the target knowledge group;

[0039] Expanding the query problem based on multiple local descriptions to obtain an expanded problem;

[0040] Based on the query question and the extended question, at least one target knowledge group is searched to obtain a final knowledge group, and a search response is generated based on the final knowledge group.

[0041] In a second aspect, the present application further provides a data processing device, comprising:

[0042] An acquisition module is used to acquire a picture-text knowledge data set, where the picture-text knowledge data set includes multiple groups of data, each group of data includes an image, a knowledge text corresponding to the image, and a query question corresponding to the knowledge text;

[0043] A construction module is used to obtain, for each set of data, a global description of a target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description to obtain a target detection frame, and obtain a local image set based on the target detection frame; obtain a local description of each local image in the local image set to obtain a local description set; the global description, the local image set, the local description set and the current data constitute an image-text matching pair;

[0044] The training module is used to construct positive and negative sample pairs based on the image-text matching pairs corresponding to each set of data; and to train the multimodal model based on the positive and negative sample pairs.

[0045] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0046] Obtaining a graphic and text knowledge dataset, the graphic and text knowledge dataset includes multiple groups of data, each group of data includes an image, a knowledge text corresponding to the image, and a query question corresponding to the knowledge text;

[0047] For each set of data, obtain the global description of the target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description, obtain the target detection frame, and obtain the local image set based on the target detection frame; obtain the local description of each local image in the local image set to obtain the local description set; the global description, the local image set, the local description set and the current data constitute an image-text matching pair;

[0048] Based on the image-text matching pairs corresponding to each set of data, positive and negative sample pairs are constructed; based on the positive and negative sample pairs, the multimodal model is trained.

[0049] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0050] Obtaining a graphic and text knowledge dataset, the graphic and text knowledge dataset includes multiple groups of data, each group of data includes an image, a knowledge text corresponding to the image, and a query question corresponding to the knowledge text;

[0051] For each set of data, obtain the global description of the target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description, obtain the target detection frame, and obtain the local image set based on the target detection frame; obtain the local description of each local image in the local image set to obtain the local description set; the global description, the local image set, the local description set and the current data constitute an image-text matching pair;

[0052] Based on the image-text matching pairs corresponding to each set of data, positive and negative sample pairs are constructed; based on the positive and negative sample pairs, the multimodal model is trained.

[0053] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0054] Obtaining a graphic and text knowledge dataset, the graphic and text knowledge dataset includes multiple groups of data, each group of data includes an image, a knowledge text corresponding to the image, and a query question corresponding to the knowledge text;

[0055] For each set of data, obtain the global description of the target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description, obtain the target detection frame, and obtain the local image set based on the target detection frame; obtain the local description of each local image in the local image set to obtain the local description set; the global description, the local image set, the local description set and the current data constitute an image-text matching pair;

[0056] Based on the image-text matching pairs corresponding to each set of data, positive and negative sample pairs are constructed; based on the positive and negative sample pairs, the multimodal model is trained.

[0057] The above data processing method, device, computer equipment, computer-readable storage medium and computer program product obtain a graphic knowledge data set, which includes multiple groups of data, each group of data includes an image, a knowledge text corresponding to the image and a query question corresponding to the knowledge text; for each group of data, obtain a global description of the target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description, obtain a target detection frame, and obtain a local image set based on the target detection frame; obtain a local description of each local image in the local image set to obtain a local description set; the global description, local image set, local description set and current data constitute a graphic-text matching pair; construct a positive and negative sample pair based on the graphic-text matching pair corresponding to each group of data; the positive and negative sample pairs are used to train a multimodal model. The image detail information is extracted in a fine-grained description manner, and the original graphic knowledge is data enhanced from multiple angles and from global to local. The multimodal model obtained by training the graphic-text knowledge data set after data enhancement can output a more precise response. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 is a flow chart of a data processing method in one embodiment;

[0060] Figure 2 An example diagram of data in an embodiment;

[0061] Figure 3 An example diagram for generating a global description of an embodiment;

[0062] Figure 4 An example diagram of keyword extraction in one embodiment;

[0063] Figure 5 An example diagram of a local image of a region in one embodiment;

[0064] Figure 6 is a flow chart of a data processing method in another embodiment;

[0065] Figure 7 A flowchart of data storage in one embodiment;

[0066] Figure 8 Retrieve a flowchart for an embodiment;

[0067] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] In one embodiment, Figure 1 As shown, a data processing method is provided. This embodiment takes the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0070] Step 102: Acquire a graphic and text knowledge data set, where the graphic and text knowledge data set includes multiple groups of data, each group of data includes an image, a knowledge text corresponding to the image, and a query question corresponding to the knowledge text.

[0071] Optionally, a graphic and text knowledge dataset of a certain professional field may be collected, the graphic and text knowledge dataset comprising multiple groups of data, each group of data comprising at least one image of the professional field, a knowledge text corresponding to the image, and a query question corresponding to the knowledge text.

[0072] In order to obtain a massive amount of graphic knowledge data sets, data is generally obtained from multiple channels, so there is a possibility that images may contain multiple elements that are irrelevant to text content. Figure 2 , illustrates a set of images and knowledge texts in data. The knowledge text mainly explains the meaning of the warning lights in the image, but there are other information in the image, such as average fuel consumption, speedometer, etc. Figure 2 The middle arrow indicates.

[0073] Step 104, for each group of data, obtain the global description of the target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description to obtain the target detection frame, and obtain the local image set based on the target detection frame; obtain the local description of each local image in the local image set to obtain the local description set; the global description, the local image set, the local description set and the current data constitute a picture-text matching pair.

[0074] Optionally, a native visual text multimodal model M to be fine-tuned may be prepared in advance, such as BLIPv2, GLM-4V-9B, etc. For the convenience of explanation, the visual text multimodal model mentioned in the embodiment of the present application may be referred to as a multimodal model. For each set of data, the multimodal model M may be used to obtain a global description of the target image contained in the current data.

[0075] Specifically, for each set of data, the multimodal model M can be used to process the target image contained in the current data to generate a global description T g When using the multimodal model M to process the target image, the prompt word P can be used g , P g It can be expressed as "Describe this picture", which can be expressed by the following mathematical expression T g The generation process:

[0076] T g =M(I,P g )

[0077] For example, see Figure 3 As shown in the figure, using the prompt word "Describe this picture", the multimodal model M is used to Figure 3 After the image is processed, the global description generated is "The picture shows the hood of a car opened. In the middle of the picture is a silver engine with a silver logo on it that says TURBO. In the upper right corner of the picture, there is a red container filled with liquid, which may be coolant or brake fluid. In addition, there are other car parts and pipes, but their specific functions are not clear."

[0078] For each set of data, after obtaining a global description of the target image contained in the current data, keywords can be extracted from the global description.

[0079] Optionally, the natural language sub-model M in the multimodal model M can be used L From the global description T g Extract the keyword set K and the prompt word P k Can be "from <quote>< / quote> Extract potential keywords from the content of the tag", <quote>< / quote> The content in the tag is the global description T g The above process can be expressed by the following mathematical expression, Figure 4 As an example of this process:

[0080] K=M L (T g ,P k )

[0081] Optionally, the Grounded-SAM model M sam , use each keyword in the keyword set K to perform target detection on the target image, obtain the target detection frame, the keyword and the target detection frame correspond one to one, and obtain the target detection frame set B.

[0082] Optionally, after obtaining the target detection frame set B, the target detection frame in the target detection frame set B is used to perform cutting processing on the target image to obtain a target partial image. The target detection frame and the target partial image correspond one to one. The set composed of each target partial image can be used as the partial image set I. loc .

[0083] Optionally, a multimodal model M can be used for the local image set I loc Each local image in is processed to generate the corresponding local description, and the local description set T is obtained. loc The above global description, local image set, local description set and current data constitute a picture-text matching pair; the above is the process of processing a group of data in the picture-text knowledge data set D. The same method can be used to process all data to obtain the picture-text matching pair corresponding to each group of data.

[0084] After the above data enhancement, a new graphic knowledge dataset D * , Graph and Text Knowledge Dataset D * It consists of multiple image-text matching pairs, where the i-th image-text matching pair includes the image I contained in the i-th group of data i , local image set I i loc 、The query question Q contained in the i-th group of data i , the knowledge text T contained in the i-th group of data i , global description T i g , and the local description set T i loc .

[0085] Step 106, constructing positive and negative sample pairs based on the image-text matching pairs corresponding to each set of data; the positive and negative sample pairs are used to train the multimodal retrieval enhancement generation multimodal model.

[0086] Optional, for the graph-text knowledge dataset D * For each image-text matching pair in the algorithm, the image-text matching pair can be split to obtain positive and negative sample groups; based on the positive and negative sample groups corresponding to each image-text matching pair, positive and negative sample pairs are constructed; based on the positive and negative sample pairs, the multimodal model to be trained can be compared and learned to obtain a trained multimodal model for actual reasoning.

[0087] Optionally, the multimodal model can be a multimodal retrieval-augmented generation (RAG) model, and the method provided in the embodiment of the present application can improve the generation performance of the multimodal RAG model. The multimodal RAG model is an artificial intelligence model that combines retrieval and generation capabilities. It is mainly used to process and generate data involving multiple modalities (such as text, images, sounds, etc.). The following are some typical usage scenarios of the multimodal RAG model: 1) Image to text generation: The multimodal RAG model can generate descriptive text based on the input image, such as automatically generating a title, description or story for the image. 2) Visual question answering system: In the visual question answering system, the multimodal RAG model can process user questions about the content of the image and generate accurate answers. 3) Semantic retrieval of images and text: The multimodal RAG model can retrieve semantically related images from the image database based on the text query provided by the user, or retrieve relevant text information based on the image query. In combination with the usage scenarios of the multimodal model, the above-mentioned positive and negative samples can be used to perform comparative learning on the multimodal model to be trained.

[0088] In the above embodiment, a graphic-text knowledge data set is obtained, and the graphic-text knowledge data set includes multiple groups of data, each group of data includes an image, a knowledge text corresponding to the image, and a query question corresponding to the knowledge text; for each group of data, a global description of the target image contained in the current data is obtained, and target detection is performed on the target image based on the keywords contained in the global description to obtain a target detection frame, and a local image set is obtained based on the target detection frame; a local description of each local image in the local image set is obtained to obtain a local description set; the global description, the local image set, the local description set, and the current data constitute a graphic-text matching pair; based on the graphic-text matching pair corresponding to each group of data, a positive and negative sample pair is constructed; the positive and negative sample pairs are used to train the multimodal model. The image detail information is extracted in a fine-grained description manner, and the original graphic-text knowledge is data enhanced from multiple angles and from global to local. The multimodal model obtained by training the graphic-text knowledge data set after data enhancement can output a more precise response.

[0089] In some embodiments, obtaining a local image set based on a target detection frame includes: performing cutting processing on a target image based on the target detection frame to obtain a target local image; obtaining a number of horizontal divisions and a number of vertical divisions based on a preset image segmentation rule, and dividing the target image into at least one regional local image based on the number of horizontal divisions and the number of vertical divisions; and constructing a local image set based on the target local image and the regional local image.

[0090] Optionally, after obtaining the target detection frame set B, the target detection frames in the target detection frame set B are used to perform cutting processing on the target image to obtain a target partial image. The target detection frames correspond to the target partial images one by one, and the set composed of each target partial image can be used as the target partial image set. Since some potential targets may not be detected in the above target detection process, the embodiments of the present application supplement these targets.

[0091] Optionally, the preset image segmentation rule can be h g *w g Grid, where h g It means that the image is evenly divided into h along the y axis g Quantity, that is, h g is the number of vertical divisions. Similarly, w g Refers to the average division of the image into w along the x-axis g Part, that is, w g is the number of horizontal divisions. Based on the number of horizontal divisions and the number of vertical divisions, the target image can be divided into h g *w g These regional local images constitute a regional local image set. like Figure 5 As shown, h g is 3, w g is 3, so this figure can obtain 9 regional local images.

[0092] Optionally, after obtaining the target local image set and regional local image set After that, the local image set can be constructed

[0093] In the above embodiment, the target image is cut based on the target detection frame to obtain a target partial image; the number of horizontal and vertical divisions is obtained based on the preset image segmentation rules, and the target image is divided into at least one regional partial image based on the number of horizontal and vertical divisions; a partial image set is constructed based on the target partial image and the regional partial image. The targets that may be missed in the target detection process are supplemented, which improves the effect of data enhancement.

[0094] In some embodiments, based on the image-text matching pairs corresponding to each group of data, positive and negative sample pairs are constructed, including: for each image-text matching pair corresponding to each group of data, obtaining the target knowledge text and the target local description set contained in the current image-text matching pair; encoding the target knowledge text to obtain a knowledge text vector; respectively encoding each local description in the target local description set to obtain a local text vector for each local description; based on the knowledge text vector and the local text vector of each local description, splitting the current image-text matching into positive and negative sample groups; constructing positive and negative sample pairs based on the positive and negative sample groups obtained by splitting the image-text matching pairs corresponding to each group of data.

[0095] Optionally, since there may be irrelevant descriptions in the local description set, such as the local description corresponding to the local image of the region, it is very likely to be irrelevant. Therefore, for the image-text matching pair corresponding to each set of data, the target knowledge text and the target local description set contained in the current image-text matching pair can be obtained; the text embedding model M of the multimodal model M is used te Encode the target knowledge text into a knowledge text vector V i T , encode each local description in the target local description set into a local text vector Based on the knowledge text vector and the local text vector of each local description, the current image-text matching is split into positive and negative sample groups; based on the positive and negative sample groups obtained by splitting the image-text matching pairs corresponding to each group of data, positive and negative sample pairs are constructed.

[0096] In the above embodiment, an implementation method for constructing positive and negative sample pairs is provided, and subsequently, the multimodal model to be trained can be compared and learned based on the positive and negative sample pairs. Since the positive and negative sample pairs are constructed based on the data-enhanced graphic knowledge dataset, the trained multimodal model can output more precise responses.

[0097] In some embodiments, based on the knowledge text vector and the local text vector of each local description, the current image-text match is split into positive and negative sample groups, including: for each local description, the cosine distance between the knowledge text vector and the local text vector of the current local description is calculated, if the cosine distance is greater than or equal to a preset cosine distance threshold, the current local description is used as a local description positive sample; if the cosine distance is less than the preset cosine distance threshold, the current local description is used as a local description negative sample; all local description positive samples constitute a local description positive sample set, and all local description negative samples constitute a local description negative sample set; based on the local description positive sample set, the local image corresponding to each local description in the local description positive sample set, the corresponding global description and the corresponding data, a positive sample group is constructed; based on the local description negative sample set and the local image corresponding to each local description in the local description negative sample set, a negative sample group is constructed.

[0098] Optionally, use the text embedding model M of the multimodal model M te Encode the target knowledge text into a knowledge text vector V i T , encode each local description in the target local description set into a local text vector Then, calculate V i T and The cosine distance between them is the cosine distance between them, and the local descriptions that are greater than or equal to the preset cosine distance threshold are classified as local description positive samples; all local description positive samples constitute the local description positive sample set T i loc_p , otherwise it is classified as a local description negative sample; all local description negative samples constitute the local description negative sample set T i loc_n . The local description positive sample set T can be obtained i loc_p Corresponding local image positive sample set Get the local description negative sample set T i loc_n The corresponding local image negative sample set Therefore, the i-th image-text matching pair can form positive and negative sample groups, and the positive sample group is The negative sample group is According to this allocation mechanism, the data set D * Further optimized into the graph and text knowledge dataset D ** .

[0099] In the above embodiment, an implementation method for splitting the image-text matching into positive and negative sample groups is provided. This process is the basis for constructing positive and negative sample pairs. Subsequently, the multimodal model to be trained can be compared and learned based on the positive and negative sample pairs. Since the positive and negative sample pairs are constructed based on the data-enhanced image-text knowledge dataset, the trained multimodal model can output more precise responses.

[0100] In some embodiments, based on the positive and negative sample groups obtained by splitting the image-text matching pairs corresponding to each group of data, constructing positive and negative sample pairs, including: for the image-text matching pairs corresponding to each group of data, obtaining the target positive and negative sample groups obtained by splitting the current image-text matching pairs, and taking the positive sample group in the target positive and negative sample groups as a positive sample pair; randomly selecting target data from all data in the image-text knowledge data set except the data corresponding to the current image-text matching pair, and constructing a negative sample pair based on the negative sample group in the target positive and negative sample groups, the global description of the image contained in the target data, and the target data.

[0101] Optional, for the graph-text knowledge dataset D ** , the positive sample group obtained by splitting the i-th image-text matching pair The images and texts in the image can form positive sample pairs with each other, while the negative sample pairs Other data, the global description of the image contained in other data, forms negative sample pairs with each other.

[0102] In the above embodiment, a method for constructing positive and negative sample pairs after splitting each image-text matching pair into positive and negative sample groups is provided. Subsequently, based on the positive and negative sample pairs, comparative learning can be performed on the multimodal model to be trained to output more precise and accurate responses.

[0103] In some embodiments, the data processing method provided by the embodiments of the present application also includes: based on positive and negative sample pairs, performing comparative learning on the multimodal model to be trained to obtain a basic multimodal model; based on the image-text matching pairs corresponding to each group of data and the preset instruction template, generating an instruction fine-tuning data set; based on the instruction fine-tuning data set, performing instruction fine-tuning training on the basic multimodal model to obtain a trained multimodal model.

[0104] Among them, contrastive learning is a self-supervised learning technology that learns feature representation by comparing positive sample pairs and negative sample pairs. The core is to shorten the distance between positive sample pairs and increase the distance between negative sample pairs.

[0105] Optionally, LoRA is a parameter-efficient fine-tuning technology (PEFT technology for short). The application of LoRA can not only train the model efficiently, but also avoid the model from falling into underfitting due to insufficient data for full training of the model. The embodiment of the present application can use LoRA for parameter fine-tuning.

[0106] Among them, instruction fine-tuning technology can enable the model to learn to follow instructions to complete specific tasks and improve the model's ability to respond to specific instructions or task requests. Based on the basic multimodal model obtained through comparative learning, the embodiment of the present application can continue to perform instruction fine-tuning training to improve the retrieval efficiency and effect in actual application scenarios. Because in the retrieval process, based on real-time response considerations, it is impossible to perform local image extraction as described above, so it is necessary to fine-tune the model to have the ability to complete the extraction of feature information of local areas of the image through instructions and keywords. In order to achieve this effect, the embodiment of the present application pre-sets an instruction template, and based on the template, the data set D ** Transformed into an instruction fine-tuning dataset for instruction fine-tuning

[0107] Exemplarily, the preset instruction template may be:

[0108] ###Summarize

[0109] <Global Description>

[0110] ###Knowledge

[0111] <Lore Text>

[0112] ###Keywords

[0113] <Knowledge text keyword 1>

[0114] <Knowledge text keyword 2> ...

[0116] <Knowledge text keyword n>

[0117] ###question

[0118] <Query Question>

[0119] ###instruction

[0120] <Image local description preset instructions>

[0121] ###detail

[0122] <Local Description 1>

[0123] <Local Description 2> ...

[0125] <local description>

[0126] Among them, <Preset instructions for local image description> requires a set of instructions to be set in advance, and other contents can be obtained from the dataset D ** <Image local description preset instructions> can refer to the following instructions (including but not limited to):

[0127] 1. Combining the question and knowledge, please help me extract the main key content in the picture.

[0128] 2. Based on the question, use your knowledge to analyze the content of the picture and find the key information points.

[0129] 3. Based on the question, find the key elements in the picture.

[0130] 4. Generate a concise description of the image content, highlighting the key content related to the problem and knowledge.

[0131] 5. Recommend relevant key search descriptions to me based on the image content, knowledge and questions.

[0132] 6. Determine the theme or focus of the image and summarize it with keywords.

[0133] Optionally, fine-tune the dataset after getting the instructions After that, LoRA can also be used to perform instruction fine-tuning training on the basic multimodal model to obtain the trained multimodal model M * .

[0134] In the above embodiment, the training process of the multimodal model is introduced. Through fine-grained description fine-tuning, the multimodal model has the ability to extract local key information and align local multimodal features, thereby improving the multimodal model's ability to extract image and text embedding vectors of knowledge in a certain field and enhancing the fine-grainedness of image and text embedding vectors.

[0135] In some embodiments, a method for processing data is provided. Figure 6 As shown, the following steps are included: for each set of data in the image-text knowledge data set, the data includes an image, a knowledge text corresponding to the image, and a query question corresponding to the knowledge text, so the data is also called multimodal data. First, a global description of the image content can be generated, then keyword extraction based on the global description and image target detection based on keywords are performed; then, target image cutting and image preset area cutting are performed; then, a description of the local image content is generated; then, a comparison study is performed based on the fine-grained image-text positive and negative sample pairs; then, the instruction is fine-tuned based on the image local description instruction fine-tuning template; and finally, a fine-tuned multimodal model is output.

[0136] In some embodiments, the data processing method provided by the embodiments of the present application also includes: obtaining data to be stored in the warehouse, the data to be stored in the warehouse includes a stored image, a stored knowledge text corresponding to the stored image, and a stored query question corresponding to the stored knowledge text; obtaining a global description of the stored image as the stored global description, performing target detection on the stored image based on the keywords contained in the stored global description, and obtaining a stored local image set based on the detection result; obtaining a local description of each local image in the stored local image set to obtain a stored local description set; encoding the stored global description to obtain a stored global description vector; encoding the stored knowledge text to obtain a stored Knowledge text vector; encode each local description in the stored local description set to obtain the stored local text vector; calculate the cosine distance between the stored knowledge text vector and each stored local text vector, and sort the local descriptions in the stored local description set in order of cosine distance from small to large, and obtain a preset number of target local descriptions in the front; extract the local image embedding vector of the local image corresponding to each target local description; extract the global image embedding vector of the stored image; based on the stored global description vector, the target local description, the local image embedding vector, the global image embedding vector and the stored knowledge text vector, construct a knowledge group and store it in the knowledge base.

[0137] Optionally, the steps of obtaining a global description of the stored images, performing target detection on the stored images based on the keywords contained in the stored global description, obtaining a stored local image set based on the detection results, and obtaining a local description of each local image in the stored local image set can adopt the Grounded-SAM model and the multimodal model M * The detailed process can be referred to the above embodiments and will not be described again here.

[0138] Among them, the multimodal model M is used * Text embedding model Encode the global description of the input data to obtain the global description vector, encode the knowledge text of the input data to obtain the knowledge text vector, and encode each local description in the local description set to obtain the local text vector. Calculate the cosine distance between the knowledge text vector and each local text vector, and sort the local descriptions in the local description set in ascending order of cosine distance. Take the local description with the closest distance of TOP-K (K value is set according to different situations) as the target local description. And use the multimodal model M * Image embedding model Extract the local image embedding vector of each local description of the target corresponding to the local image; extract the global image embedding vector of the stored image. The stored global description vector, target local description, local image embedding vector, global image embedding vector, stored knowledge text vector, data to be stored, stored global description, and local image corresponding to the target local description form a knowledge group and store it in the knowledge base.

[0139] In some embodiments, see Figure 7 As shown, the storage step includes: for the multimodal data to be stored, first, a global description of the image content can be generated, and then, keyword extraction based on the global description and image target detection based on keywords are performed; then, target image cutting and image preset area cutting are performed; then, description generation about the local image content is performed; then, embedding vectors are extracted for all texts, and the cosine distance between the knowledge text embedding vector and all generated text embedding vectors is calculated; the TOP-K closest distance is taken to generate text and the corresponding local image, and embedding vectors are extracted for the original image and the selected local image, and the effective embedding vectors are entered into the database.

[0140] In the above embodiment, a multimodal fine-grained embedding vector matching mechanism is proposed to optimize the current mainstream warehousing strategy.

[0141] In terms of retrieval strategy, the current mainstream retrieval strategy is: first expand the query question, and then search in the knowledge base based on the original query question and multiple expanded query questions. Although expanding the query question can improve the hit rate, if the knowledge base is large and the query question is not clearly stated, it will lead to unsatisfactory retrieval results and inaccurate generated results. Therefore, the present invention adopts a method of "global rough matching of image description, fine matching of local description and query question" to optimize the current retrieval strategy.

[0142] In some embodiments, the data processing method provided by the embodiments of the present application also includes: receiving a retrieval request, the retrieval request includes a query image and a query question; obtaining a global description of the query image; based on the global description of the query image, retrieving at least one target knowledge group whose similarity meets preset conditions in the knowledge base; randomly selecting an image local description instruction from the image local description preset instruction set, and generating multiple local descriptions based on the image local description instruction, the global description, the query question, and the knowledge text in the target knowledge group; expanding the query question based on the multiple local descriptions to obtain an expanded question; searching in at least one target knowledge group based on the query question and the expanded question to obtain a final knowledge group, and generating a retrieval reply based on the final knowledge group.

[0143] Among them, the multimodal model M * The query image is processed to generate a global description of the query image. Using the multimodal model M * Text embedding model The global description is processed to generate a query text vector.

[0144] Among them, at least one target knowledge group with TOP-N (N value is set according to different situations) similarities that meet preset conditions can be retrieved from the knowledge base. Specifically, the global description vector contained in each knowledge group in the knowledge base can be obtained, the cosine distance between the query text vector and the global description vector can be calculated, and the knowledge groups in the knowledge base can be sorted in order of cosine distance from small to large, and the top TOP-N knowledge groups are used as target knowledge groups.

[0145] As described above, the preset instruction set for local description of an image includes but is not limited to the following instructions: 1. Based on the question and knowledge, please help me extract the main key content in the image. 2. Based on the question, use your knowledge to analyze the image content and find the key information points. 3. Based on the question, find the key elements in the image. 4. Generate a concise description of the image content, highlighting the key content related to the question and knowledge. 5. Based on the image content, knowledge and question, recommend relevant key search descriptions for me. 6. Determine the theme or focus of the image and summarize it with keywords.

[0146] Among them, the trained multimodal model M can be used based on the randomly selected local description instructions of the image, the global description of the query image, the query question, and the knowledge text in the target knowledge group. * Generate multiple local descriptions. The instructions can be extended based on multiple local descriptions to allow the multimodal model M * Expand the query questions in a targeted manner. Finally, use the query questions and the expanded questions to search in at least one target knowledge group to obtain the final knowledge group. The knowledge text in the final knowledge group is converted into knowledge items, and retrieval replies are generated based on the knowledge items. Since local description generation no longer requires preprocessing of local images, the retrieval timeliness is guaranteed while the retrieval accuracy and granularity are improved, which ultimately improves the retrieval reply generation effect.

[0147] In some embodiments, see Figure 8 As shown, the retrieval step includes: receiving a retrieval request, the retrieval request includes a query image and a query question; performing global description generation on the image content; extracting the embedding vector of the global description, and detecting TOP-N knowledge groups from the knowledge base; then generating local descriptions through instructions; expanding the query question; extracting the embedding vector of the expanded question; retrieving knowledge items within the TOP-N knowledge groups, and generating answers in combination with the retrieved knowledge items.

[0148] In the above embodiment, the image is first globally described through a multimodal model, and then TOP-N similar knowledge groups are retrieved from the knowledge base. Finally, relevant answers are located in the knowledge groups through query questions and extended questions, thereby improving the retrieval response generation effect.

[0149] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0150] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0151] In an exemplary embodiment, a data processing device is provided, comprising:

[0152] An acquisition module is used to acquire a picture-text knowledge data set, where the picture-text knowledge data set includes multiple groups of data, each group of data includes an image, a knowledge text corresponding to the image, and a query question corresponding to the knowledge text;

[0153] A construction module is used to obtain, for each set of data, a global description of a target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description to obtain a target detection frame, and obtain a local image set based on the target detection frame; obtain a local description of each local image in the local image set to obtain a local description set; the global description, the local image set, the local description set and the current data constitute an image-text matching pair;

[0154] The training module is used to construct positive and negative sample pairs based on the image-text matching pairs corresponding to each set of data; and to train the multimodal model based on the positive and negative sample pairs.

[0155] In some embodiments, the acquisition module is specifically configured to:

[0156] Perform cutting processing on the target image based on the target detection frame to obtain a local image of the target;

[0157] Obtaining the number of horizontal and vertical divisions based on a preset image segmentation rule, and dividing the target image into at least one regional local image based on the number of horizontal and vertical divisions;

[0158] Based on the target local image and the regional local image, a local image set is constructed.

[0159] In some embodiments, the training module is specifically configured to:

[0160] For each image-text matching pair corresponding to each set of data, the target knowledge text and the target local description set contained in the current image-text matching pair are obtained; the target knowledge text is encoded to obtain a knowledge text vector; each local description in the target local description set is encoded respectively to obtain a local text vector for each local description;

[0161] Based on the knowledge text vector and the local text vector of each local description, the current image-text matching is split into positive and negative sample groups;

[0162] Based on the positive and negative sample groups obtained by splitting the image-text matching pairs corresponding to each group of data, positive and negative sample pairs are constructed.

[0163] In some embodiments, the training module is specifically configured to:

[0164] For each local description, the cosine distance between the knowledge text vector and the local text vector of the current local description is calculated. If the cosine distance is greater than or equal to the preset cosine distance threshold, the current local description is taken as a local description positive sample; if the cosine distance is less than the preset cosine distance threshold, the current local description is taken as a local description negative sample; all local description positive samples constitute a local description positive sample set, and all local description negative samples constitute a local description negative sample set;

[0165] Constructing a positive sample group based on the local description positive sample set, the local image corresponding to each local description in the local description positive sample set, the corresponding global description, and the corresponding data;

[0166] A negative sample group is constructed based on the local description negative sample set and the local image corresponding to each local description in the local description negative sample set.

[0167] In some embodiments, the training module is specifically configured to:

[0168] For each image-text matching pair corresponding to each group of data, obtain the target positive and negative sample groups obtained by splitting the current image-text matching pair, and use the positive sample group in the target positive and negative sample groups as a positive sample pair;

[0169] The target data is randomly selected from all data in the image-text knowledge dataset except the data corresponding to the current image-text matching pair, and a negative sample pair is constructed based on the negative sample group in the target positive and negative sample groups, the global description of the image contained in the target data, and the target data.

[0170] In some embodiments, the data processing device further includes a storage module, which is used to:

[0171] Acquire the data to be stored, the data to be stored including the stored image, the stored knowledge text corresponding to the stored image, and the stored query question corresponding to the stored knowledge text;

[0172] Obtain a global description of the stored image as the stored global description, perform target detection on the stored image based on the keywords contained in the stored global description, and obtain a stored local image set based on the detection result; obtain a local description of each local image in the stored local image set to obtain a stored local description set;

[0173] Encode the global description of the input data to obtain the global description vector of the input data; encode the knowledge text of the input data to obtain the knowledge text vector of the input data; respectively encode each local description in the local description set of the input data to obtain the local text vector of the input data;

[0174] Calculate the cosine distance between the stored knowledge text vector and each stored local text vector, and sort the local descriptions in the stored local description set in ascending order of the cosine distance to obtain the target local descriptions that are ranked in the front by a preset number;

[0175] Extract the local image embedding vector of each local image corresponding to the local description of the target; extract the global image embedding vector of the stored image;

[0176] Based on the stored global description vector, target local description, local image embedding vector, global image embedding vector and stored knowledge text vector, a knowledge group is constructed and stored in the knowledge base.

[0177] In some embodiments, the data processing device further comprises a retrieval module, which is used to:

[0178] receiving a search request, the search request including a query image and a query question;

[0179] Get a global description of the query image;

[0180] Based on the global description of the query image, at least one target knowledge group whose similarity meets a preset condition is retrieved from the knowledge base;

[0181] A local image description instruction is randomly selected from a preset local image description instruction set, and multiple local descriptions are generated based on the local image description instruction, the global description, the query question, and the knowledge text in the target knowledge group;

[0182] Expanding the query problem based on multiple local descriptions to obtain an expanded problem;

[0183] Based on the query question and the extended question, at least one target knowledge group is searched to obtain a final knowledge group, and a search response is generated based on the final knowledge group.

[0184] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0185] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store graphic knowledge data sets. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0186] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0187] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0188] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0189] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0190] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0191] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0192] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire a picture-text knowledge data set, the picture-text knowledge data set comprising multiple groups of data, each group of data comprising an image, a knowledge text corresponding to the image, and a query question corresponding to the knowledge text; For each group of data, a global description of a target image contained in the current data is obtained, target detection is performed on the target image based on keywords contained in the global description to obtain a target detection frame, and a local image set is obtained based on the target detection frame; a local description of each local image in the local image set is obtained to obtain a local description set; the global description, the local image set, the local description set and the current data constitute an image-text matching pair; Based on the image-text matching pairs corresponding to each set of data, positive and negative sample pairs are constructed; and the multimodal model is trained based on the positive and negative sample pairs.

2. The method according to claim 1, characterized in that The acquiring of a local image set based on the target detection frame comprises: Performing a cutting process on the target image based on the target detection frame to obtain a target partial image; Obtaining a number of horizontal divisions and a number of vertical divisions based on a preset image segmentation rule, and dividing the target image into at least one regional local image based on the number of horizontal divisions and the number of vertical divisions; A local image set is constructed based on the target local image and the regional local image.

3. The method according to claim 1, characterized in that The constructing of positive and negative sample pairs based on the image-text matching pairs corresponding to each set of data includes: For each image-text matching pair corresponding to each set of data, the target knowledge text and the target local description set contained in the current image-text matching pair are obtained; the target knowledge text is encoded to obtain a knowledge text vector; each local description in the target local description set is encoded respectively to obtain a local text vector of each local description; Based on the knowledge text vector and the local text vector of each local description, splitting the current image-text matching into positive and negative sample groups; Based on the positive and negative sample groups obtained by splitting the image-text matching pairs corresponding to each group of data, positive and negative sample pairs are constructed.

4. The method according to claim 3, characterized in that The step of splitting the current image-text matching into positive and negative sample groups based on the knowledge text vector and the local text vector of each local description includes: For each local description, the cosine distance between the knowledge text vector and the local text vector of the current local description is calculated. If the cosine distance is greater than or equal to a preset cosine distance threshold, the current local description is taken as a local description positive sample; if the cosine distance is less than the preset cosine distance threshold, the current local description is taken as a local description negative sample; all local description positive samples constitute a local description positive sample set, and all local description negative samples constitute a local description negative sample set; Constructing a positive sample group based on the local description positive sample set, the local image corresponding to each local description in the local description positive sample set, the corresponding global description and the corresponding data; A negative sample group is constructed based on the local description negative sample set and the local image corresponding to each local description in the local description negative sample set.

5. The method according to claim 3, characterized in that: The positive and negative sample groups obtained by splitting the image-text matching pairs corresponding to each group of data, and constructing the positive and negative sample pairs, include: For each image-text matching pair corresponding to each group of data, a target positive and negative sample group obtained by splitting the current image-text matching pair is obtained, and a positive sample group in the target positive and negative sample group is used as a positive sample pair; Target data is randomly selected from all data in the image-text knowledge dataset except data corresponding to the current image-text matching pair, and a negative sample pair is constructed based on a negative sample group in the target positive and negative sample groups, a global description of the image contained in the target data, and the target data.

6. The method according to claim 1, characterized in that The training of the multimodal model based on the positive and negative sample pairs includes: Based on the positive and negative sample pairs, comparative learning is performed on the multimodal model to be trained to obtain a basic multimodal model; Generate an instruction fine-tuning dataset based on the image-text matching pairs and preset instruction templates corresponding to each set of data; Based on the instruction fine-tuning data set, instruction fine-tuning training is performed on the basic multimodal model to obtain a trained multimodal model.

7. The method according to claim 1, characterized in that The method further comprises: Acquire data to be stored, the data to be stored comprising a storage image, storage knowledge text corresponding to the storage image, and a storage query question corresponding to the storage knowledge text; Obtain a global description of the stored image as the stored global description, perform target detection on the stored image based on keywords included in the stored global description, and obtain a stored local image set based on the detection result; obtain a local description of each local image in the stored local image set to obtain a stored local description set; Encoding the global description of the storage to obtain a global description vector of the storage; encoding the knowledge text of the storage to obtain a knowledge text vector of the storage; encoding each local description in the local description set of the storage to obtain a local text vector of the storage; Calculate the cosine distance between the stored knowledge text vector and each stored local text vector, and sort the local descriptions in the stored local description set in ascending order of the cosine distance to obtain a preset number of target local descriptions in the front; Extracting a local image embedding vector of a local image corresponding to each target local description; extracting a global image embedding vector of the stored image; Based on the stored global description vector, the target local description, the local image embedding vector, the global image embedding vector and the stored knowledge text vector, a knowledge group is constructed and stored in a knowledge base.

8. The method according to claim 1, characterized in that The method further comprises: receiving a search request, wherein the search request includes a query image and a query question; Obtaining a global description of the query image; Based on the global description of the query image, at least one target knowledge group whose similarity meets a preset condition is retrieved from the knowledge base; Randomly select an image local description instruction from a preset image local description instruction set, and generate multiple local descriptions using a trained multimodal model based on the image local description instruction, the global description, the query question, and the knowledge text in the target knowledge group; Expanding the query question based on the multiple local descriptions to obtain an expanded question; Based on the query question and the extended question, the at least one target knowledge group is searched to obtain a final knowledge group, and a search reply is generated based on the final knowledge group.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images