Joint training method and electronic device for image-text vector model and image-text large model

Abstract

The present disclosure provides a joint training method and electronic device for a graphic-text vector model and a graphic-text large model, the method comprising: obtaining a general question and answer sample set, an industry question and answer sample set, and an industry knowledge base; using the graphic-text large model to be trained to process general question samples in the general question and answer sample set to obtain general answers; using the graphic-text vector model to be trained, the industry knowledge base, and the graphic-text large model to be trained to process industry question samples in the industry question and answer sample set to obtain industry answers; determining a loss value based on the general answers, the general question and answer sample set, the industry answers, and the industry question and answer sample set; updating the parameters of the graphic-text large model to be trained based on the loss value, and updating the parameters of the graphic-text vector model to be trained based on the industry answers to obtain the graphic-text large model and the graphic-text vector model.

Description

Technical Field

[0001] The present disclosure generally relates to the fields of artificial intelligence and deep learning, and more specifically, to a joint training method and electronic device for a graphic vector model and a graphic large model. Background Art

[0002] With the development of artificial intelligence (AI) technology, image-text retrieval and generation models have been widely applied in various fields, with retrieval-enhanced generation being one of the most common application scenarios. In this scenario, image-text vector models are typically used to extract features from user-entered questions, retrieving images or text fragments related to the question as contextual information. The large image-text model is then used to process the question and contextual information to generate a response. This makes the generated response more accurate and rich, enhancing the performance of the generation task.

[0003] However, in existing retrieval-enhanced generation methods, the image-text vector model and the image-text large model are usually trained independently, resulting in insufficient coordination between the two and affecting the overall performance. Summary of the Invention

[0004] The present disclosure provides a joint training method and electronic device for a graphic-text vector model and a graphic-text large model, for solving at least one of the above problems.

[0005] According to the first aspect of the embodiments of the present disclosure, a joint training method for a graphic-text vector model and a graphic-text large model is provided, the method comprising: obtaining a general question and answer sample set, an industry question and answer sample set and an industry knowledge base; using the graphic-text large model to be trained to process the general question samples in the general question and answer sample set to obtain a general answer; using the graphic-text vector model to be trained, the industry knowledge base and the graphic-text large model to be trained to process the industry question samples in the industry question and answer sample set to obtain an industry answer; determining a loss value based on the general answer, the general question and answer sample set, the industry answer and the industry question and answer sample set; updating the parameters of the graphic-text large model to be trained based on the loss value, and updating the parameters of the graphic-text vector model to be trained based on the industry answer to obtain the graphic-text large model and the graphic-text vector model.

[0006] Optionally, the use of the image-text vector model to be trained, the industry knowledge base and the image-text large model to be trained to process the industry problem samples in the industry question and answer sample set to obtain industry answers includes: using the image-text vector model to be trained to extract the problem feature vector of the industry problem sample; retrieving a preset number of industry knowledge fragments that match the problem feature vector from the industry knowledge base; using the image-text large model to be trained to process the industry problem samples and the preset number of industry knowledge fragments to obtain the industry answer.

[0007] Optionally, the use of the large image and text model to be trained to process the industry problem samples and the preset number of industry knowledge fragments to obtain the industry answer includes: using the large image and text model to be trained to process the industry problem samples and the preset number of industry knowledge fragments to obtain the preset number of candidate industry answers, wherein each candidate industry answer corresponds one-to-one to each industry knowledge fragment; and determining a candidate industry answer from the preset number of candidate industry answers as the industry answer.

[0008] Optionally, updating the parameters of the image-text vector model to be trained based on the industry answer includes: determining the industry knowledge fragment corresponding to the industry answer as a positive sample, and determining the other industry knowledge fragments among the preset number of industry knowledge fragments except the positive sample as negative samples; based on the positive samples and the negative samples, using a comparative learning method to update the parameters of the image-text vector model to be trained.

[0009] Optionally, the large graphic model and the graphic vector model are obtained after multiple parameter updates, wherein the method further includes: when the preset update conditions are met, determining the latest graphic vector model as the knowledge base graphic vector model, and using the knowledge base graphic vector model to extract the fragment feature vectors of each industry knowledge fragment in the industry knowledge base, wherein the fragment feature vector is used to calculate the matching degree between the corresponding industry knowledge fragment and the problem feature vector.

[0010] Optionally, the preset update condition includes at least one of the following: the number of updates of the image-text vector model is equal to 0 or greater than or equal to a preset number, and a preset time has passed since the last update of the knowledge base image-text vector model.

[0011] Optionally, the large image-text model and the image-text vector model are obtained after multiple parameter updates, wherein the parameters of the large image-text model to be trained are updated according to the loss value, and the parameters of the image-text vector model to be trained are updated according to the industry answer to obtain the large image-text model and the image-text vector model, including: when the synchronous update condition is met, the parameters of the large image-text model to be trained are updated according to the loss value, and the parameters of the image-text vector model to be trained are updated according to the industry answer to obtain the large image-text model and the image-text vector model; when the synchronous update condition is not met, the parameters of the large image-text model to be trained are updated according to the loss value, and the parameters of the image-text vector model to be trained are kept unchanged to obtain the large image-text model and the image-text vector model.

[0012] According to the second aspect of an embodiment of the present disclosure, a joint training device for a graph-text vector model and a graph-text large model is provided, the device comprising: an acquisition unit, configured to acquire a general question and answer sample set, an industry question and answer sample set and an industry knowledge base; a general question and answer unit, configured to use the graph-text large model to be trained to process general question samples in the general question and answer sample set to obtain general answers; an industry question and answer unit, configured to use the graph-text vector model to be trained, the industry knowledge base and the graph-text large model to be trained to process industry question samples in the industry question and answer sample set to obtain industry answers; a determination unit, configured to determine a loss value based on the general answers, the general question and answer sample set, the industry answers and the industry question and answer sample set; an updating unit, configured to update the parameters of the graph-text large model to be trained based on the loss value, and to update the parameters of the graph-text vector model to be trained based on the industry answers, to obtain the graph-text large model and the graph-text vector model.

[0013] Optionally, the industry question and answer unit is further configured to: use the image-text vector model to be trained to extract the question feature vector of the industry question sample; retrieve a preset number of industry knowledge fragments that match the question feature vector from the industry knowledge base; use the image-text large model to be trained to process the industry question sample and the preset number of industry knowledge fragments to obtain the industry answer.

[0014] Optionally, the industry question and answer unit is further configured to: use the large graphic and text model to be trained to process the industry question samples and the preset number of industry knowledge fragments to obtain the preset number of candidate industry answers, wherein each candidate industry answer corresponds one-to-one to each industry knowledge fragment; and determine a candidate industry answer from the preset number of candidate industry answers as the industry answer.

[0015] Optionally, the update unit is further configured to: determine the industry knowledge fragment corresponding to the industry answer as a positive sample, and determine the other industry knowledge fragments among the preset number of industry knowledge fragments except the positive sample as negative samples; based on the positive samples and the negative samples, use a comparative learning method to update the parameters of the image-text vector model to be trained.

[0016] Optionally, the large graphic model and the graphic vector model are obtained after multiple parameter updates, wherein the device also includes an extraction unit, which is configured to determine the latest graphic vector model as the knowledge base graphic vector model when the preset update conditions are met, and use the knowledge base graphic vector model to extract the fragment feature vectors of each industry knowledge fragment in the industry knowledge base, wherein the fragment feature vector is used to calculate the matching degree between the corresponding industry knowledge fragment and the problem feature vector.

[0017] Optionally, the preset update condition includes at least one of the following: the number of updates of the image-text vector model is equal to 0 or greater than or equal to a preset number, and a preset time has passed since the last update of the knowledge base image-text vector model.

[0018] Optionally, the large image-text model and the image-text vector model are obtained after multiple parameter updates, wherein the update unit is further configured to: when the synchronous update condition is met, update the parameters of the large image-text model to be trained according to the loss value, and update the parameters of the image-text vector model to be trained according to the industry answer, to obtain the large image-text model and the image-text vector model; when the synchronous update condition is not met, update the parameters of the large image-text model to be trained according to the loss value, and keep the parameters of the image-text vector model to be trained unchanged, to obtain the large image-text model and the image-text vector model.

[0019] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute a joint training method for a graphic-text vector model and a graphic-text large model according to an exemplary embodiment of the present disclosure.

[0020] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the joint training method of the image-text vector model and the image-text large model according to the exemplary embodiment of the present disclosure.

[0021] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, prompt the at least one processor to execute a joint training method for a graphic-text vector model and a graphic-text large model according to an exemplary embodiment of the present disclosure.

[0022] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: according to the joint training method and electronic device of the graphic vector model and graphic large model disclosed in the present disclosure, by jointly training the graphic vector model and graphic large model that are related in application, the graphic vector model and graphic large model can be closely coordinated, which helps to improve the overall accuracy of multimodal retrieval enhancement generation and improve the overall performance of the model, and can also reduce the overall computational amount in training, reduce the computing resource consumption of the electronic equipment performing the training, and reduce hardware loss.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0025] Figure 1 4 is a flowchart of a joint training method of a graphic-text vector model and a graphic-text large model according to an exemplary embodiment of the present disclosure.

[0026] Figure 2 is a flowchart of generating industry answers according to an exemplary embodiment of the present disclosure.

[0027] Figure 3 4 is a flow chart of generating a segment feature vector database according to an exemplary embodiment of the present disclosure.

[0028] Figure 4 4 is a flowchart of retrieving industry knowledge fragments according to an exemplary embodiment of the present disclosure.

[0029] Figure 5 3 is a schematic diagram of an asynchronous update mechanism of a graph-text vector model and a knowledge base graph-text vector model according to an exemplary embodiment of the present disclosure.

[0030] Figure 6 1 is a flowchart of generating answers in different situations according to an exemplary embodiment of the present disclosure.

[0031] Figure 7 2 is a block diagram of a joint training apparatus for a graphic-text vector model and a graphic-text large model according to an exemplary embodiment of the present disclosure.

[0032] Figure 8 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0034] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0035] It should be noted that the phrase "at least one of the several items" in this disclosure includes three types of parallel situations: "any one of the several items", "a combination of any multiple of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step 1 and step 2" means the following three parallel situations: (1) performing step 1; (2) performing step 2; and (3) performing both step 1 and step 2.

[0036] Hereinafter, a joint training method and electronic device for a graphic-text vector model and a graphic-text large model according to exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0037] Figure 1 The present invention is a flowchart of a joint training method for a text-image vector model and a text-image large model according to an exemplary embodiment of the present invention. The method can be executed on an electronic device with sufficient computing power.

[0038] Reference Figure 1 In step S101, a general question and answer sample set, an industry question and answer sample set and an industry knowledge base are obtained.

[0039] Specifically, the general question and answer sample set includes multiple pairs of general question samples and general answer samples, and the industry question and answer sample set includes multiple pairs of industry question samples and industry answer samples. Each general question sample, each general answer sample, each industry question sample, and each industry answer sample can include only text data, that is, questions or answers in text form, or include both text data and image data, that is, questions or answers in the form of both text and images. Regarding the difference between these two sample sets, as the name suggests, industry questions are questions that require the use of industry knowledge to answer, and general questions are questions that do not require the use of industry knowledge. The industry knowledge base is used to provide industry knowledge, including multiple industry knowledge fragments, each of which can include only text data, only image data, or both text data and image data. The industry knowledge base can be constructed manually, automatically by electronic equipment, or by combining manual and electronic equipment. This disclosure does not limit this.

[0040] It should be understood that model training typically involves multiple parameter updates, and steps S101 to S105 describe the process performed for a single update. For multiple updates, the complete sample set can be divided into multiple sample subsets, with each update performed based on a single sample subset. Therefore, the general Q&A sample set and the industry Q&A sample set here represent the sample subsets used for a single update, while the industry knowledge base can be shared across different parameter updates.

[0041] In step S102, the large image-text model to be trained is used to process the general question samples in the general question and answer sample set to obtain general answers.

[0042] This step directly uses the large image and text model to be trained to answer general questions.

[0043] In step S103, the image-text vector model to be trained, the industry knowledge base and the image-text large model to be trained are used to process the industry question samples in the industry question and answer sample set to obtain industry answers.

[0044] This step uses the image-text vector model to be trained to retrieve relevant industry knowledge from the industry knowledge base as a reference for the image-text large model to be trained to answer industry questions.

[0045] In step S104, the loss value is determined based on the general answers, the general question and answer sample set, the industry answers, and the industry question and answer sample set.

[0046] Optionally, this step compares the general answer obtained in step S102 with the general answer sample corresponding to the same general question sample in the general question and answer sample set to obtain a first loss value, and compares the industry answer obtained in step S103 with the industry answer sample corresponding to the same industry question sample in the industry question and answer sample set to obtain a second loss value. Finally, the weighted value of the first loss value and the second loss value is determined as the loss value to describe the error of the answer as a reference for updating the large graph and text model. As an example, the first loss value or the second loss value can be determined based on the loss of the next token predicted by the large graph and text model.

[0047] In step S105, the parameters of the image-text large model to be trained are updated according to the loss value, and the parameters of the image-text vector model to be trained are updated according to the industry answer, so as to obtain the image-text large model and the image-text vector model.

[0048] For large image and text models, the previous steps are the forward propagation step in model training, while this step is the backpropagation step. Backpropagation algorithms can be used to calculate gradients, and optimization algorithms (such as gradient descent) can be used to update model parameters. For image and text vector models, the accuracy of the industry knowledge retrieved in step S103 can be evaluated based on the industry responses, and the evaluation results can be used as the basis for updating the image and text vector model parameters.

[0049] According to the joint training method of the graphic-text vector model and the graphic-text large model of the exemplary embodiment of the present disclosure, by jointly training the graphic-text vector model and the graphic-text large model that are related in application, the graphic-text vector model and the graphic-text large model can be closely coordinated, which helps to improve the overall accuracy of multimodal retrieval enhancement generation and improve the overall performance of the model. It can also reduce the overall amount of calculation in training, reduce the computing resource consumption of the electronic equipment performing the training, and reduce hardware loss. In addition, by distinguishing between general questions and industry questions and not using the industry knowledge base to search for general questions, it can not only reduce the amount of data processing and the computing resource consumption of the hardware, but also reduce the interference of industry knowledge on the answers to general questions, thereby improving the accuracy of the answers.

[0050] As an example, a pre-trained image-text vector model and an image-text large model can be used as basic models, and steps S101 to S105 are used to jointly fine-tune the two basic models.

[0051] It should be understood that the above step numbers are for the purpose of distinguishing different steps and are not intended to limit the order of execution. Figure 1 The execution order of each step in can be adjusted, for example, the execution order of step S102 and step S103 can be swapped, or they can be executed simultaneously if the electronic device has parallel computing capabilities. This disclosure does not limit this.

[0052] Next, the joint training method of the image-text vector model and the image-text large model according to the exemplary embodiment of the present disclosure is further introduced.

[0053] Regarding answers to industry questions, step S103 optionally includes: extracting question feature vectors from industry question samples using the image-text vector model to be trained; retrieving a preset number of industry knowledge snippets that match the question feature vectors from the industry knowledge base; and processing the industry question samples and the preset number of industry knowledge snippets using the image-text macro model to be trained to generate industry answers. By extracting question feature vectors using the image-text vector model to be trained, vectorized processing of industry question samples is achieved, enabling retrieval and matching of industry knowledge snippets, providing reference information for generating industry answers.

[0054] As an example, Figure 2 As shown, we can first use a knowledge base graph-text vector model similar to the graph-text vector model to uniformly vectorize the industry knowledge fragments in the industry knowledge base. The specific processing process is as follows: Figure 3 As shown, we can first use a multimodal parser to parse industry knowledge fragments of different modalities. The image data is directly input into the knowledge base image-text vector model. The text data (including the original text and the introduction text of the image) is preliminarily processed by the sentence segmenter and then input into the knowledge base image-text vector model to obtain the fragment feature vector of each industry knowledge fragment. The fragment feature vectors are summarized to obtain a fragment feature vector database, which can be regarded as a vectorized database of the industry knowledge base. Figure 4 As shown, when performing retrieval matching (since the processing method of the model inference stage is the same, the "to be trained" model is no longer emphasized), the image-text vector model is used to extract the problem feature vector of the industry problem (the industry problem may include image data and / or text data), and match it with each fragment feature vector in the fragment feature vector database. For example, the Euclidean distance between each fragment feature vector and the problem feature vector can be calculated as the matching degree, and the ranking model sorts these matching degrees, selects the K (i.e., the preset number below) fragment feature vectors with the highest matching degree (e.g., the shortest Euclidean distance), and returns the corresponding K industry knowledge fragments in the industry knowledge base as Top-K industry knowledge fragments, thereby retrieving the preset number of industry knowledge fragments with the highest matching degree. Then, as shown in FIG. Figure 2 As shown, the returned Top-K industry knowledge fragments are input into the graphic and text model together with the industry questions to generate industry answers.

[0055] Further optionally, the operation of using the large image-text model to be trained to process industry problem samples and a preset number of industry knowledge fragments to obtain industry answers in step S103 includes: using the large image-text model to be trained to process industry problem samples and a preset number of industry knowledge fragments to obtain a preset number of candidate industry answers, wherein each candidate industry answer corresponds one-to-one to each industry knowledge fragment; and determining a candidate industry answer from the preset number of candidate industry answers as the industry answer. By generating a candidate industry answer for each industry knowledge fragment, the mutual influence between the retrieved different industry knowledge fragments can be reduced, the accuracy of each candidate industry answer itself can be improved, and then one can be selected from them to achieve the answer to the industry question.

[0056] It should be understood that the method of combining industry question samples and industry knowledge fragments to generate candidate industry answers is an existing technology in the field and will not be elaborated here. As to how to determine the industry answer from a preset number of candidate industry answers, it can be determined by considering multiple factors. As an example, content relevance can be considered, such as the semantic relevance between the candidate industry answer and the industry question (for example, including but not limited to semantic similarity), and the fit between the candidate industry answer and the corresponding industry knowledge fragment; the quality of the answer can also be considered, such as the language fluency, information completeness, logical consistency, etc. of the candidate industry answer; the credibility and authority of the information source can also be considered. Of course, other reasonable factors can also be considered, which are not listed here one by one.

[0057] Based on an embodiment of obtaining a preset number of industry knowledge fragments and corresponding candidate industry answers, optionally, the operation of updating the parameters of the image-text vector model to be trained according to the industry answers in step S105 includes: determining the industry knowledge fragments corresponding to the industry answers as positive samples, and determining the other industry knowledge fragments other than the positive samples in the preset number of industry knowledge fragments as negative samples; based on the positive samples and negative samples, using the contrastive learning method to update the parameters of the image-text vector model to be trained. By determining the positive and negative correlation between the corresponding industry knowledge fragments and the industry answers based on whether a candidate industry answer is selected, the retrieval results obtained based on the image-text vector model can be easily evaluated, providing a reference for the parameter update of the image-text vector model. On this basis, the contrastive learning method is used to realize the chain update of the image-text vector model, thereby ensuring the training quality.

[0058] Based on an embodiment of retrieving a preset number of industry knowledge fragments that match the problem feature vector from an industry knowledge base, as mentioned above, a knowledge base graph-text vector model similar to the graph-text vector model can be used to uniformly vectorize the industry knowledge fragments in the industry knowledge base, and the obtained fragment feature vectors are used to calculate the matching degree of the corresponding industry knowledge fragments with the problem feature vector. In this regard, optionally, the graph-text large model and the graph-text vector model are obtained after multiple parameter updates. The joint training method of the graph-text vector model and the graph-text large model according to the exemplary embodiment of the present disclosure also includes: when the preset update conditions are met, the latest graph-text vector model is determined as the knowledge base graph-text vector model, and the knowledge base graph-text vector model is used to extract the fragment feature vectors of each industry knowledge fragment in the industry knowledge base.

[0059] This embodiment proposes an asynchronous update mechanism for the graph-text vector models on the problem side and the knowledge base side during the training process. Specifically, using the same graph-text vector model on the problem side and the knowledge base side (that is, the structure and parameters of the graph-text vector model and the knowledge base graph-text vector model are the same) can ensure the consistent extraction of feature vectors for industry problems and industry knowledge fragments. However, each time the knowledge base graph-text vector model is updated, the fragment feature vector database corresponding to the industry knowledge base needs to be updated synchronously. Due to the large amount of data in the industry knowledge base, a large amount of calculation will be required, which will seriously affect the update efficiency and cause a large amount of hardware computing resources to be consumed. By adopting an asynchronous update mechanism, such as Figure 5 As shown, timely updating of the problem-side graph-text vector model parameters and copying of the knowledge base graph-text vector model from the problem side when preset update conditions are met, along with updating of the segment feature vector database, can effectively reduce the update frequency of the segment feature vector database, reduce the computational effort and the resulting consumption of hardware computing resources, and improve model training efficiency. It should be understood that by properly configuring the preset update conditions, the knowledge base graph-text vector model can be updated appropriately, thus balancing the update frequency and the effectiveness of the segment feature vector database, ensuring training quality.

[0060] Optionally, the preset update conditions include at least one of the following: the number of consecutive updates of the graphic vector model is equal to 0 or greater than or equal to the preset number, and a preset time has passed since the last update of the knowledge base graphic vector model. The preset number and the preset time can be set as needed. It should be understood that both are positive numbers, and the preset number threshold is greater than 1, so that the update frequency of the graphic vector model is higher than that of the knowledge base graphic vector model, that is, the graphic vector model is updated first, and then the knowledge base graphic vector model is updated after a delay of several times (specifically, the preset number minus 1). In particular, when the number of consecutive updates of the graphic vector model is equal to 0, it means that the parameters of the graphic vector model have not been updated. At this time, the unupdated graphic vector model can also be copied as the knowledge base graphic vector model to realize the initialization of the model and ensure that the initial update of the model parameters is carried out smoothly.

[0061] Optionally, the large image-text model and the image-text vector model are obtained after multiple parameter updates, which means that the large image-text model obtained in one update will be used as the large image-text model to be trained in the next update, and the image-text vector model obtained in one update will be used as the image-text vector model to be trained in the next update. Step S105 specifically includes: when the synchronous update condition is met, the parameters of the large image-text model to be trained are updated according to the loss value, and the parameters of the image-text vector model to be trained are updated according to the industry answer, to obtain the large image-text model and the image-text vector model; when the synchronous update condition is not met, the parameters of the large image-text model to be trained are updated according to the loss value, and the parameters of the image-text vector model to be trained are kept unchanged, to obtain the large image-text model and the image-text vector model. By configuring the synchronous update condition, the large image-text model and the image-text vector model can be updated simultaneously only when the condition is met, otherwise only the large image-text model needs to be updated, which can reasonably control the amount of calculation for model updates and help optimize the computing resource consumption of electronic devices.

[0062] As an example, the synchronous update conditions include at least one of the following: the number of consecutive updates of the large image and text model reaches a number threshold, and the time interval since the last update of the image and text vector model has passed a threshold. Both the number threshold and the time threshold can be set as needed. It should be understood that both are positive numbers, and the number threshold can be greater than 1, so that the large image and text model is updated more frequently than the image and text vector model. In other words, the large image and text model is updated first, and then the image and text vector model is updated after a delay of several times (specifically, the number threshold minus 1). In particular, if the number threshold is set to 1, it means that the parameters of the large image and text model and the image and text vector model are always updated synchronously.

[0063] In a specific embodiment, the multimodal retrieval enhancement generation system is divided into three main parts: knowledge base vectorization, knowledge base retrieval, and answer generation. The system uses the pre-trained image-text vector model and the image-text large model as the basic model to build the multimodal retrieval enhancement generation system framework. First, the industry knowledge base data is vectorized using the knowledge base image-text vector model and stored in the segment feature vector database. The image-text vector model is used to target industry problems, such as Figure 4 As shown in the figure, industry questions are searched in the industry knowledge base and the corresponding fragment feature vector database, and the relevant Top-K industry knowledge fragments are obtained as the search results. The large picture and text model is used to generate answers. For general questions, no search is performed and general answers are generated directly. For industry questions, the following is performed first: Figure 4 The search shown in the figure is then combined with the search results to generate industry answers. Specifically, Figure 6 As shown, when it is determined that there are search results, it means that the question (query) is an industry question. At this time, the question and the search results are input into the picture and text model as prompts to generate an industry answer; when there are no search results, it means that the question is a general question. If there is no refusal to answer script set in advance, the question is input into the picture and text model as a prompt to generate a general answer. If there is a refusal to answer script set in advance, the refusal to answer script is directly used as the answer.

[0064] Figure 7 is a block diagram of a joint training device for a graph-text vector model and a graph-text large model according to an exemplary embodiment of the present disclosure. Figure 7 The joint training device 700 of the image-text vector model and the image-text large model includes an acquisition unit 701, a general question-answering unit 702, an industry question-answering unit 703, a determination unit 704, and an update unit 705.

[0065] The acquisition unit 701 can acquire a general question and answer sample set, an industry question and answer sample set, and an industry knowledge base.

[0066] The general question and answer unit 702 can use the large image and text model to be trained to process the general question samples in the general question and answer sample set to obtain a general answer.

[0067] The industry question and answer unit 703 can use the image and text vector model to be trained, the industry knowledge base and the image and text large model to be trained to process the industry question samples in the industry question and answer sample set to obtain industry answers.

[0068] Optionally, the industry question and answer unit 703 can also: use the image-text vector model to be trained to extract the question feature vector of the industry question sample; retrieve a preset number of industry knowledge fragments that match the question feature vector from the industry knowledge base; use the image-text large model to be trained to process the industry question sample and a preset number of industry knowledge fragments to obtain industry answers.

[0069] Optionally, the industry question and answer unit 703 can also: use the large graphic and text model to be trained to process industry question samples and a preset number of industry knowledge fragments to obtain a preset number of candidate industry answers, wherein each candidate industry answer corresponds one-to-one to each industry knowledge fragment; determine a candidate industry answer from the preset number of candidate industry answers as the industry answer.

[0070] The determination unit 704 may determine the loss value based on the general answers, the general question and answer sample set, the industry answers, and the industry question and answer sample set.

[0071] The updating unit 705 can update the parameters of the image-text large model to be trained according to the loss value, and update the parameters of the image-text vector model to be trained according to the industry answer, to obtain the image-text large model and the image-text vector model.

[0072] Optionally, the update unit 705 can also: determine the industry knowledge fragments corresponding to the industry answers as positive samples, and determine other industry knowledge fragments among a preset number of industry knowledge fragments except the positive samples as negative samples; based on the positive samples and negative samples, use the comparative learning method to update the parameters of the image-text vector model to be trained.

[0073] Optionally, the large image-text model and the image-text vector model are obtained after multiple parameter updates, wherein the update unit 705 can also: when the synchronous update conditions are met, update the parameters of the large image-text model to be trained according to the loss value, and update the parameters of the image-text vector model to be trained according to the industry answer, to obtain the large image-text model and the image-text vector model; when the synchronous update conditions are not met, update the parameters of the large image-text model to be trained according to the loss value, and keep the parameters of the image-text vector model to be trained unchanged, to obtain the large image-text model and the image-text vector model.

[0074] Optionally, the large image-text model and the image-text vector model are obtained after multiple parameter updates, wherein the joint training device 700 of the image-text vector model and the large image-text model according to the exemplary embodiment of the present disclosure also includes an extraction unit (not shown in the figure). The extraction unit can determine the latest image-text vector model as the knowledge base image-text vector model when the preset update conditions are met, and use the knowledge base image-text vector model to extract the fragment feature vectors of each industry knowledge fragment in the industry knowledge base, wherein the fragment feature vector is used to calculate the matching degree of the corresponding industry knowledge fragment and the problem feature vector.

[0075] Optionally, the preset update condition includes at least one of the following: the number of updates of the image-text vector model is equal to 0 or greater than or equal to a preset number, and a preset time has passed since the last update of the knowledge base image-text vector model.

[0076] Regarding the apparatus in the above embodiment, the specific manner in which each unit performs operations has been described in detail in the embodiment of the method, and will not be elaborated on here.

[0077] Figure 8 FIG. 8 is a structural block diagram of an electronic device 800 according to an exemplary embodiment of the present disclosure.

[0078] Reference Figure 8 The electronic device 800 includes: at least one memory 801 and at least one processor 802, wherein the at least one memory 801 stores computer executable instructions. When the computer executable instructions are executed by the at least one processor 802, the at least one processor is prompted to execute the joint training method of the image-text vector model and the image-text large model as described in the above exemplary embodiment.

[0079] As an example, electronic device 800 may be a PC, tablet device, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, electronic device 800 is not necessarily a single electronic device 800, but may also be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction set) individually or in combination. Electronic device 800 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device 800 that is interconnected locally or remotely (e.g., via wireless transmission) via an interface.

[0080] In electronic device 800, processor 802 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor 802 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0081] The processor 802 can execute instructions or codes stored in the memory 801, wherein the memory 801 can also store data. Instructions and data can also be sent and received over the network via the network interface device, wherein the network interface device can use any known transmission protocol.

[0082] The memory 801 can be integrated with the processor 802, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. Furthermore, the memory 801 can include a separate device, such as an external disk drive, a storage array, or any other storage device usable by a database system. The memory 801 and the processor 802 can be operatively coupled or can communicate with each other, for example, via an I / O port, a network connection, or the like, such that the processor 802 can access files stored in the memory.

[0083] In addition, the electronic device 800 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 800 may be connected to each other via a bus and / or a network.

[0084] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein the instructions, when executed by at least one processor, prompt the at least one processor to execute the joint training method of the image-text vector model and the image-text large model as described in the above exemplary embodiment. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as a multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device configured to store a computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0085] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, comprising computer instructions, which, when executed by at least one processor, execute the joint training method of the image-text vector model and the image-text large model as described in the above exemplary embodiments.

[0086] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

[0087] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A joint training method for a graphic-text vector model and a graphic-text large model, characterized in that: The method comprises: Obtain general Q&A sample sets, industry Q&A sample sets, and industry knowledge base; Using the large image-text model to be trained to process general question samples in the general question-answering sample set to obtain general answers; Using the image-text vector model to be trained, the industry knowledge base, and the image-text large model to be trained, processing industry question samples in the industry question and answer sample set to obtain industry answers; Determining a loss value based on the general answer, the general question and answer sample set, the industry answer, and the industry question and answer sample set; The parameters of the large image-text model to be trained are updated according to the loss value, and the parameters of the image-text vector model to be trained are updated according to the industry answer to obtain the large image-text model and the image-text vector model.

2. The method according to claim 1, wherein The method of using the image-text vector model to be trained, the industry knowledge base, and the image-text large model to be trained to process industry question samples in the industry question and answer sample set to obtain industry answers includes: Using the image-text vector model to be trained, extracting the problem feature vector of the industry problem sample; Retrieving a preset number of industry knowledge fragments matching the problem feature vector from the industry knowledge base; The industry question samples and the preset number of industry knowledge fragments are processed using the large image and text model to be trained to obtain the industry answers.

3. The method according to claim 2, wherein The step of using the large image-text model to be trained to process the industry question sample and the preset number of industry knowledge fragments to obtain the industry answer includes: Using the large image-text model to be trained, processing the industry question sample and the preset number of industry knowledge fragments, to obtain the preset number of candidate industry answers, wherein each candidate industry answer corresponds to each industry knowledge fragment one by one; A candidate industry answer is determined from the preset number of candidate industry answers as the industry answer.

4. The method according to claim 3, wherein The updating of the parameters of the image-text vector model to be trained according to the industry answer includes: Determine the industry knowledge fragment corresponding to the industry answer as a positive sample, and determine the other industry knowledge fragments among the preset number of industry knowledge fragments except the positive sample as negative samples; Based on the positive samples and the negative samples, a contrastive learning method is used to update the parameters of the image-text vector model to be trained.

5. The method according to claim 2, wherein The image-text large model and the image-text vector model are obtained after multiple parameter updates, wherein the method further includes: When the preset update conditions are met, the latest graphic vector model is determined as the knowledge base graphic vector model, and the knowledge base graphic vector model is used to extract the fragment feature vectors of each industry knowledge fragment in the industry knowledge base, wherein the fragment feature vector is used to calculate the matching degree between the corresponding industry knowledge fragment and the problem feature vector.

6. The method according to claim 5, wherein The preset update condition includes at least one of the following: the number of updates of the image-text vector model is equal to 0 or greater than or equal to a preset number, and a preset time has passed since the last update of the knowledge base image-text vector model.

7. The method according to claim 1, wherein The large image-text model and the image-text vector model are obtained after multiple parameter updates, wherein the parameters of the large image-text model to be trained are updated according to the loss value, and the parameters of the image-text vector model to be trained are updated according to the industry answer, to obtain the large image-text model and the image-text vector model, including: When the synchronous update condition is met, the parameters of the image-text large model to be trained are updated according to the loss value, and the parameters of the image-text vector model to be trained are updated according to the industry answer, to obtain the image-text large model and the image-text vector model; When the synchronous update condition is not met, the parameters of the large image-text model to be trained are updated according to the loss value, and the parameters of the image-text vector model to be trained are kept unchanged to obtain the large image-text model and the image-text vector model.

8. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer-executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, they prompt the at least one processor to execute the joint training method of the image-text vector model and the image-text large model as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by at least one processor, they prompt the at least one processor to execute the joint training method of the image-text vector model and the image-text large model as described in any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by at least one processor, they prompt the at least one processor to execute the joint training method of the image-text vector model and the image-text large model as described in any one of claims 1 to 7.

Images