A knowledge updating method for multimodal models based on knowledge representation and dynamic prompting
Through the multimodal model knowledge update method based on knowledge representation and dynamic prompts, the problem of multimodal model refinement representation in knowledge update is solved, effective integration of new and old knowledge and model performance improvement is achieved, and knowledge representation and utilization are improved. efficiency.
Patent Information
- Application Number
- CN202411116647.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-15
AI Technical Summary
Multimodal models face the problem of difficult to achieve in terms of knowledge updates, which affects the overall performance of the model.
Using a multimodal model knowledge update method based on knowledge representation and dynamic prompts, we can collect and label new knowledge data, build an updated data set, and use a meta controller to generate dynamic prompts for multiple rounds of training, integrate new and old knowledge, and improve knowledge representation. fineness and utilization efficiency.
Effectively integrate new and old knowledge, maintain the continuous learning ability of the model, improve the overall performance of the multimodal model, and improve the efficiency of knowledge representation and utilization through dynamic prompts.
Smart Images

Figure CN118627610B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of deep learning and natural language understanding, and in particular to a multimodal model knowledge updating method based on knowledge representation and dynamic prompting. Background Art
[0002] The development of knowledge representation of multimodal models is mainly reflected in models that combine multimodal information such as text, images, videos, and audio for training. The development of this model not only improves the basic capabilities of the model, such as supported input / output forms, performance indicators, etc., but also explores interesting issues and directions such as multimodal hallucinations, multimodal contextual learning, multimodal thought chains, and LLM-assisted visual reasoning.
[0003] Although multimodal models have made significant progress in knowledge representation and application, there are still many challenges in efficiently updating knowledge. Traditional methods often regard knowledge as a whole, which makes it difficult to achieve refined knowledge representation and easily affects the overall performance of the model. Summary of the invention
[0004] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a multimodal model knowledge updating method based on knowledge representation and dynamic prompts, which can effectively integrate new and old knowledge, maintain the continuous learning ability of the model, thereby improving the overall performance of the polymorphic model, and at the same time help to improve the precision and utilization efficiency of knowledge representation through dynamic prompts.
[0005] In order to implement the above technical solution, in a first aspect, the present invention provides a multimodal model knowledge updating method based on knowledge representation and dynamic prompting, comprising the following steps:
[0006] 1. A multimodal model knowledge updating method based on knowledge representation and dynamic prompting, characterized in that it comprises the following steps:
[0007] Step 1: Collect and label the newly added multi-granular knowledge data;
[0008] Step 2: Select multi-granularity knowledge data related to the newly added multi-granularity knowledge data from the knowledge base as the retained knowledge data sample;
[0009] Step 3: Based on the labeled new knowledge data and the retained multi-granularity knowledge data samples, an updated knowledge dataset is constructed, and the updated knowledge dataset is divided into a training set and a validation set;
[0010] Step 4: vectorize the multi-granularity knowledge data in the training set to generate a training knowledge data vector of uniform dimension;
[0011] Step 5: Initialize the optimizer;
[0012] Step 6: Based on the generated training knowledge data vector of unified dimension, multiple rounds of training are performed on the multimodal model that has completed the initialization optimizer to update the multimodal model and the knowledge base, wherein the step 6 includes: based on the generated training knowledge data vector of unified dimension, using the meta-controller architecture to generate dynamic prompts;
[0013] The generating of dynamic prompts using a meta-controller architecture based on the generated training knowledge data vector of uniform dimension includes:
[0014] The generated training knowledge data vectors of uniform dimension are fused and interacted to obtain fused training knowledge data vectors;
[0015] Generate a dynamic prompt vector based on the fused training knowledge data vector;
[0016] The generating of the dynamic prompt vector based on the fusion training knowledge data vector comprises:
[0017] Encoding the fusion training knowledge vector to obtain an encoded fusion training knowledge vector;
[0018] Processing the obtained encoded fusion training knowledge vector to generate a latent vector;
[0019] Based on the latent vector and different tasks, a dynamic hint vector is output.
[0020] Furthermore, the method further comprises:
[0021] Pre-build a knowledge base with multi-granularity knowledge ontology structure.
[0022] Furthermore, the step 2 includes:
[0023] Filter out multi-granularity knowledge data related to the newly added multi-granularity knowledge data from the knowledge base;
[0024] Based on the importance and usage frequency of knowledge data of different granularities among the screened multi-granularity knowledge data, the selection ratio of knowledge data of different granularities is determined;
[0025] Based on the determined selection ratio, each type of granularity knowledge data is selected from the screened multi-granularity knowledge data as a retained multi-granularity knowledge data sample.
[0026] Furthermore, the step six comprises:
[0027] Use meta controllers to generate dynamic prompts;
[0028] Perform forward propagation and calculate the total loss of the multimodal model; the total loss includes: update loss, retention loss, knowledge distillation loss, and regularization loss;
[0029] The multimodal model parameters and meta-controller parameters are updated through back-propagation, and the knowledge base is updated.
[0030] Furthermore, the method further comprises:
[0031] During multiple rounds of training, the performance of the multimodal model is regularly evaluated.
[0032] Furthermore, the total loss is calculated as follows:
[0033]
[0034] in, : represents the total loss function; : Update loss; : retain loss; : Knowledge distillation loss; : regularization loss; λ1, λ2, λ3 and λ4 are weight coefficients, which are tuned by the validation set.
[0035] In a second aspect, the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned multimodal model knowledge updating method based on knowledge representation and dynamic prompting.
[0036] The beneficial effects of the present invention are:
[0037] The present invention discloses a method of selecting knowledge data from a pre-constructed knowledge base as retained knowledge data samples, constructing an updated data set based on labeled new knowledge data and knowledge data samples, vectorizing the knowledge data in the training set to generate training knowledge data vectors of uniform dimensionality, and performing multiple rounds of training on a multimodal model for which an optimizer has been initialized based on the generated training knowledge data vectors of uniform dimensionality to update the multimodal model and the knowledge base, thereby realizing the integration of new and old knowledge data and helping to maintain the continuous learning ability of the model.
[0038] At the same time, dynamic prompts are used to improve the efficiency of knowledge representation and utilization. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0040] Figure 1The present invention is a flowchart of a multimodal model knowledge updating method based on knowledge representation and dynamic prompting.
[0041] Figure 2 The flowchart of constructing the knowledge base of the present invention. DETAILED DESCRIPTION
[0042] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0043] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present invention. Unless otherwise specified, each technical and scientific term used in this embodiment has the same meaning as commonly understood by those skilled in the art to which the present invention belongs.
[0044] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0045] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention and should not be understood as limitations on the present invention.
[0046] In the present invention, terms such as "fixed connection", "connected", "connection", etc. should be understood in a broad sense, indicating that it can be fixedly connected, integrally connected or detachably connected; it can be directly connected or indirectly connected through an intermediate medium. Relevant scientific research or technical personnel in this field can determine the specific meanings of the above terms in the present invention according to specific circumstances, and they should not be understood as limiting the present invention.
[0047] Among them, the optimizer is used to optimize the parameters of the model. After the forward calculation is completed, after obtaining the gradient of the parameters according to the loss, the optimizer will use the gradient to update and calculate the network parameters of the model so that the loss of the model can be minimized in the subsequent calculation. The optimizer fits the training data by continuously updating the model parameters, so that it performs well on new data.
[0048] Embodiment 1:
[0049] like Figure 1As shown, this embodiment provides a multimodal model knowledge updating method based on knowledge representation and dynamic prompting, including the following steps (such as Figure 2 shown):
[0050] S1: Pre-build a multi-granularity knowledge ontology as a knowledge base for storing multi-granularity knowledge data.
[0051] Among them, multi-granularity refers to different levels of knowledge representation, such as disease, lung disease, pneumonia, which is equivalent to 3 levels. The multi-granular knowledge data in the knowledge base consists of two parts, the external knowledge base and the multimodal model. The external knowledge base stores long-term stable knowledge, and the multimodal model contains short-term new knowledge.
[0052] S11: Acquire multi-granularity knowledge data from various data sources.
[0053] The multi-granularity data includes: classes representing entity types, attributes including data attributes and object attributes; individuals representing specific instances, and rules defined using, for example, SWRL (Semantic Web Rule Language).
[0054] S12: Label multi-granularity knowledge data.
[0055] The knowledge data in the knowledge base are labeled according to their properties. For example, if the knowledge data is: disease, it is labeled as a class.
[0056] S13: Annotate the credibility, source, and timeliness of each granularity of the acquired multi-granularity knowledge data so that the multimodal model can correctly evaluate and use this information when integrating it.
[0057] In the construction process, a graph database (such as Neo4j) can be used to store the knowledge structure. This embodiment takes the medical field as an example:
[0058] (1) Classes are usually represented as nodes with specific labels, which contain the class name and related descriptive information, indicating the entity type. For example, the following classes can be defined: Disease, Symptom, Treatment, Medication, and Organ.
[0059] (2) Attributes: They are divided into data attributes and object attributes. Attributes can be stored as independent nodes, containing information such as attribute name, data type, and applicable scope.
[0060] For example, data attributes: Name: the name of the disease or drug, Dosage: the dosage of the drug.
[0061] Object attributes: Symptom: the relationship between disease and symptoms, treatedBy: the relationship between disease and treatment methods, affects: the organs affected by the disease.
[0062] (3) Individuals are specific instances of classes and are stored in the database as nodes with corresponding class labels and specific attribute values.
[0063] For example, Pneumonia (as an instance of the Disease class), Fever (as an instance of the Symptom class), Remdesivir (as an instance of the Medication class).
[0064] (4) Rules are used to express complex logical relationships and reasoning rules. In the database, rules are stored as special nodes, which contain rule name, description, formal rule expression (such as SWRL format), and metadata such as creation and update date. Rules can be associated with related classes or attributes to indicate their scope of application.
[0065] For example, you can define a rule like: "If a patient has fever and dry cough, and has a recent international travel history, then the patient may be a suspected case of pneumonia."
[0066] This structured knowledge base allows us to flexibly store complex knowledge networks. Searching for knowledge can be done through graph query languages such as Cypher, for example, to find all symptoms of a specific disease. Knowledge updates can be done by adding new nodes (such as new classes, attributes, individuals, or rules), creating new relationships, or modifying the properties of existing nodes.
[0067] S2: Collect new multi-granularity knowledge data and label the new multi-granularity knowledge data.
[0068] S21: Acquire multi-granularity knowledge data from multiple reliable data sources as newly added multi-granularity knowledge data.
[0069] For example, if new knowledge data in the medical field is added, the latest research results, clinical guideline update information, and newly discovered disease information are obtained from reliable medical sources as the new knowledge data.
[0070] S22: Label the acquired new knowledge data with increased granularity.
[0071] A: Classify the newly added multi-granularity knowledge according to its nature.
[0072] Among them, the newly added categories of multi-granular knowledge data include: classes (such as newly discovered viruses), attributes (such as new symptoms), relationships (such as new etiology associations) and rules (such as updated diagnostic criteria).
[0073] B: Label the credibility, source, and timeliness of the newly added multi-granular knowledge data so that the model can correctly evaluate and use this information when integrating it.
[0074] S3: Select multi-granularity knowledge data related to the newly added multi-granularity knowledge data from the knowledge base as the retained multi-granularity knowledge data sample.
[0075] It should be noted that the multi-granularity knowledge data retained in the multi-granularity knowledge data sample is the multi-granularity knowledge data stored in the knowledge base, which can also be called old multi-granularity knowledge data or original multi-granularity knowledge data.
[0076] S31: Filtering out multi-granularity knowledge data related to the newly added multi-granularity knowledge data from the knowledge base;
[0077] S32: Determine the selection ratio of knowledge data of different granularities based on the importance and usage frequency of knowledge data of different granularities in the screened multi-granularity knowledge data.
[0078] For basic medical knowledge or knowledge data of common diseases, more knowledge data may be selected as samples, while for old information that may be updated or replaced by new knowledge, fewer knowledge data may be selected as samples.
[0079] In addition, when determining the selection ratio of each granularity of knowledge data in the knowledge data sample, it is also necessary to consider the balanced distribution in various knowledge fields and knowledge types to maintain the performance of the model on comprehensive medical knowledge.
[0080] S33: Based on the determined selection ratio, each type of granularity knowledge data is selected from the screened multi-granularity knowledge data as a retained multi-granularity knowledge data sample.
[0081] S4: Based on the labeled new increased granularity knowledge data and the retained multi-granularity knowledge data samples, an updated knowledge dataset is constructed, and the updated knowledge dataset is divided into a training set and a validation set.
[0082] It should be noted that each piece of knowledge data in the knowledge base has been annotated in advance, so all the knowledge data in the constructed update data set have been annotated.
[0083] S5: Vectorize the multi-granularity knowledge data in the training set to generate training knowledge data vectors of uniform dimension.
[0084] A: Use the Transformer structure to encode multi-granularity knowledge data representing entity categories to generate knowledge data vectors of entity types.
[0085] Specifically, the self-attention mechanism of Transformer can capture the internal complex structure and contextual information of entity type knowledge data, and is particularly suitable for processing entity descriptions with rich semantics. The encoder receives entity type knowledge data (e.g., text description) as input and outputs a fixed-dimensional entity type knowledge data vector.
[0086] In addition to text descriptions, Transformer can also receive multimodal inputs such as images and audio.
[0087] B: Use attribute encoders such as multi-layer perceptron (MLP) to encode attributes and process different types of attribute data such as numerical values, categories, and time series to generate knowledge data of attribute categories.
[0088] Specifically, MLP is suitable for processing knowledge data of attribute categories with relatively simple structures. It takes the received attribute names and values as input and generates knowledge data vectors of attribute categories through multiple layers of nonlinear transformations.
[0089] C: Use a relation encoder such as a graph attention network (GAT) to encode the knowledge data of the relation category to obtain a knowledge data vector of the relation category.
[0090] Specifically, GAT can effectively capture the complex relationships and graph structure information between entities, take the vector representation of connected entities as input, and learn through the graph attention mechanism to obtain the knowledge data vector of the relationship category.
[0091] D: Use a rule encoder such as a recurrent neural network (RNN) to encode the knowledge data of the rule category to obtain a knowledge data vector of the rule category.
[0092] Specifically, RNN is suitable for processing rule information with sequentiality and logical structure. It takes the received logical expression of the rule as input and learns the rule through a recursive structure to obtain a knowledge data vector of the rule category.
[0093] E: The entity encoder, attribute encoder, relationship encoder and rule encoder jointly map knowledge data of different granularities into a unified vector space to obtain new knowledge data vectors of unified dimension.
[0094] The vectorization processing of the multi-granularity knowledge data in the training set to generate a training knowledge data vector of uniform dimension also includes:
[0095] If the entity encoder, attribute encoder, relation encoder, and rule encoder cannot output a new knowledge data vector of uniform dimension, an additional linear layer is required for dimensionality adjustment. Then a multi-head attention mechanism is used to fuse the outputs of different encoders. Finally, a feedforward network can be used for further feature extraction. A unified vector representation containing multi-granularity knowledge information is obtained.
[0096] S6: Initialize the optimizer for the multimodal model parameters and meta-controller parameters respectively.
[0097] Specifically, select a suitable optimization algorithm such as Adam or AdamW, set parameters such as learning rate and weight decay, and initialize the optimizer for the multimodal model parameters and meta-controller parameters respectively to perform differentiated parameter updates.
[0098] Among them, the multimodal model parameters are: learning rate: 1e-5, weight decay: 0.01, Beta1: 0.9, Beta2: 0.999 (hyperparameters of AdamW optimizer);
[0099] Meta-controller parameters: learning rate: 5e-5, weight decay: 0.001, Beta1: 0.9, Beta2: 0.999.
[0100] S7: Based on generating a training knowledge data vector of uniform dimension, multiple rounds of training are performed on the multimodal model that has completed the initialization optimizer to update the multimodal model and the knowledge base.
[0101] Each round of training includes:
[0102] S71: Generate dynamic prompts using a meta-controller architecture based on the generated training knowledge data vector of uniform dimensionality.
[0103] S71-1: Fuse and interact the generated training knowledge data vectors of unified dimension to obtain a fused training knowledge data vector.
[0104] The generated training knowledge data vector of uniform dimension is input into a model with a multi-head cross attention layer and a Transformer-XL architecture, wherein the multi-head cross attention layer is used to fuse and interact the knowledge data vectors of different modalities in the generated training knowledge data vector of uniform dimension to obtain a fused training knowledge data vector. The Transformer-XL architecture is used to capture the long-distance dependencies and complex patterns between the knowledge data vectors of each modality in the fused training knowledge data vector.
[0105] For example, take the medical multimodal large model as an example. The updated knowledge data is described as a task. For example, the task is "update knowledge about new respiratory diseases", and the input data includes multimodal information such as text descriptions of new viruses, electron microscope images, and gene sequence data. Among them, the multimodal information such as the text description of the new virus, electron microscope images, and gene sequence data are vectorized and input into the model with a multi-head cross-attention layer and a Transformer-XL architecture. The multi-head cross-attention layer fuses the vectorized multimodal information such as the text description of the new virus, electron microscope images, and gene sequence data to obtain a multimodal fusion information vector.
[0106] S71-2: Generate a dynamic prompt vector based on the fused training knowledge data vector.
[0107] Specifically, a conditional variational autoencoder constructed by Transformer is used to process the fused training knowledge data vector to generate a dynamic prompt vector, wherein the conditional variational autoencoder constructed by Transformer includes an encoder, a latent space mapping, and a decoder. The specific steps are as follows:
[0108] A: The encoder encodes the fused training knowledge data vector to obtain an encoded fused training knowledge data vector, and inputs the encoded fused training knowledge data vector into the latent space mapping.
[0109] B: Latent space mapping processes the encoded fused training knowledge data vector to generate the output mean vector (μ) and log variance vector (log σ²), where σ is the population standard deviation, and uses the mean vector (μ) and the population standard deviation vector (σ²) to generate the latent vector z = μ + ε * σ, where ε is the noise sampled from the standard normal distribution.
[0110] C: The decoder uses the Transformer decoder structure and outputs a dynamic hint vector based on the latent vector z and the input conditional information (such as different tasks or different task descriptions). The conditional information (such as different tasks or different task descriptions) is processed into a vector through the embedding layer.
[0111] The meta-controller generates dynamic prompts in real time based on different tasks and input data (i.e., vectorized tasks and input data). For example, the above data may generate the following dynamic prompt: "Integrate text descriptions, image features, and gene sequences of new viruses to update disease diagnosis standards and treatment plans."
[0112] The generated dynamic prompts are vectorized to generate dynamic prompt vectors, which are then spliced with the input sequences of each modality (i.e., knowledge data vectors of uniform dimension) and injected into the medical big model. When processing new multimodal inputs, the medical big model will perform the following operations based on the injected dynamic prompts: update the disease knowledge base to add the characteristics of new viruses; adjust the diagnostic algorithm to consider new symptoms and detection indicators; update treatment recommendations to include the latest methods for new viruses. As more relevant medical data and cases are processed, the medical multimodal big model will continue to optimize.
[0113] S72: Perform forward propagation and calculate the total loss of the multimodal model.
[0114] Among them, the total loss includes update loss, retention loss, knowledge distillation loss and regularization loss.
[0115] Specifically, the calculation formula of the total loss function is as follows:
[0116]
[0117] in, : represents the total loss function; : New knowledge data set update loss. This part can use cross entropy loss to ensure that the model correctly learns the new knowledge data; : Preservation loss, through KL divergence to ensure that the multimodal model is similar to the teacher model output when processing knowledge data in the knowledge base, : Knowledge distillation loss, constraining the output distribution before and after the update; : Regularization loss to prevent overfitting; λ1, λ2, λ3 and λ4 are weight coefficients, which are tuned by the validation set.
[0118]
[0119] in, : The multimodal model predicts the output of the new data; : The true label of the newly added data.
[0120]
[0121] in, : The prediction output of the current multimodal model for the knowledge data in the current knowledge base; : The predicted output of the teacher model (usually the previous version of the multimodal model) on the knowledge data in the previous knowledge base; T: Temperature parameter, used to soften the probability distribution, usually greater than 1, a higher T value will make the distribution smoother; T2: used to scale the KL divergence to balance the impact of temperature on the loss.
[0122]
[0123] in, : The prediction output of the multimodal model for the newly added knowledge data; : The multimodal model predicts the output of the knowledge data in the current knowledge base. Here we compare the difference in the output distribution of the model on the newly added knowledge data and the original knowledge data.
[0124]
[0125] Where, θ: all parameters of the multimodal model.
[0126] For example, taking the medical diagnosis task as an example, suppose we have a new knowledge data containing data of 5 patients, each of whom may suffer from one of 3 diseases (cold, flu or pneumonia).
[0127] (True label): This is the result that the doctor has confirmed, represented by a digital code: [0, 1, 2,1, 0], where 0 represents a cold, 1 represents influenza, and 2 represents pneumonia.
[0128] (Multimodal model prediction output): This is the probability predicted by the multimodal model for each patient to belong to each disease:
[0129] [0.7, 0.2, 0.1], / / Patient 1
[0130] [0.1, 0.8, 0.1], / / Patient 2
[0131] [0.2, 0.3, 0.5], / / Patient 3
[0132] [0.3, 0.6, 0.1], / / Patient 4
[0133] [0.6, 0.3, 0.1] / / Patient 5
[0134] Each inner array represents a patient, and the three numbers in the array represent the model's prediction of the probability that the patient has a cold, flu, and pneumonia, respectively.
[0135] The above calculations help to balance the learning of new knowledge data and the retention of original knowledge data.
[0136] S73: Update the multimodal model parameters and meta-controller parameters through back-propagation, and update the knowledge base regularly.
[0137] Specifically, it includes: clearing previous gradients, back-propagating gradients, gradient clipping, and updating parameters.
[0138] Furthermore, the method further comprises:
[0139] During multiple rounds of training, the performance of the multimodal model is regularly evaluated.
[0140] Specifically, evaluation indicators are set, where the evaluation indicators include: accuracy and F1 (the harmonic mean of accuracy and recall), etc.
[0141] The performance of the multimodal model after multiple rounds of training is evaluated through the validation set to obtain the accuracy and F1 of the multimodal model;
[0142] According to the importance of the task, the obtained accuracy and F1 are weighted to obtain the comprehensive score of the multimodal model.
[0143] Embodiment 2:
[0144] This embodiment provides a multimodal model knowledge updating system based on knowledge representation and dynamic prompting, including:
[0145] The knowledge data collection module is used to collect and annotate the newly added multi-granular knowledge data;
[0146] The sample selection module is used to select multi-granularity knowledge data related to the newly added multi-granularity knowledge data from the knowledge base as the retained knowledge data sample.
[0147] The data set construction module is used to construct an updated knowledge data set based on the labeled new knowledge data and the retained multi-granularity knowledge data samples, and divide the updated knowledge data set into a training set and a validation set.
[0148] A vectorization module is used to vectorize the multi-granularity knowledge data in the training set to generate training knowledge data vectors of uniform dimension;
[0149] The updating module is used to perform multiple rounds of training on the multimodal model for which the optimizer has been initialized based on generating a training knowledge data vector of uniform dimension, so as to update the multimodal model and the knowledge base.
[0150] The optimization module is used to initialize the optimizer for the multimodal model parameters and the meta-controller parameters respectively.
[0151] Knowledge base is used to store knowledge data of different granularities.
[0152] The knowledge base construction module is used to construct the knowledge base.
[0153] A query interface for querying results for input text processed using the updated multimodal model.
[0154] Embodiment 3:
[0155] This embodiment provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute a multimodal model knowledge updating method based on knowledge representation and dynamic prompting as described in Example 1.
[0156] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.
[0157] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.
[0158] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0159] In addition, it should be noted that the flowchart in the accompanying drawings shows the method of the embodiment of the present disclosure. In the corresponding description in the flowchart or block diagram in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be performed substantially in parallel, and they can sometimes be performed in the opposite order, which can depend on the functions involved. Each block in the block diagram and / or flow chart, and the combination of blocks in the block diagram and / or flow chart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0160] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A multimodal model knowledge updating method based on knowledge representation and dynamic prompting, characterized in that: The following steps are involved: Step 1: Collect and label the newly added multi-granularity knowledge data; Step 2: Select multi-granularity knowledge data related to the newly added multi-granularity knowledge data from the knowledge base as the retained knowledge data sample; Step 3: Based on the labeled new knowledge data and the retained multi-granularity knowledge data samples, an updated knowledge dataset is constructed, and the updated knowledge dataset is divided into a training set and a validation set; Step 4: vectorize the multi-granularity knowledge data in the training set to generate a training knowledge data vector of uniform dimension; Step 5: Initialize the optimizer; Step 6: Based on the training knowledge data vector generated with unified dimension, multiple rounds of training are performed on the multimodal model that has completed the initialization optimizer to update the multimodal model and knowledge base; Wherein, the step six comprises: generating a dynamic prompt vector using a meta-controller based on the generated training knowledge data vector of uniform dimension; The generating a dynamic prompt vector using a meta-controller based on the generated training knowledge data vector of uniform dimension comprises: The generated training knowledge data vectors of unified dimension are fused and interacted to obtain fused training knowledge data vectors; based on the fused training knowledge data vectors, a dynamic prompt vector is generated; the generating the dynamic prompt vector based on the fused training knowledge data vector includes: Encoding the fused training knowledge vector to obtain an encoded fused training knowledge vector; processing the obtained encoded fused training knowledge vector to generate a latent vector; outputting a dynamic prompt vector based on the latent vector and different tasks; The step of fusing and interacting the generated training knowledge data vectors of uniform dimension to obtain a fused training knowledge data vector includes: inputting the generated training knowledge data vectors of uniform dimension into a model having a multi-head cross attention layer and a Transformer-XL architecture, wherein the multi-head cross attention layer is used to fuse and interact knowledge data vectors of different modalities in the generated training knowledge data vectors of uniform dimension to obtain a fused training knowledge data vector; the Transformer-XL architecture is used to capture long-distance dependencies and complex patterns between knowledge data vectors of different modalities in the fused training knowledge data vectors; The generating of the dynamic prompt vector based on the fused training knowledge data vector includes: processing the fused training knowledge data vector using a conditional variational autoencoder constructed by a Transformer to generate a dynamic prompt vector, wherein the conditional variational autoencoder constructed by the Transformer includes an encoder, a latent space mapping, and a decoder.
2. The multimodal model knowledge updating method based on knowledge representation and dynamic prompting according to claim 1 is characterized in that: The method further comprises: Pre-build a knowledge base with multi-granularity knowledge ontology structure.
3. The multimodal model knowledge updating method based on knowledge representation and dynamic prompting according to claim 1 is characterized in that: The second step comprises: Filter out multi-granularity knowledge data related to the newly added multi-granularity knowledge data from the knowledge base; Based on the importance and usage frequency of knowledge data of different granularities among the screened multi-granularity knowledge data, the selection ratio of knowledge data of different granularities is determined; Based on the determined selection ratio, each type of granularity knowledge data is selected from the screened multi-granularity knowledge data as a retained multi-granularity knowledge data sample.
4. The multimodal model knowledge updating method based on knowledge representation and dynamic prompting according to claim 1 is characterized in that: The step six also includes: Perform forward propagation and calculate the total loss of the multimodal model; the total loss includes: update loss, retention loss, knowledge distillation loss, and regularization loss; The multimodal model parameters and meta-controller parameters are updated through back-propagation, and the knowledge base is updated.
5. The multimodal model knowledge updating method based on knowledge representation and dynamic prompting according to claim 1 is characterized in that: The method further comprises: During multiple rounds of training, the performance of the multimodal model is regularly evaluated.
6. The multimodal model knowledge updating method based on knowledge representation and dynamic prompting according to claim 4 is characterized in that: The total loss is calculated as follows: in, : represents the total loss function; : Update loss; : retain loss; : Knowledge distillation loss; : regularization loss; λ1, λ2, λ3 and λ4 are weight coefficients, which are tuned by the validation set.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the multimodal model knowledge updating method based on knowledge representation and dynamic prompting as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Knowledge base completion method based on multi-modal representation learning
CN112348191A
Domain knowledge updating system and method for large model
CN116719899A
Multi-modal knowledge graph unified representation learning framework
CN117035078A
Knowledge graph construction method and system
CN117952209A
Multi-task persistent evolution learning method based on multi-modal parameter extension
CN118094414A
