Multi-modal model training method and device, information processing method and device, computing equipment, storage medium and computer program product

Abstract

Provided are a multi-modal model training method, an information processing method and apparatus, a computing device, a storage medium and a computer program product, the multi-modal model training method comprising: inputting first multi-modal information into a plurality of multi-modal models to obtain first multi-modal features corresponding to the plurality of multi-modal models, the plurality of multi-modal models comprise a student model and at least one teacher model; for the multi-modal model, according to a first multi-modal feature corresponding to the multi-modal model, a first correlation degree set corresponding to the multi-modal model is determined, and the first correlation degree represents the correlation degree of the correlation relation between the first modal information and other first modal information; according to the plurality of first association degree sets, at least one first proximity corresponding to each of the plurality of first modal information is obtained, and the first proximity represents the proximity between the student model and the teacher model; and training a student model according to the at least one first proximity of each of the plurality of pieces of first modal information.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology. Specifically, it relates to a training method, information processing method, apparatus, computing device, storage medium, and computer program product for a multimodal model. Background Technology

[0002] A multimodal model (MM) refers to a model capable of simultaneously processing modal information corresponding to multiple modalities and utilizing the interactions between these modalities to handle tasks. Multimodal models possess strong generalization capabilities, such as zero-shot learning (ZSL), and hold the promise of becoming intelligent assistants for humans, propelling artificial intelligence towards Artificial General Intelligence (AGI). Furthermore, achieving AGI requires multimodal models to possess continuous learning (CL) capabilities. Therefore, multimodal models and continuous learning can be combined to achieve continuous learning within the multimodal model. Continuous learning refers to the technique of continuously learning knowledge from the current task while retaining knowledge from previous tasks within a sequence of tasks.

[0003] To achieve continuous learning of multimodal models, fine-tuning can be performed using multimodal information corresponding to sequential tasks. However, fine-tuning may weaken the zero-shot learning capability of multimodal models and may also lead to a gradual loss of performance gains during continuous learning. Therefore, how to alleviate the conflict between maintaining zero-shot learning capability and adapting to new information in the continuous learning of multimodal models has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides a training method, information processing method, apparatus, computing device, storage medium, and computer program product for a multimodal model, which can alleviate the conflict between maintaining zero-shot learning ability and adapting to new information in the continuous learning of multimodal models.

[0005] Firstly, this application provides a method for training a multimodal model, which may include: inputting first multimodal information into multiple multimodal models respectively to obtain first multimodal features corresponding to each of the multiple multimodal models. The first multimodal information includes at least one first modal information corresponding to each of the multiple modalities. The first multimodal features corresponding to the multimodal models include first modal features corresponding to each of the multiple first modal information. The multiple multimodal models include a student model and at least one teacher model. For each multimodal model, a first association degree set corresponding to the multimodal model is determined based on the first multimodal features corresponding to the multimodal model. The first association degree set includes a first association degree subset corresponding to each of the multiple first modal information. The first association degree subset corresponding to the first modal information includes at least one first association degree corresponding to the first modal information. The first association degree corresponding to the first modal information characterizes the degree of association between the first modal information and other first modal information. The other first modal information is any one of the first multimodal information other than the first modal information. Based on the first set of correlation degrees corresponding to each of the multiple multimodal models, at least one first proximity degree is obtained corresponding to each of the multiple first modal information. For example, for the first modal information and the teacher model, a subset of the first correlation degrees corresponding to the first modal information and the student model is determined, along with the proximity degree between the subsets of the first correlation degrees corresponding to the first modal information and the teacher model, thus obtaining the first proximity degree corresponding to the first modal information, the student model, and the teacher model. The first proximity degree characterizes the degree of proximity between the student model and the teacher model. The student model is trained based on at least one first proximity degree corresponding to each of the multiple first modal information. The multiple multimodal models are multimodal models in continuous learning. The student model is the multimodal model corresponding to the current task in the sequence task. The teacher model is a trained multimodal model. The first multimodal information is the multimodal information matched with the current task.

[0006] The multimodal model training method provided in this application, because the first correlation subset corresponding to the first modality information is constructed based on the first multimodal features corresponding to the first multimodal information, can fully mine the information carried by the first modality information itself, and the determination of the correlation between the first modality information does not depend on the data distribution. Therefore, it can not only utilize the information carried by the first modality information itself, but also integrate the correlation between the first modality information, facilitating knowledge preservation, i.e., maintaining the zero-shot learning capability of the multimodal model. In addition, zero-shot learning requires the use of the mapping information between known modality information and unknown modality information to obtain the representation of unknown modality information. Therefore, the correlation between the first modality information can help the multimodal model maintain its zero-shot learning capability while having sufficient degrees of freedom to adapt to new information. Based on this, since the student model is trained according to the first proximity corresponding to the first modal information, the student model, and the teacher model, the first proximity between the student model and the teacher model for the same first modal information can reflect the degree of proximity between the reasoning abilities of the student model and the teacher model. The teacher model is a multimodal model that has been trained in continuous learning. At least one teacher model can include a multimodal model with the ability to maintain zero-shot learning and adapt to new information. Therefore, based on the first proximity corresponding to the student model and at least one teacher model, the student model can learn knowledge from at least one teacher model. Thus, in the continuous learning of multimodal models, the conflict between maintaining zero-shot learning ability and adapting to new information can be alleviated.

[0007] In one possible implementation, determining the first association set corresponding to the multimodal model based on the first multimodal features corresponding to the multimodal model may include: obtaining the first association set corresponding to the multimodal model based on the first graph structure and the first multimodal features corresponding to the multimodal model. The first graph structure is constructed based on the first multimodal information. The first graph structure includes multiple first nodes and at least one first edge. The first nodes represent the first modal information. The first edge is used to connect two first nodes with a direct association relationship. The two first modal information with a direct association relationship have different modalities.

[0008] Graph structures are data structures used to describe relationships between objects. A graph structure can include a set of nodes and a set of edges. A node set can include multiple nodes, and an edge set can include at least one edge. An edge represents the relationship between two connected nodes. The nodes and edges in a graph structure can be understood as relationships between objects, which aligns perfectly with the concept of zero-shot learning. Zero-shot learning utilizes the mapping information between known and unknown modal information to obtain a representation of unknown modal information. Therefore, by obtaining the first correlation between the first modal information based on the first graph structure, the correlation between the first modal information can constrain the model parameters of the multimodal model, acting as a constraint term in the loss function, such as a regularization term, thereby guiding the loss function and the direction of information transmission. In other words, by introducing the representation of the first graph structure during the training process of the student model, the correlation between the first modal information constructed based on the first graph structure restricts the loss function and the direction of information transmission. This not only alleviates the conflict between maintaining zero-shot learning capability and adapting to new information but also enhances the adaptability and stability of the multimodal model in continuous learning.

[0009] In another possible implementation, the first correlation set includes a first correlation subset corresponding to each of the multiple first modal information pieces. The first correlation subset includes at least one of the following: a first direct correlation subset and a first indirect correlation subset. The first direct correlation subset includes at least one first direct correlation. The first direct correlation characterizes the degree of direct correlation between the first modal information corresponding to the multimodal model and other first modal information pieces. The first indirect correlation subset includes at least one first indirect correlation. The first indirect correlation characterizes the degree of indirect correlation between the first modal information corresponding to the multimodal model and other first modal information pieces.

[0010] Regarding the first direct correlation, since the student model learns the correlation between first-modal information from different first modalities, the correlation between first-modal information from different first modalities with matching relationships is stronger, while the correlation between first-modal information from different first modalities without matching relationships is weaker. This allows the student model to learn first-modal features with good generalization ability. The student model's classification mechanism is based on the similarity between first-modal information from different first modalities, which is similar to the purpose of the first direct correlation. Therefore, in the above case, the first direct correlation can effectively align with the student model's classification mechanism. Furthermore, if a loss function term corresponding to the first direct correlation is added to the loss function, this term can serve as a constraint term. For example, the constraint term can include a regularization term; that is, the loss function term corresponding to the first direct correlation can be used as a regularization term.

[0011] For the first indirect correlation, the advantages of multimodal models in capturing cross-modal interactions can be effectively utilized to enhance the modeling of same-modal proximity.

[0012] In another possible implementation, the first multimodal feature corresponding to the multimodal model includes first modal features corresponding to each of the multiple first modal information pieces. When the first correlation subset includes a first direct correlation subset, obtaining the first correlation set corresponding to the multimodal model based on the first graph structure and the first multimodal feature corresponding to the multimodal model may include: determining at least one first directly connected node pair from the first graph structure. The first directly connected node pair includes two first nodes connected by a first edge. For each first directly connected node pair, determining the similarity between the first modal features corresponding to the two first nodes in the first directly connected node pair, obtaining a first similarity between the two first nodes in the first directly connected node pair. The first modal feature corresponds to the multimodal model. Based on the first similarity between the two first nodes in each of the at least one first directly connected node pair, a first direct correlation subset corresponding to each of the multiple first modal information pieces is obtained.

[0013] In another possible implementation, obtaining a first direct association subset corresponding to each of the plurality of first modal information based on a first similarity between two first nodes in each of at least one first direct-connected node pair may include: determining at least one target first direct-connected node pair from the at least one first direct-connected node pair for each first node in the at least one first direct-connected node pair. The target first direct-connected node pair includes the first node. The first direct association subset corresponding to the first modal information represented by the first node is obtained based on the first similarity between two first nodes in each of the at least one target first direct-connected node pair.

[0014] In another possible implementation, obtaining a subset of first direct association degrees corresponding to the first modal information represented by the first node based on a first similarity between two first nodes in each of at least one target first direct-connected node pair may include: obtaining a first comprehensive similarity degree corresponding to the first node based on the first similarity between two first nodes in each of the at least one target first direct-connected node pair; obtaining a second similarity degree between the two first nodes in the target first direct-connected node pair based on the first comprehensive similarity degree corresponding to the first node and the first similarity between the two first nodes in the target first direct-connected node pair; obtaining a first direct association degree between the two first nodes in the target first direct-connected node pair based on the second similarity degree; and obtaining a subset of first direct association degrees corresponding to the first modal information represented by the first node based on the first direct association degree between two first nodes in each of the at least one target first direct-connected node pair.

[0015] In another possible implementation, the first multimodal feature corresponding to the multimodal model includes the first modal feature corresponding to each of the multiple first modal information pieces. When the first association subset includes a first indirect association subset, obtaining the first association set corresponding to the multimodal model based on the first graph structure and the first multimodal feature corresponding to the multimodal model may include: determining at least one first inter-node pair from the first graph structure. The first inter-node pair includes two first nodes that are not connected by a first edge. For each first inter-node pair, a third similarity between the two first nodes in the first inter-node pair is obtained based on the first association information corresponding to the first inter-node pair. Based on the third similarity between the two first nodes in each of the at least one first inter-node pair, a first indirect association subset corresponding to each of the multiple first modal information pieces is obtained. The first association information corresponding to the first inter-node pair includes one of the following: a first direct association subset corresponding to each of the two first nodes in the first inter-node pair. The first modal features corresponding to at least one first associated node, and the first modal features corresponding to the two first nodes in the first inter-connection node pair, wherein the first associated node represents the first node in the first graph structure that plays the role of context association, and the context association function represents the function of associating the two first nodes in the first inter-connection node pair.

[0016] The multimodal proximity established by the multimodal model is used to construct the first indirect correlation between the first modal information of the same modality. That is, the proximity between the first nodes of the same other modality can be defined by using the contextual correlation provided by the context modality.

[0017] In another possible implementation, obtaining a subset of first indirect association degrees corresponding to each of the multiple first modalities based on a third similarity between two first nodes in each of at least one first indirect link pair may include: for a first node in at least one first indirect link pair, determining at least one target first indirect link pair from the at least one first indirect link pair. The target first indirect link pair includes the first node. Based on the third similarity between two first nodes in each of the at least one target first indirect link pair, obtaining a subset of first indirect association degrees corresponding to the first modalities represented by the first node.

[0018] In another possible implementation, obtaining a subset of first indirect association degrees corresponding to the first modal information represented by the first node based on a third similarity between two first nodes in each of at least one target first indirect node pair may include: obtaining a second comprehensive similarity degree corresponding to the first node based on the third similarity between two first nodes in each of the at least one target first indirect node pair; obtaining a fourth similarity degree between the two first nodes in the target first indirect node pair based on the second comprehensive similarity degree corresponding to the first node and the third similarity between two first nodes in the target first indirect node pair; obtaining a first indirect association degree between the two first nodes in the target first indirect node pair based on the fourth similarity between the two first nodes in the target first indirect node pair; and obtaining a subset of first indirect association degrees corresponding to the first modal information represented by the first node based on the first indirect association degree between two first nodes in each of the at least one target first direct node pair.

[0019] In another possible implementation, the first association set includes a first association subset corresponding to each of the multiple first modal information. Obtaining at least one first proximity degree corresponding to each of the multiple first modal information based on the first association sets corresponding to each of the multiple multimodal models may include: for the first modal information, for the teacher model, determining the first association subset corresponding to the first modal information and the student model, and the proximity degree between the first association subsets corresponding to the first modal information and the teacher model, to obtain the first proximity degree corresponding to the first modal information, the student model, and the teacher model.

[0020] Since the student model is trained based on the first proximity to the first modal information, the student model, and the teacher model, and this first proximity is obtained based on the proximity between the first association subset corresponding to the first modal information and the student model, and the first association subset corresponding to the first modal information and the teacher model, the first proximity between the student model and the teacher model for the same first modal information can reflect the degree of proximity between their reasoning abilities. The teacher model is a multimodal model that has been trained in continuous learning. At least one teacher model can include a multimodal model with the ability to maintain zero-shot learning and adapt to new information. Therefore, based on the first proximity corresponding to the student model and at least one teacher model, the student model can learn knowledge from at least one teacher model. Thus, in the continuous learning of multimodal models, the conflict between maintaining zero-shot learning ability and adapting to new information can be alleviated.

[0021] In another possible implementation, when the first correlation subset includes a first direct correlation subset and a second indirect correlation subset, determining the proximity between the first correlation subset corresponding to the first modality information and the student model, and the first correlation subset corresponding to the first modality information and the teacher model, to obtain a first proximity corresponding to the first modality information, the student model, and the teacher model may include: determining the proximity between the first direct correlation subset corresponding to the first modality information and the student model, and the first direct correlation subset corresponding to the first modality information and the teacher model, to obtain a second proximity corresponding to the first modality information, the student model, and the teacher model; determining the proximity between the first indirect correlation subset corresponding to the first modality information and the student model, and the first indirect correlation subset corresponding to the first modality information and the teacher model, to obtain a third proximity corresponding to the first modality information, the student model, and the teacher model; and obtaining the first proximity corresponding to the first modality information, the student model, and the teacher model based on the second and third proximity corresponding to the first modality information, the student model, and the teacher model.

[0022] In another possible implementation, training the student model based on at least one first proximity corresponding to each of the multiple first modal information pieces can include: training the student model based on at least one first proximity corresponding to each of the multiple first modal information pieces and at least one adjustable coefficient. The at least one adjustable coefficient corresponding to each of the first modal information pieces includes adjustable coefficients corresponding to the first modal information, the student model, and the teacher model. The adjustable coefficients corresponding to the first modal information, the student model, and the teacher model represent the contribution of the teacher model among all teacher models in terms of its contribution to processing the first modal information. The adjustable coefficient corresponding to the first modal information is obtained based on the comprehensive distance and distance corresponding to the first modal information. The comprehensive distance corresponding to the first modal information is obtained based on at least one distance corresponding to the first modal information. The at least one distance corresponding to the first modal information is obtained based on second association information corresponding to the first modal information. At least one distance corresponding to the first modality information is obtained based on second association information corresponding to the first modality information, including one of the following: at least one distance corresponding to the first modality information is obtained based on a first association subset corresponding to the first modality information and the student model, and a first association subset corresponding to the first modality information and at least one teacher model respectively. At least one distance corresponding to the first modality information is obtained based on first modality features corresponding to the first modality information and the student model, and first modality features corresponding to the first modality information and at least one teacher model respectively.

[0023] For a first modality among multiple first modal information, and for a teacher model among at least one teacher model, since the adjustable coefficients corresponding to the first modality information, the student model, and the teacher model can be determined based on at least one subset of first correlation degrees corresponding to the first modality information, the student model, and the teacher model, or the adjustable coefficients corresponding to the first modality information, the student model, and the teacher model can be determined based on at least one first modal feature corresponding to the first modality information, the student model, and the teacher model, the adjustable coefficients corresponding to the first modality information, the student model, and the teacher model can reflect the degree of contribution of the teacher model to processing the first modality information, and the degree of contribution of the teacher model among all teacher models. Therefore, based on the adjustable coefficients corresponding to the first modality information and the teacher model, the degree of learning of the first modality information from each teacher model can be determined, and the learning focus can be determined, so that for the first modality information, the student model focuses more on having the ability similar to which teacher model.

[0024] In another possible implementation, for the teacher model, at least one distance corresponding to the first modality information is obtained based on a first subset of correlation degree corresponding to the first modality information and the student model, and a first subset of correlation degree corresponding to the first modality information and at least one teacher model, which may include: the distances corresponding to the first modality information, the student model and the teacher model are obtained based on the distances between the first subset of correlation degree corresponding to the first modality information and the student model, and the first subset of correlation degree corresponding to the first modality information and the teacher model.

[0025] In another possible implementation, for the teacher model, at least one distance corresponding to the first modal information is obtained based on the first modal features corresponding to the first modal information and the student model, and the first modal features corresponding to the first modal information and at least one teacher model respectively. This may include: the distances corresponding to the first modal information, the student model, and the teacher model are obtained based on the first modal features corresponding to the first modal information and the student model, and the distances between the first modal features corresponding to the first modal information and the teacher model.

[0026] In another possible implementation, training the student model based on at least one first proximity degree and at least one adjustable coefficient corresponding to each of the multiple first modal information can include: for the first modal information, multiplying the first proximity degree and adjustable coefficient corresponding to the first modal information, the student model, and the teacher model by the first proximity degree and the adjustable coefficient corresponding to the teacher model to obtain a weighted proximity degree corresponding to the first modal information, the student model, and the teacher model. The student model is then trained based on the at least one weighted proximity degree corresponding to each of the multiple first modal information.

[0027] In another possible implementation, training the student model based on at least one first proximity corresponding to each of the plurality of first modal information may include: training the student model based on a subset of first association degrees corresponding to each of the plurality of first modal information and at least one first proximity.

[0028] In another possible implementation, when the first multimodal information is multimodal information that does not match the current task, training the student model based on at least one first proximity corresponding to each of the multiple first modal information can include: inputting second multimodal information into the student model to obtain second multimodal features corresponding to the student model. The second multimodal information includes at least one second modal information corresponding to each of the multiple modalities. The second multimodal features include second modal features corresponding to each of the multiple second modal information. Based on the second multimodal features corresponding to the student model, a second set of association degrees corresponding to the student model is obtained. The second set of association degrees includes a subset of second association degrees corresponding to each of the multiple second modal information. The subset of second association degrees corresponding to the second modal information includes at least one second association degree corresponding to the second modal information. The second association degree corresponding to the second modal information represents the degree of association between the second modal information and other second modal information. The other second modal information is any one of the second multimodal information other than the second modal information itself. The student model is trained based on at least one first proximity degree corresponding to each of the multiple first modal information sets, and a subset of second correlation degrees corresponding to each of the multiple second modal information sets. The second multimodal information is multimodal information matched with the current task.

[0029] In another possible implementation, training a student model based on at least one first proximity corresponding to each of the multiple first modal information pieces and a second subset of association corresponding to each of the multiple second modal information pieces can include: training the student model based on at least one first proximity and at least one adjustable coefficient corresponding to each of the multiple first modal information pieces and a second subset of association corresponding to each of the multiple second modal information pieces. The at least one adjustable coefficient corresponding to each of the first modal information pieces includes adjustable coefficients corresponding to the first modal information, the student model, and the teacher model. The adjustable coefficients corresponding to the first modal information, the student model, and the teacher model represent the contribution of the teacher model among all teacher models in terms of its contribution to processing the first modal information. The adjustable coefficients corresponding to the first modal information are obtained based on the comprehensive distance and distance corresponding to the first modal information. The comprehensive distance corresponding to the first modal information is obtained based on at least one distance corresponding to the first modal information. The at least one distance corresponding to the first modal information is obtained based on the second association information corresponding to the first modal information. At least one distance corresponding to the first modality information is obtained based on second association information corresponding to the first modality information, including one of the following: at least one distance corresponding to the first modality information is obtained based on a first association subset corresponding to the first modality information and the student model, and a first association subset corresponding to the first modality information and at least one teacher model respectively. At least one distance corresponding to the first modality information is obtained based on first modality features corresponding to the first modality information and the student model, and first modality features corresponding to the first modality information and at least one teacher model respectively.

[0030] In another possible implementation, obtaining a second set of association degrees corresponding to the student model based on the second multimodal features corresponding to the student model can include: obtaining a second set of association degrees corresponding to the student model based on the second graph structure and the second multimodal features corresponding to the student model. The second graph structure is constructed based on the second multimodal information. The second graph structure includes multiple second nodes and at least one second edge. Second nodes represent second modal information. Second edges are used to connect two second nodes with a direct association relationship. The modalities of the two second modal information with a direct association relationship are different.

[0031] In another possible implementation, where at least one teacher model is included, the teacher model is one of the following: a multimodal pre-trained model and a legacy multimodal model. Where at least two teacher models are included, both teacher models include a multimodal pre-trained model and a legacy multimodal model. The legacy multimodal model is the multimodal model corresponding to the previous task of the current task.

[0032] In cases where at least one teacher model may include a multimodal pre-trained model and an old multimodal model, since the first proximity between the student model and the multimodal pre-trained model for the same first modal information can reflect the proximity between the reasoning abilities of the student model and the multimodal pre-trained model, and the first proximity between the student model and the old multimodal model for the same first modal information can reflect the proximity between the reasoning abilities of the student model and the old multimodal model, the student model can learn knowledge from the multimodal pre-trained model and the old multimodal model based on the first proximity corresponding to the student model and the multimodal pre-trained model, thereby mitigating the conflict between maintaining zero-shot learning ability and adapting to new information in the continuous learning of the multimodal model.

[0033] Secondly, this application provides an information processing method, which may include: inputting third modality information into a trained student model to obtain third modality features; and obtaining processed information based on the third modality features. The trained student model may be trained using the multimodal model training method described in the embodiments of this application.

[0034] The information processing method provided in this application, because the trained student model has good zero-shot learning ability and the ability to adapt to new information, can provide more accurate, reliable and efficient multimodal services when applied to downstream tasks, thereby meeting the needs of the target.

[0035] Thirdly, this application provides a training apparatus for a multimodal model, which may include: a first obtaining module, configured to input first multimodal information into multiple multimodal models respectively to obtain first multimodal features corresponding to each of the multiple multimodal models. The first multimodal information includes at least one first modal information corresponding to each of the multiple modalities. The first multimodal features corresponding to the multimodal models include first modal features corresponding to each of the multiple first modal information. The multiple multimodal models include a student model and at least one teacher model. A first determining module, configured to determine a first association degree set corresponding to the multimodal model based on the first multimodal features corresponding to the multimodal model. The first association degree set includes a first association degree subset corresponding to each of the multiple first modal information. The first association degree subset corresponding to the first modal information includes at least one first association degree corresponding to the first modal information. The first association degree corresponding to the first modal information characterizes the degree of association between the first modal information and other first modal information. The other first modal information is any one of the first multimodal information other than the first modal information. The second acquisition module is used to obtain at least one first proximity degree corresponding to each of the multiple first modal information based on the first correlation degree set corresponding to each of the multiple multimodal models. For example, for the first modal information and the teacher model, the second acquisition module is used to determine the first correlation degree subset corresponding to the first modal information and the student model, and the proximity degree between the first correlation degree subset corresponding to the first modal information and the teacher model, thus obtaining the first proximity degree corresponding to the first modal information, the student model, and the teacher model. The first proximity degree represents the degree of proximity between the student model and the teacher model. The training module is used to train the student model based on the at least one first proximity degree corresponding to each of the multiple first modal information. The multiple multimodal models are multimodal models in continuous learning. The student model is the multimodal model corresponding to the current task in the sequence task. The teacher model is a trained multimodal model. The first multimodal information is the multimodal information matched with the current task.

[0036] Fourthly, this application provides an information processing apparatus, which may include: a third obtaining module, configured to input third modality information into a trained student model to obtain third modality features; and a fourth obtaining module, configured to obtain processing information based on the third modality features. The trained student model may be trained using a multimodal model training apparatus according to embodiments of this application.

[0037] Fifthly, embodiments of this application provide a computing device, including a memory and a processor, wherein the memory stores instructions that, when executed by the processor, cause the methods described in the first and second aspects of embodiments of this application to be implemented.

[0038] Sixthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the methods described in the first and second aspects of embodiments of this application to be implemented.

[0039] In a seventh aspect, embodiments of this application also provide a computer program product, which may include a computer program that, when run by a processor, causes the methods described according to the first and second aspects of embodiments of this application to be implemented.

[0040] Eighthly, embodiments of this application also provide a chip, including at least one processor and a communication interface, wherein the processor is used to execute the methods described in the first and second aspects of embodiments of this application. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 The diagram illustrates the principle of the training phase of image classification based on zero-shot learning.

[0043] Figure 2 This illustration shows an application diagram of the application stage based on zero-shot learning;

[0044] Figure 3 This illustration schematically shows an example diagram of selecting a teacher model from multiple multimodal models in continuous learning according to an embodiment of this application;

[0045] Figure 4 The illustrations show example diagrams of a dual-teacher knowledge distillation architecture and a single-teacher knowledge distillation rack according to embodiments of this application;

[0046] Figure 5 This schematic diagram illustrates an example of the relationship between first multimodal information, first multimodal features, and first association set according to an embodiment of this application.

[0047] Figure 6 The illustration shows a schematic diagram illustrating the principle of obtaining a first proximity between a student model and a teacher model according to an embodiment of this application;

[0048] Figure 7A The illustration shows a schematic diagram illustrating the principle of determining the adjustable coefficients corresponding to the first modality information, the student model, and the teacher model according to an embodiment of this application;

[0049] Figure 7B This illustration schematically shows the principle of determining the integrated distance corresponding to the first modality information according to an embodiment of this application;

[0050] Figure 8A The illustration shows a schematic diagram of the principle of a training method for a multimodal model according to an embodiment of this application;

[0051] Figure 8B The illustration shows a schematic diagram of another method for training a multimodal model according to an embodiment of this application;

[0052] Figure 9A The illustration shows a schematic diagram of another method for training a multimodal model according to an embodiment of this application;

[0053] Figure 9B The illustration shows a schematic diagram of another method for training a multimodal model according to an embodiment of this application;

[0054] Figure 10 A flowchart illustrating a training method for a multimodal model according to an embodiment of this application is shown schematically.

[0055] Figure 11 This schematic diagram illustrates an example of a first-figure structure according to an embodiment of this application;

[0056] Figure 12 A flowchart illustrating an information processing method according to an embodiment of this application is shown schematically.

[0057] Figure 13 The illustration schematically shows an exemplary system architecture of a training method, information processing method, and apparatus for multimodal models that can be applied according to embodiments of this application;

[0058] Figure 14 A block diagram of a training apparatus for a multimodal model according to an embodiment of this application is shown schematically;

[0059] Figure 15 A block diagram of an information processing apparatus according to an embodiment of this application is schematically shown; and

[0060] Figure 16 A block diagram of a computing device according to an embodiment of this application is shown as an example. Detailed Implementation

[0061] The term "and / or" used in this article describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0062] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first feature representation" and "second feature representation," etc., are used to distinguish different feature representations, not to describe a specific order of feature representations.

[0063] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0064] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0065] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0066] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0067] To facilitate understanding, some of the concepts involved in this application will be explained first.

[0068] A modality can refer to the source of information or the form in which information is represented. For example, a modality can include one of the following: perceptual modality, media modality, sensor modality, spectral modality, hyperspectral modality, and bioinformatics modality. A perceptual modality can include at least one of the following: touch, hearing, vision, and smell. A media modality can include at least one of the following: text, vision, audio, point clouds, code, time series, graphs, tables, and web pages. Vision can include at least one of the following: images and videos. Images can include at least one of the following: two-dimensional images and three-dimensional images. A sensor modality can include at least one of the following: visual sensors, LiDAR (Light Detection and Ranging), Radar (Radio Detection and Ranging), and wheel speedometers. A spectral modality can include at least one of the following: X-rays, ultraviolet, infrared, terahertz, and microwaves. A bioinformatics modality can include proteins. Furthermore, a modality can have a broad definition; for example, two languages ​​can be considered two modalities, and even information acquired in two different contexts can be considered two modalities. Multimodal can refer to at least two modes.

[0069] Modal information can refer to information corresponding to a mode. Multiple modal information can include at least two of the following: text information, image information, video information, audio information, point cloud information, code information, time series information, graph information, table information, web page information, visual sensor signals, lidar signals, radar signals, wheel speedometer signals, X-ray spectroscopy, ultraviolet spectroscopy, infrared spectroscopy, terahertz spectroscopy, microwave spectroscopy, and protein information, etc.

[0070] Multimodal information may include at least one modal information corresponding to each of the multiple modalities. For example, multimodal information may include at least one text information corresponding to text, at least one image information corresponding to an image, and at least one video information corresponding to a video.

[0071] Multimodal information has corresponding multimodal features. Multimodal features can include modal features corresponding to each of the multiple modalities within the multimodal information. Modal features can refer to the characteristic information of the modal information.

[0072] A multimodal model can refer to a model that can simultaneously process modal information corresponding to multiple modalities and utilize the interaction between these modal information to handle tasks. The deep learning model described in this application is a multimodal model. Multimodal models can be configured according to actual business needs and are not limited here, as long as they can obtain modal features of the modal information. For example, a multimodal model can include one of the following: a multimodal model based on a single-stream structure and a multimodal model based on a two-stream structure. For example, a two-stream multimodal model can include a CLIP (Contrastive Language-Image Pre-training) model.

[0073] Representation learning (RL) refers to the process of extracting modal features from modal information. Metric learning (ML) refers to the process of measuring multiple modal features and determining the similarity between them.

[0074] Zero-shot learning refers to constructing a mapping between modal information and auxiliary information during the training phase by mapping sample modal features and auxiliary features of sample auxiliary information to a shared feature space. This mapping allows the trained deep learning model to learn the model's information. Auxiliary information can include at least one of the following: attribute descriptions, text descriptions, and class hierarchy relationships. Metric learning can be used to determine the similarity between sample modal features and auxiliary features during the construction of this mapping. Since this mapping reflects the information between sample modal information (i.e., known modal information) and unknown modal information, zero-shot learning can be understood as using the mapping between known and unknown modal information to obtain a representation of the unknown modal information. Therefore, in the application phase, the trained deep learning model can be directly applied to the current task without further fine-tuning. This allows the trained deep learning model to process modal information and corresponding auxiliary information to obtain inference information about the modal information. The deep learning model can be a unimodal model or a multimodal model. This application pertains to multimodal models. Zero-shot learning can be applied to at least one of computer vision tasks, natural language processing tasks, and speech processing tasks. Computer vision tasks may include at least one of the following: visual classification tasks, visual retrieval tasks, object recognition tasks, visual segmentation tasks, visual question answering tasks, image captioning tasks, visual object detection tasks, object tracking tasks, cross-modal retrieval tasks, multimodal matching tasks, video summarization tasks, emotion recognition tasks, and person re-identification tasks. Natural language processing tasks may include at least one of the following: text classification tasks, text retrieval tasks, text translation tasks, and dialogue tasks. Speech processing tasks may include at least one of the following: speech recognition tasks.

[0075] To better understand zero-shot learning, we will use image classification as an example to further illustrate it, but this application can also be applied to other tasks mentioned above. Image classification refers to the task of determining the category of an image to be classified using a deep learning model. Zero-shot learning-based image classification refers to training a deep learning model so that, when faced with the current task (i.e., a task with an unseen class), the trained deep learning model can use the learned mapping information to identify the unseen class. See below for reference. Figure 1 This section explains the training phase for image classification based on zero-shot learning.

[0076] Figure 1 The diagram illustrates the principle of the training phase in image classification based on zero-shot learning.

[0077] like Figure 1As shown, a mapping information h(f(x, w1), g(y, w2)) between the image and auxiliary information is constructed by mapping the sample image features f(x, w1) and the sample auxiliary information features g(y, w2) to a shared feature space, enabling the trained deep learning model to learn this mapping information. x can represent the image. y can represent the auxiliary information. When the sample auxiliary features are obtained using a deep learning model, the model parameters of the deep learning model include model parameters w1 and w2. When the sample auxiliary features are not obtained using a deep learning model, the model parameters of the deep learning model include model parameter w1.

[0078] exist Figure 1 On this basis, Figure 2 The illustration shows an application diagram of the application stage based on zero-shot learning. For example... Figure 2 As shown, the image to be classified is a zebra image. Auxiliary information corresponding to the image can include the text "black and white, horse outline, and tiger stripes". Inputting the image and auxiliary information into a trained deep learning model results in the image being classified as a zebra.

[0079] Continuous learning refers to the technique in sequential tasks that enables deep learning models to continuously learn knowledge from the current task without forgetting knowledge from previous tasks. In other words, a deep learning model in continuous learning can still retain the reasoning ability for older tasks while learning knowledge from the current task. Sequential tasks can include both untrained and trained tasks. Untrained tasks can include the current task. The current task can refer to the task that needs to be learned at the current moment in continuous learning. Trained tasks can include the tasks from the starting task to the previous task. The previous task can refer to the task preceding the current task. There are no strict constraints between the tasks included in a sequence.

[0080] For example, a sequence of tasks can include Task 1, Task 2, ..., Task n, ..., Task N-1 and Task N. The starting task can be represented by Task 1. The previous task can be represented by Task n-1. The current task can be represented by Task n. The current deep learning model can be represented by deep learning model n-1. The initial deep learning model can be represented by deep learning model 0. The initial deep learning model can include a pre-trained model. The old deep learning model can be represented by deep learning model ns. The current sample information set can be represented by sample information set n. Deep learning model n can be obtained by training deep learning model n-1 using sample information set n. Deep learning model n has the reasoning ability for tasks 1 to n. N can be an integer greater than 1. n can be an integer greater than or equal to 1 and less than or equal to N. n∈{1, 2, ..., N-1, N}. s can be an integer greater than 1 and less than n. s∈{1, 2, ..., n-1, n}.

[0081] As discussed above, the goal of continuous learning is to gradually learn new knowledge from sequential task information while retaining old knowledge. Continuous learning faces a stability-plasticity dilemma: high stability weakens the ability to learn new knowledge, while high plasticity leads to catastrophic forgetting (CF). Stability refers to maintaining the deep learning model's reasoning ability for old tasks. Plasticity refers to enabling the deep learning model to learn the necessary knowledge from the current task. Catastrophic forgetting refers to the phenomenon where, as the deep learning model learns sequential task information, its parameters are continuously modified during training to adapt to the needs of the current task, causing the deep learning model to forget the knowledge learned from old tasks.

[0082] In continuous learning, multimodal models suffer from catastrophic forgetting. This catastrophic forgetting occurs because, when the data distribution of new information differs from that of old information, the multimodal model tends to learn the distribution of the new information, which then interferes with the old information. New information can refer to the multimodal information used for training the current task, while old information can refer to the multimodal information used for training previous tasks. Because multimodal models possess strong generalization capabilities, such as strong zero-shot learning ability, they can be directly applied to certain tasks without further training. This is crucial for adapting to constantly changing real-world scenarios and offers considerable accuracy. Therefore, multimodal models have strong resistance to catastrophic forgetting, making them well-suited for continuous learning. Furthermore, for tasks with limited multimodal information, multimodal models offer significant improvements in training efficiency. However, due to limitations in the scale and quality of the multimodal information used to train the model, multimodal models may struggle to meet the demands of certain tasks in continuous learning. To meet the requirements of these tasks, multimodal models can be fine-tuned, for example, by utilizing multimodal information corresponding to the task. However, fine-tuning multimodal models may weaken their generalization ability, such as reducing their zero-shot learning capability. Furthermore, it may lead to a gradual loss of performance gains during continuous learning. Therefore, how to mitigate the conflict between maintaining zero-shot learning capability and adapting to new information during continuous learning of multimodal models has become an urgent problem to be solved.

[0083] To alleviate the conflict between maintaining zero-shot learning capability and adapting to new information in multimodal models during continuous learning—that is, to seek a balance between maintaining zero-shot learning capability and adapting to new information—solutions can be formulated from both the data level and the model architecture level. The data level can include data mining and training sample levels.

[0084] At the data mining level, it was found that related techniques directly utilize the modal features corresponding to each of the multiple modal information to train multimodal models, that is, training multimodal models using only the feature information of the modal information itself. Research revealed that the relationships between modal information are also a form of knowledge, providing richer information and stronger representational capabilities. This allows for the full extraction of information inherent in the modal information itself, and the determination of these relationships is independent of data distribution. Therefore, not only can the information inherent in the modal information itself be utilized, but the relationships between modal information can also be integrated, facilitating knowledge preservation, i.e., maintaining the inherent capabilities of multimodal models, such as zero-shot learning. Furthermore, zero-shot learning requires utilizing the mapping information between known and unknown modal information to obtain representations of unknown modal information. Therefore, the relationships between modal information help multimodal models maintain zero-shot learning capabilities while having sufficient flexibility to adapt to new information.

[0085] The following example, image classification, illustrates how the relationships between modal information can provide richer information, possess greater representational power, and fully exploit the information carried by modal information. Furthermore, the determination of these relationships is independent of data distribution. However, this application can also be applied to other tasks mentioned above. For image classification, in the feature space, the intra-class distance of modal information should be closer, while the inter-class distance should be farther. Relationships based on modal information can effectively reduce intra-class differences and amplify inter-class differences, which is more valuable for image classification tasks.

[0086] Associations can include at least one of the following: direct associations and indirect associations. A direct association represents a direct relationship between two modal information pieces. A direct relationship means that the association between two modal information pieces does not require other modal information to be established. An indirect association represents an indirect relationship between two modal information pieces. An indirect relationship means that the association between two modal information pieces requires other modal information to be established. In other words, if two modal information pieces are associated through C-1 other modal information pieces—that is, one modal information piece is associated with the other modal information piece through C-1 other modal information pieces—then the association between the two modal information pieces can be called a C-order association. C can be an integer greater than or equal to 1. Therefore, a first-order association refers to a direct association. Second-order associations and above refer to indirect associations.

[0087] The degree of association in a relationship can be measured by its degree of association. The degree of association corresponding to a direct relationship is called the direct degree of association. The degree of association corresponding to an indirect relationship is called the indirect degree of association. The degree of association corresponding to a C-order relationship is called the C-order degree of association.

[0088] Building upon the above, it is necessary to consider how to obtain the correlation between modal information. Further research has revealed that the correlation between modal information can be obtained directly based on modal features. That is, both direct and indirect correlations between modal information can be obtained directly from modal features.

[0089] Furthermore, graph structures can be used to achieve this, specifically by obtaining the correlation between modal information based on graph structures. This is because graph structures are data structures used to describe the relationships between objects. A graph structure can include a set of nodes and a set of edges. A node set can include multiple nodes. An edge set can include at least one edge. An edge can represent the relationship between two connected nodes. The nodes and edges in a graph structure can be understood as relationships between objects, which coincides with the essence of zero-shot learning. Zero-shot learning utilizes the mapping information between known and unknown modal information to obtain a representation of unknown modal information. Therefore, this application proposes that the correlation between modal information can be obtained based on graph structures, so that the correlation between modal information can constrain the model parameters of a multimodal model, that is, it can act as a constraint term in the loss function, for example, a regularization term, thereby guiding the loss function and the direction of information transmission. In other words, by introducing the representation of graph structures during the training process of a multimodal model, the correlation between modal information constructed based on graph structures can restrict the loss function and the direction of information transmission. This not only alleviates the conflict between maintaining zero-shot learning ability and adapting to new information, but also enhances the adaptability and stability of multimodal models in continuous learning.

[0090] Building upon the above, it is necessary to consider how to obtain the correlation between modal information based on the graph structure. As mentioned above, the graph structure can include a set of nodes and a set of edges. The set of nodes can include multiple nodes. The set of edges can include at least one edge. It is discovered that at least one of the local structural information and global structural information of the graph structure can be used to fully explore the correlation between nodes.

[0091] In graph structures, first-order proximity describes the similarity between two nodes in a directly connected pair, reflecting the local structural information of the graph. A directly connected pair can consist of two nodes connected by an edge. If there is no edge connecting two nodes, the first-order proximity can be defined as zero. Furthermore, a higher first-order proximity indicates greater similarity between two nodes. For example, people who are friends in a social network tend to share similar information. It has also been found that due to the sparsity of graph structures (i.e., the proportion of edges is relatively small), the first-order proximity between two nodes without an edge connection is zero. However, the absence of an edge connection does not necessarily mean that two nodes are not related. For example, in a social network, two users who are friends with the same user may have similar interests and are more likely to become friends in a social network. In word co-occurrence networks, words that appear together in the same word group often have similar meanings. In recommender systems, two users who are friends with the same user are likely to purchase similar products. Therefore, using first-order proximity to describe the structural information of a graph may result in the loss of certain structural information, making it insufficient to fully describe the structural information of a graph. To fully describe the structural information of a graph, second-order proximity can be introduced. In a graph, second-order proximity can characterize the similarity of the neighborhood structures of two nodes, reflecting the global structural information of the graph. Neighborhood structure refers to the set of neighboring nodes of a node. Neighboring nodes refer to nodes connected to a node by edges.

[0092] Therefore, first-order proximity and direct correlation (i.e., first-order correlation) have similar meanings, as do second-order proximity and second-order correlation. Thus, first-order proximity of a graph structure can be used to determine the direct correlation (i.e., first-order correlation) between modal information. Second-order proximity of a graph structure can be used to determine the second-order correlation in the indirect correlation between modal information.

[0093] It should be noted that this application defines the relationships between modalities in a multimodal model. Two modalities from different modalities may have a direct relationship. Two modalities from the same modality do not have a direct relationship. When using nodes to represent modal information and edges to represent direct relationships in a graph structure, two nodes from different modalities may be connected by edges. Two nodes from the same modality do not have edges connected. Therefore, the first-order proximity between two nodes from the same modality is zero.

[0094] It should be noted that utilizing at least one of the direct correlations (i.e., first-order proximity) and indirect correlations (e.g., second-order proximity) between modal information in multimodal model training can not only alleviate the conflict between maintaining zero-shot learning capability and adapting to new information during continuous learning, but also enhance the adaptability and stability of the multimodal model during continuous learning. Furthermore, the choice of using direct correlations, indirect correlations, or a combination of both can be determined based on actual business needs.

[0095] For example, regarding direct correlation, when the multimodal model is trained in the following way, direct correlation can effectively align with the classification mechanism of the multimodal model. The multimodal model can learn the correlations between modal information of different modalities, making the correlation between modal information of different modalities with matching relationships stronger, and the correlation between modal information of different modalities without matching relationships weaker. Thus, the multimodal model can learn modal features with good generalization ability.

[0096] A matching relationship between modal information of different modalities means that the semantic information expressed by these modal information is consistent. Consistent semantic information expressed by different modal information means that the similarity between the semantic information expressed by these modal information is greater than or equal to a predetermined similarity threshold. No matching relationship between different modal information means that the semantic information expressed by these modal information is inconsistent. Inconsistent semantic information expressed by different modal information means that the similarity between the semantic information expressed by these modal information is less than a predetermined similarity threshold. The predetermined similarity threshold can be configured according to actual business needs and is not limited here.

[0097] For example, the semantic information expressed by text is "cat." The semantic information expressed by the image is "dog." Because the semantic information "cat" expressed by the text and the semantic information "dog" expressed by the image are inconsistent, there is no matching relationship between the text and the image. As another example, the semantic information expressed by speech is "opening a door." The semantic information expressed by video is "opening a door." Because the semantic information "opening a door" expressed by speech and the semantic information "opening a door" expressed by video are consistent, there is a matching relationship between speech and video.

[0098] The classification mechanism of a multimodal model can refer to using a multimodal model trained in the above manner to determine target candidate modality information corresponding to the target modality information of the target modality from at least one candidate modality information corresponding to each of at least one candidate modality, and to obtain the category corresponding to the target modality information based on the target candidate modality information. The target modality and the candidate modality are different modalities. For example, the multimodal model processes the target modality information corresponding to the target modality to obtain target modality features corresponding to the target modality information. The similarity between the target modality features and at least one candidate modality feature is determined. Based on the similarity between the target modality features and at least one candidate modality feature, target candidate modality information matching the target modality information is determined from at least one candidate modality information. Based on the target candidate modality information, the category corresponding to the target modality information is obtained. The similarity between the target modality information and the target candidate modality information can be greater than the similarity between the target modality information and at least one other candidate modality information. Other candidate modality information refers to modality information other than the target candidate modality information among at least one candidate modality information.

[0099] For example, at least one candidate modality may include text. The target modality may refer to an image. Therefore, a multimodal model can be used to process at least one candidate text to obtain candidate modal features corresponding to each candidate text. The multimodal model is then used to process the target image to obtain target modal features corresponding to the target image. The similarity between the target modal features and each of the at least one candidate modal features is determined. The maximum similarity is determined from the similarity between the target modal features and each of the at least one candidate modal features. The candidate text corresponding to the maximum similarity is identified as the target candidate text. The category corresponding to the target candidate text is identified as the category corresponding to the target image.

[0100] Therefore, it is evident that the classification mechanism of multimodal models is based on the similarity between modal information of different modalities, which is precisely similar to the purpose of direct correlation. Thus, in the above scenario, direct correlation can effectively align with the classification mechanism of multimodal models. Furthermore, if a loss function term corresponding to direct correlation is added to the loss function, this term can serve as a constraint term. For example, the constraint term can include a regularization term; that is, the loss function term corresponding to direct correlation can be used as a regularization term.

[0101] For example, regarding indirect correlation, the advantages of multimodal models in capturing cross-modal interactions can be effectively utilized to enhance the modeling of proximity within the same modality. For an explanation of this, please refer to the section below on obtaining indirect correlation between modal information based on graph structures.

[0102] At the training sample level, it was found that auxiliary multimodal information can be used as training samples in the training process of multimodal models. Auxiliary multimodal information can possess sufficient semantic diversity. Sufficient semantic diversity can be understood as random sampling in the feature space. Auxiliary multimodal information may also lack label information. Auxiliary multimodal information can refer to multimodal information that does not match the current task in a sequence task. Auxiliary multimodal information can include at least one auxiliary modal information corresponding to each of multiple modalities. A mismatch can include at least one of the following: First, the auxiliary multimodal information may contain auxiliary modal information of the target modality whose semantic information differs from that of auxiliary modal information of other modalities. In other words, the auxiliary multimodal information contains auxiliary modal information of the target modality that cannot form auxiliary modal information pairs with auxiliary modal information of other modalities. An auxiliary modal information pair can include two auxiliary modal information from different modalities that have the same semantic information. For example, auxiliary multimodal information includes multiple auxiliary modal information corresponding to images and multiple auxiliary modal information corresponding to text. Regarding the auxiliary modal information containing the semantic information "kitten," the modality of this auxiliary modal information is an image. There is no auxiliary modal information containing the semantic information "kitten" and whose modality is text. Secondly, the auxiliary multimodal information is not entirely formed for the current task. Auxiliary multimodal information has general applicability.

[0103] Auxiliary multimodal information can overcome the difficulties caused by the limited access to multimodal information corresponding to the trained task. Furthermore, rich transferable representations can be mined from auxiliary multimodal information, enabling effective knowledge transfer.

[0104] Furthermore, it was found that current multimodal information can be used as training samples to participate in the training process of multimodal models. Current multimodal information can refer to multimodal information matching the current task in a sequence task. Current multimodal information can include at least one current modality corresponding to each of multiple modalities. The modalities included in the current multimodal information and the auxiliary multimodal information can be the same. Current multimodal information can have a subset of true direct correlations corresponding to it. The subset of true direct correlations can include at least one true direct correlation. True direct correlations can correspond to the current task. True direct correlations can be represented by a predetermined identifier. The predetermined identifier can be configured according to actual business needs and is not limited here. For example, true direct correlation can represent the degree of correlation of a true direct relationship between two pieces of modality information. If there is a direct relationship between two pieces of modality information, the true direct correlation can be represented by a first predetermined identifier. The first predetermined identifier can be "1". If there is no direct relationship between two pieces of modality information, the true direct correlation can be represented by a second predetermined identifier. The second predetermined identifier can be "0". A direct correlation between two modal information means that the semantic information corresponding to the two modal information is consistent.

[0105] In addition, auxiliary multimodal information and current multimodal information can be used as training samples to participate in the training process of multimodal models.

[0106] At the modal architecture level, it was found that the capabilities of multimodal models vary at different stages of continuous learning. To alleviate the conflict between maintaining zero-shot learning ability and adapting to new information during continuous learning, a learning approach can be adopted, which involves learning from multimodal models with stronger adaptability. For example, learning can be done by learning from multimodal models with strong zero-shot learning ability to preserve this capability as much as possible, and by learning from multimodal models with strong adaptability to new information to better accommodate it.

[0107] Further investigation revealed that fine-tuning the multimodal model led to a decrease in its zero-shot learning capability, indicating that multimodal models with lower fine-tuning levels possess stronger zero-shot learning capabilities. For example, a multimodal model with strong zero-shot learning capability can include one of the following: a multimodal pre-trained model and a target multimodal model. The multimodal pre-trained model can refer to the initial multimodal model. The multimodal pre-trained model can refer to the multimodal model corresponding to the pre-training task. The difference between the model parameters of the target multimodal model and the model parameters of the multimodal pre-trained model satisfies predetermined conditions. Based on this, one of the multimodal pre-trained model and the target multimodal model can be used as the learning object to preserve zero-shot learning capability as much as possible.

[0108] Because fine-tuned multimodal models possess a strong ability to adapt to new information, they can be used as learning objects to adapt to new information. For example, a fine-tuned multimodal model can include an older multimodal model. An older multimodal model can refer to the multimodal model corresponding to the previous task of the current task.

[0109] Based on this, at the model architecture level, to alleviate the conflict between maintaining zero-shot learning capability and adapting to new information during continuous learning of multimodal models, this application proposes a knowledge distillation scheme based on teacher models. This involves selecting at least one of the following as the teacher model from multiple multimodal models in continuous learning: a multimodal model with strong zero-shot learning capability and a multimodal model with strong adaptability to new information. This allows the student model to learn knowledge from the teacher model. This scheme emphasizes maintaining relative proximity between modal information rather than directly learning individual modal features. For example, a knowledge distillation scheme based on dual teacher models allows the student model to learn knowledge from two teacher models simultaneously. One teacher model focuses on maintaining zero-shot learning capability, while the other focuses on adapting to new information. This scheme emphasizes maintaining relative proximity between modal information rather than directly learning individual multimodal features. Maintaining relative proximity between modal information is more conducive to ensuring that the multimodal model adapts to new information while preserving its zero-shot learning capability.

[0110] Building upon this, for knowledge distillation schemes, it is also necessary to consider the relationship between teacher multimodal information, student multimodal information, and the auxiliary and current multimodal information mentioned above. Teacher multimodal information can refer to the training samples of the teacher model. Student multimodal information can refer to the training samples of the student model.

[0111] Both teacher and student multimodal information can include current multimodal information, meaning the current multimodal information can be used as training samples for both the teacher and student models. Optionally, both teacher and student multimodal information can include auxiliary multimodal information. Furthermore, student multimodal information can also include current multimodal information, meaning the auxiliary multimodal information can be used as training samples for the teacher model. Both auxiliary and current multimodal information can be used as training samples for the student model.

[0112] Building upon the multi-teacher knowledge distillation architecture, this application proposes a modal information reweighting scheme to balance insights from multiple teacher models, addressing the issue of varying capabilities among different teacher models. This scheme prioritizes modal information, focusing on either maintaining zero-shot learning capabilities or adapting to new information. For example, adjustable coefficients are determined for each modal information and teacher model. These coefficients characterize the teacher model's contribution to processing modal information within the broader context of all teacher models.

[0113] The following is a general description of the various technical solutions in this application.

[0114] First, combined Figure 3 and Figure 4 This section explains how to select a teacher model from multiple multimodal models in continuous learning.

[0115] Figure 3 The illustration shows an example diagram of selecting a teacher model from multiple multimodal models in continuous learning according to an embodiment of this application.

[0116] like Figure 3 As shown, the sequence of tasks can include Task 1, ..., Task n-1, and Task n. Task n can refer to the current task. Task n-1 can refer to the previous task. The multimodal model corresponding to Task 1 is Multimodal Model 1. The multimodal model corresponding to Task n-1 is Multimodal Model n-1. The multimodal model corresponding to Task n is Multimodal Model n. The multimodal pre-trained model can be represented by Multimodal Model 0. Multimodal Model n can refer to the current multimodal model. Multimodal Model n can be used as a student model. Multimodal Model n-1 can refer to the old multimodal model. Multimodal Model n can be trained based on Multimodal Model n-1. n can be an integer greater than 1.

[0117] M multimodal models can be selected from multimodal models 0 to n-1 as teacher models, namely the first teacher model, ..., the m-th teacher model and the M-th teacher model. M can be an integer greater than or equal to 1 and less than or equal to n. m can be an integer greater than or equal to 1 and less than or equal to M. m∈{1, 2, ..., M-1, M}. The number of teacher models can be determined according to actual business needs and is not limited here. For example, M=1. Optionally, M=2.

[0118] exist Figure 3 On this basis, Figure 4 The illustration shows an example schematic diagram of a dual-teacher knowledge distillation architecture according to an embodiment of this application.

[0119] like Figure 4 As shown, M=2, and since multimodal model 0 has a strong zero-shot learning ability and multimodal model n-1 has a strong ability to adapt to new information, multimodal model 0 (i.e., multimodal pre-trained model) can be used as the first teacher model and multimodal model n-1 (i.e., old multimodal model) can be used as the second teacher model to form a dual-teacher knowledge distillation architecture.

[0120] It should be noted that the multiple multimodal models in this application may include student models and at least one teacher model. The at least one teacher model may include M teacher models, namely, the first teacher model, ..., the m-th teacher model, ..., the M-th teacher model. For example, see [reference needed]. Figure 3 M can be an integer greater than or equal to 1. m∈{1,2,...,M-1,M}.

[0121] It should also be noted that the teacher models can be determined based on actual business needs, and are not limited here. For example, the M teacher models may include multimodal models possessing at least one of the following capabilities: multimodal models with strong zero-shot learning capabilities and multimodal models with strong adaptability to new information. For example, you can refer to... Figure 4 .

[0122] It should also be noted that the model structures of the student model and the teacher model can be configured according to actual business needs, and are not limited here. For example, the student model may include a first feature extraction module corresponding to each of the multiple modalities. The teacher model may include a second feature extraction module corresponding to each of the multiple modalities. The structures of the first and second feature extraction modules can be configured according to actual business needs, as long as they can achieve feature extraction, and are not limited here. For example, the first feature extraction module may be a first encoder. The second feature extraction module may be a second encoder. For instructions on how to utilize the first and second feature extraction modules, please refer to the corresponding section below for operation S1010.

[0123] For example, a student model may include a first feature fusion module and a third feature extraction module corresponding to each of the multiple modalities. A teacher model may include a second feature fusion module and a fourth feature extraction module corresponding to each of the multiple modalities. The structures of the first, second, third, and fourth feature extraction modules can be configured according to actual business needs, as long as feature extraction is achieved; no specific limitations are imposed here. For instructions on how to utilize the first, third, second, and fourth feature extraction modules, please refer to the corresponding section below regarding operation S1010.

[0124] Next, combined Figure 5 and Figure 6 This paper provides an overall description of the first multimodal information, the first multimodal features, the first correlation set, the first proximity set, and the adjustable coefficient set.

[0125] Figure 5 The illustration shows an example diagram illustrating the relationship between the first multimodal information, the first multimodal feature, and the first correlation set according to an embodiment of this application.

[0126] like Figure 5 As shown, the first multimodal information may include at least one first modal information corresponding to each of the multiple modalities. That is, for a modality among the multiple modalities, the first multimodal information may include at least one first modal information corresponding to that modality.

[0127] The first multimodal feature may include first modal features corresponding to each of the multiple first modal information included in the first multimodal information; that is, the first modal information may have a first modal feature corresponding to that first modal information. The first modal feature may include at least one of a first shallow modal feature and a first deep modal feature. The first shallow modal feature may refer to a fine-grained feature used to characterize the first modal information. The first deep modal feature may be a coarse-grained feature used to characterize the first modal information. Coarse-grained features may refer to abstract features. Abstract features may refer to features capable of expressing semantic information.

[0128] A first set of correlation degrees corresponding to each of the multiple first modal information can be determined based on the first modal features corresponding to each of the multiple first modal information. The first set of correlation degrees may include at least one first correlation degree corresponding to a first modal information. A first correlation degree characterizes the degree of correlation between the first modal information and other first modal information. Other first modal information can be any one of the first multimodal information other than the first modal information itself. Therefore, a first set of correlation degrees corresponding to the first multimodal information can be obtained; that is, the first set of correlation degrees corresponding to the first multimodal information may include the first set of correlation degrees corresponding to each of the multiple first modal information. For a detailed explanation of how to determine the first set of correlation degrees corresponding to each of the multiple first modal information based on the first modal features corresponding to each of the multiple first modal information, please refer to the corresponding section below.

[0129] It should be noted that the first correlation subset corresponding to the first modality information can be in the form of a first correlation vector. The components of the first correlation vector can refer to the first correlation. The first correlation set corresponding to the first multimodal information can be in the form of a first correlation matrix. The first correlation matrix can include multiple first correlation vectors.

[0130] The following example uses a student model and an m-th teacher model to process a specific first modal information within the first multimodal information, combining... Figure 6This section explains how to obtain the first proximity between the student model and the m-th teacher model.

[0131] Figure 6 The illustration shows a schematic diagram illustrating the principle of obtaining a first proximity between a student model and a teacher model according to an embodiment of this application.

[0132] like Figure 6 As shown, given the first subset of correlation degree corresponding to the first modality information and the student model, and the first subset of correlation degree corresponding to the first modality information and the m-th teacher model, the proximity between the first subset of correlation degree corresponding to the first modality information and the student model, and the first subset of correlation degree corresponding to the first modality information and the m-th teacher model, can be determined, thereby obtaining the first proximity between the student model and the m-th teacher model.

[0133] It should be noted that for a detailed explanation of how to obtain the first degree of proximity with the student model and the teacher model, please refer to the relevant section below.

[0134] The following will combine Figures 7A-7B The method for determining the adjustable coefficients corresponding to the first modality information, the student model, and the teacher model is explained.

[0135] Figure 7A The illustration shows a schematic diagram illustrating the principle of determining the adjustable coefficients corresponding to the first modality information, the student model, and the teacher model according to an embodiment of this application.

[0136] Based on the distances corresponding to the first modal information, the student model, and the m-th teacher model, and the comprehensive distance corresponding to the first modal information, the adjustable coefficients corresponding to the first modal information, the student model, and the m-th teacher model are obtained. The distances corresponding to the first modal information, the student model, and the m-th teacher model can characterize the degree of similarity between the student model and the m-th teacher model in processing the first modal information. m∈{1, 2, ..., M-1, M}. For how to determine the comprehensive distance corresponding to the first modal information, please refer to the following... Figure 7B .

[0137] Figure 7B The illustration shows a schematic diagram of the principle of determining the integrated distance corresponding to the first modality information according to an embodiment of this application.

[0138] like Figure 7B As shown, the distance between the first subset of correlation degree corresponding to the first modality information and the student model, and the distance between the first subset of correlation degree corresponding to the first modality information and the m-th teacher model are determined, and the distance between the first modality information, the student model and the m-th teacher model is obtained.

[0139] Therefore, based on the distances corresponding to the first modal information, the student model, and the first teacher model, ..., the distances corresponding to the first modal information, the student model, and the m-th teacher model, ..., the distances corresponding to the first modal information, the student model, and the M-th teacher model, the comprehensive distance corresponding to the first modal information is obtained.

[0140] It should be noted that for specific instructions on how to determine the adjustable coefficients corresponding to the first modality information, the student model, and the teacher model, please refer to the relevant sections below.

[0141] It should also be noted that the second multimodal information, second multimodal feature, and second correlation set in this application embodiment are similar to the first multimodal information, first multimodal feature, and first correlation set described above. Furthermore, the modalities possessed by the first and second multimodal information, as well as the number and type of modal information corresponding to those modalities, can be configured according to actual business needs and are not limited here. When the first multimodal information is multimodal information matching the current task, the first multimodal information can refer to the current multimodal information. When the first multimodal information is multimodal information not matching the current task, and the second multimodal information is multimodal information matching the current task, the first multimodal information can refer to auxiliary multimodal information. The second multimodal information can refer to the current multimodal information. The current multimodal information can refer to multimodal information matching the current task of the sequence task.

[0142] then, Figure 8A The illustration shows a schematic diagram of the principle of a training method for a multimodal model according to an embodiment of this application.

[0143] like Figure 8A As shown, the first multimodal information can be input into the first teacher model, ..., the m-th teacher model, ..., the M-th teacher model, and the student model, respectively, to obtain the first multimodal features corresponding to each of the first teacher model, ..., the m-th teacher model, ..., the M-th teacher model, and the student model. It should be noted that for a multimodal model among multiple multimodal models, the first multimodal feature corresponding to that multimodal model can include the first modal features corresponding to each of the multiple first modal information pieces. That is, for the first modal information among multiple first modal information pieces, the first multimodal feature corresponding to that multimodal model can include the first modal features corresponding to that first modal information.

[0144] Based on the first multimodal features corresponding to the first teacher model, ..., the m-th teacher model, ..., the M-th teacher model and the student model respectively, the first association degree set corresponding to the first teacher model, ..., the m-th teacher model, ..., the M-th teacher modality and the student model respectively is obtained.

[0145] For the m-th teacher model, determine the first set of correlation degree corresponding to the first multimodal information and the student model, and the proximity between the first set of correlation degree corresponding to the first multimodal information and the m-th teacher model, to obtain the first set of proximity degree corresponding to the student model and the m-th teacher model.

[0146] Train the student model based on the first proximity set corresponding to the student model and the first teacher model, ..., the first proximity set corresponding to the student model and the m-th teacher model, ..., the first proximity set corresponding to the student model and the M-th teacher model.

[0147] Figure 8B This illustration schematically depicts a principle diagram of a multimodal model training method according to an embodiment of this application. Figure 8A The main difference is that, Figure 8A On this basis, Figure 8B A set of adjustable coefficients corresponding to the first multimodal information, the student model, and the m-th teacher model is introduced.

[0148] like Figure 8B As shown, based on the first correlation set corresponding to the first multimodal information and the student model, the first correlation set corresponding to the first multimodal information and the first teacher model, ..., the first correlation set corresponding to the first multimodal information and the m-th teacher model, ..., the first correlation set corresponding to the first multimodal information and the M-th teacher model, the adjustable coefficient set corresponding to the first multimodal information, the student model and the first teacher model, ..., the adjustable coefficient set corresponding to the first multimodal information, the student model and the m-th teacher model, ..., the adjustable coefficient set corresponding to the first multimodal information, the student model and the M-th teacher model is determined.

[0149] Based on the first proximity set and the set of adjustable coefficients corresponding to the student model and the m-th teacher model, the weighted proximity set corresponding to the student model and the m-th teacher model is obtained. It should be noted that for how to determine the first proximity set corresponding to the student model and the m-th teacher model, please refer to the section above. Figure 7A The explanation will not be repeated here.

[0150] Train the student model based on the weighted proximity sets corresponding to the student model and the first teacher model, ..., the weighted proximity sets corresponding to the student model and the m-th teacher model, ..., the weighted proximity sets corresponding to the student model and the M-th teacher model.

[0151] Figure 9A This illustration schematically depicts a principle diagram of another multimodal model training method according to an embodiment of this application. Figure 8A The main difference is that, Figure 8A On this basis, Figure 9A A second multimodal information was introduced.

[0152] like Figure 9A As shown, the second multimodal information is input into the student model to obtain the second multimodal features corresponding to the student model. Based on the second multimodal features corresponding to the student model, the second direct correlation set corresponding to the student model is obtained. Based on the second direct correlation set and the second true direct correlation set corresponding to the student model, the fifth proximity set corresponding to the student model is obtained. It should be noted that for the second true direct correlation, please refer to the corresponding section below for operation S1040.

[0153] The student model is trained based on the fifth proximity set corresponding to the student model, the first proximity set corresponding to the student model and the first teacher model, ..., the first proximity set corresponding to the student model and the m-th teacher model, ..., the first proximity set corresponding to the student model and the M-th teacher model. It should be noted that for how to determine the first proximity set corresponding to the student model and the m-th teacher model, please refer to the section above regarding... Figure 8A The explanation will not be repeated here.

[0154] Figure 9B This illustration schematically depicts a principle diagram of another multimodal model training method according to an embodiment of this application. Figure 9A The main difference is that, Figure 9A On this basis, Figure 9B A set of adjustable coefficients corresponding to the first multimodal information, the student model, and the m-th teacher model is introduced.

[0155] like Figure 9B As shown, the student model is trained based on the weighted proximity sets corresponding to the student model and the first teacher model, ..., the weighted proximity sets corresponding to the student model and the m-th teacher model, ..., the weighted proximity sets corresponding to the student model and the M-th teacher model, and the fifth proximity set corresponding to the student model.

[0156] It should be noted that, for how to determine the weighted proximity set corresponding to the student model and the m-th teacher model, please refer to the section above. Figure 8B For an explanation of how to determine the fifth proximity set corresponding to the student model, please refer to the section above. Figure 9A The explanation will not be repeated here.

[0157] Figure 10 A flowchart illustrating a training method for a multimodal model according to an embodiment of this application is shown schematically.

[0158] like Figure 10 As shown, the method 1000 may include operations S1010 to S1040.

[0159] In operation S1010, the first multimodal information is input into multiple multimodal models respectively to obtain the first multimodal features corresponding to each of the multiple multimodal models.

[0160] According to embodiments of this application, the first multimodal information may be obtained from a data source via a data interface in response to the detection of a model training instruction. The data source may include at least one of the following: a publicly available dataset, a local database, a cloud database, and network resources. Furthermore, in cases where it is difficult to directly obtain the first multimodal information from the data source, it may be obtained by processing the initial multimodal information. The initial multimodal information may be obtained from the data source via a data interface in response to the detection of a model training instruction.

[0161] According to embodiments of this application, the first multimodal information may include at least one first modal information corresponding to each of the plurality of modalities. The first multimodal feature corresponding to the multimodal model may include a first modal feature corresponding to each of the plurality of first modal information.

[0162] The first multimodal feature corresponding to each of the multiple multimodal models can be obtained in the following way.

[0163] As one implementation, for the first modal information in at least one first modal information corresponding to multiple modalities, for the student model, the first modal information can be input into a first feature extraction module corresponding to the modality to which the first modal information belongs, to obtain the first modal features corresponding to the first modal information and the student model. For the teacher model, the first modal information can be input into a second feature extraction module corresponding to the modality to which the first modal information belongs, to obtain the first modal features corresponding to the first modal information and the teacher model.

[0164] As another implementation, for the first modal information in at least one first modal information corresponding to multiple modalities, for the student model, the first modal information can be input into a first fusion feature extraction module to obtain a first fusion modal feature corresponding to the first modal information and the student model. The first fusion modal feature corresponding to the first modal information and the student model is then input into a third feature extraction module to obtain a first modal feature corresponding to the first modal information and the student model. For the teacher model, the first modal information can be input into a second fusion feature extraction module to obtain a second fusion modal feature corresponding to the first modal information and the teacher model. The second fusion modal feature corresponding to the first modal information and the teacher model is then input into a fourth feature extraction module to obtain a first modal feature corresponding to the first modal information and the teacher model.

[0165] It should be noted that the first feature extraction module, the second feature extraction module, the first feature fusion module, the third feature extraction module, the second feature fusion module, and the fourth feature extraction module can be found in the corresponding sections above, and will not be repeated here.

[0166] In operation S1020, for the multimodal model, based on the first multimodal feature corresponding to the multimodal model, the first correlation set corresponding to the multimodal model is obtained.

[0167] According to embodiments of this application, for a multimodal model among multiple multimodal models, a first correlation set corresponding to the multimodal model is provided. The first correlation set corresponding to the multimodal model may include a first correlation subset corresponding to each of the multiple first modal information pieces. For a first modal information among the multiple first modal information pieces, the first correlation subset corresponding to the first modal information may include at least one first correlation degree corresponding to the first modal information. The first correlation degree corresponding to the first modal information can characterize the degree of correlation between the first modal information and other first modal information. Other first modal information can be any one of the multiple first modal information pieces other than the first modal information itself. That is, the first correlation degree between the first modal information and any one of the multiple first modal information pieces other than the first modal information itself can be determined, thereby obtaining at least one first correlation degree corresponding to the first modal information.

[0168] Optionally, the association relationship may include at least one of the following: direct association relationship and indirect association relationship. Correspondingly, the association degree may include at least one of the following: direct association degree and indirect association degree. Direct association degree can characterize the degree of association with a direct association relationship. Indirect association degree can characterize the degree of association with an indirect association relationship. Thus, the first association degree set may include one of the following: a first direct association degree set and a first indirect association degree set. The first association degree subset may include at least one of the following: a first direct association degree subset and a first indirect association degree subset. The first direct association degree set may include a first direct association degree subset corresponding to each of the multiple first modal information pieces. The first indirect association degree set may include a first indirect association degree subset corresponding to each of the multiple first modal information pieces. The first direct association degree subset may include at least one first direct association degree. The first direct association degree can characterize the degree of association of the direct association relationship between the first modal information and other first modal information pieces. The first indirect association degree subset may include at least one first indirect association degree. The first indirect association degree can characterize the degree of association of the indirect association relationship between the first modal information and other first modal information pieces.

[0169] It should be noted that, for multimodal information, two modal information pieces with a direct relationship have different modalities. Two modal information pieces with an indirect relationship have the same modality. In other words, a direct relationship occurs between two modal information pieces from different modalities. An indirect relationship occurs between two modal information pieces from the same modality. Therefore, a direct relationship can be understood as an intermodal relationship, and an indirect relationship as an intramodal relationship. For example, the first multimodal information may include multiple first modal information pieces corresponding to the first modality and multiple first modal information pieces corresponding to the second modality. A direct relationship (i.e., an intermodal relationship) occurs between the first modal information pieces corresponding to the first modality and the first modal information pieces corresponding to the second modality. An indirect relationship (i.e., an intramodal relationship) occurs between at least one of the following: between two first modal information pieces corresponding to the first modality and between two first modal information pieces corresponding to the second modality.

[0170] Based on this, the first direct correlation degree can characterize the degree of direct correlation between the first modal information and the first other first modal information. The first other first modal information can be any first modal information among multiple first modal information that has a different modality from the first modal information. The first indirect correlation degree can characterize the degree of indirect correlation between the first modal information and the second other first modal information. The second other first modal information can be any first modal information among multiple first modal information that has the same modality as the first modal information.

[0171] It should be noted that since the first subset of correlation degrees corresponding to the first modality information is determined based on the first multimodal features corresponding to the first multimodal information, this subset can fully exploit the information carried by the first modality information itself, and the determination of the correlation relationships between the first modality information does not depend on the data distribution. Therefore, it can not only utilize the information carried by the first modality information itself, but also integrate the correlation relationships between the first modality information, facilitating knowledge preservation, i.e., maintaining the zero-shot learning capability of the multimodal model. Furthermore, since zero-shot learning requires using the mapping information between known and unknown modality information to obtain the representation of unknown modality information, the correlation relationships between the first modality information can help the multimodal model maintain its zero-shot learning capability while having sufficient degrees of freedom to adapt to new information.

[0172] It should also be noted that using at least one of the first direct correlation degree and the first indirect correlation degree between the first modal information to participate in the training of the multimodal model can not only alleviate the conflict between the multimodal model maintaining zero-shot learning ability and adapting to new information in continuous learning, but also enhance the adaptability and stability of the multimodal model in continuous learning.

[0173] To obtain the first subset of correlation degrees corresponding to the first modality information and the multimodal model, the following methods can be used. For example, directly based on modal features. Alternatively, indirectly based on graph structure. See the corresponding sections below for details.

[0174] In operation S1030, based on the first correlation set corresponding to each of the multiple multimodal models, at least one first proximity degree corresponding to each of the multiple first modal information is obtained.

[0175] To obtain at least one first proximity degree corresponding to each of the multiple first modal information, the following method can be used.

[0176] As one implementation method, for the first modal information and the teacher model, a first subset of correlation degree corresponding to the first modal information and the student model is determined, as well as the proximity degree between the first subset of correlation degree corresponding to the first modal information and the teacher model, to obtain the first proximity degree corresponding to the first modal information, the student model and the teacher model.

[0177] According to embodiments of this application, for a first modality among multiple first modality information and for a teacher model among at least one teacher model, a first subset of correlation degrees corresponding to the first modality information and the student model can be determined, as well as the proximity degree between the first subset of correlation degrees corresponding to the first modality information and the teacher model. Based on the proximity degree between two first subsets of correlation degrees, a first proximity degree corresponding to the first modality information, the student model, and the teacher model is obtained. Therefore, for the first modality information, since the teacher model includes at least one, and the teacher model has a first proximity degree corresponding to the teacher model, the first proximity degree corresponding to the first modality information includes at least one.

[0178] According to embodiments of this application, the first proximity degree corresponding to the first modal information, the student model, and the teacher model can characterize the degree of proximity between the student model and the teacher model. The degree of proximity between the student model and the teacher model can refer to the degree of proximity between the inference results of the first modal information and the respective inference results of the student model and the teacher model; that is, the degree of proximity between the inference results of the student model and the teacher model can be characterized by the proximity between two subsets of first correlation degrees. The inference results of the student model and the teacher model can include a first multimodal feature corresponding to the student model and a first multimodal feature corresponding to the teacher model. Therefore, the first proximity degree between the student model and the teacher model can reflect the degree of proximity between the inference abilities of the student model and the teacher model.

[0179] It should be noted that, since the first proximity between the student model and the teacher model for the same first modal information can reflect the degree of proximity between the reasoning abilities of the student model and the teacher model, and the teacher model is a multimodal model that has been trained in continuous learning, at least one teacher model can include a multimodal model with the ability to maintain zero-shot learning and adapt to new information, the student model can learn knowledge from at least one teacher model based on the first proximity corresponding to the student model and at least one teacher model. Thus, the conflict between maintaining zero-shot learning ability and adapting to new information can be alleviated in the continuous learning of multimodal models.

[0180] For example, in a scenario where at least one teacher model can include a multimodal pre-trained model and an older multimodal model, the multimodal pre-trained model can be a multimodal model with strong zero-shot learning capabilities, and the older multimodal model can be a multimodal model with strong adaptability to new information, the first proximity between the student model and the multimodal pre-trained model for the same first modality information reflects the proximity between their reasoning abilities. Similarly, the first proximity between the student model and the older multimodal model for the same first modality information reflects the proximity between their reasoning abilities. Therefore, based on the first proximity types corresponding to the student model and the multimodal pre-trained model with the first modality information, the student model and the teacher model, and the first proximity type corresponding to the student model and the older multimodal model, the student model can learn knowledge from both the multimodal pre-trained model and the older multimodal model. This allows for mitigation of the conflict between maintaining zero-shot learning capabilities and adapting to new information during continuous learning of the multimodal model.

[0181] In operation S1040, a student model is trained based on at least one first proximity degree corresponding to each of the multiple first modal information.

[0182] The following method can be used to train a student model based on the first proximity to obtain a trained student model.

[0183] One implementation involves training a student model based on a first subset of relevance values ​​corresponding to each of the multiple first modal information pieces and at least one first proximity value. For example, a fourth proximity value corresponding to each of the multiple first modal information pieces is obtained based on a first subset of direct relevance values ​​and a first subset of true direct relevance values ​​corresponding to each of the multiple first modal information pieces. For example, for a first modal information piece among the multiple first modal information pieces, the proximity value between the first subset of direct relevance values ​​and the first subset of true direct relevance values ​​corresponding to that first modal information piece is determined, resulting in a fourth proximity value corresponding to that first modal information piece. The student model is then trained based on at least one first proximity value corresponding to each of the multiple first modal information pieces and a fourth proximity value corresponding to each of the multiple first modal information pieces. For example, for a first modal information piece among the multiple first modal information pieces, a comprehensive proximity value corresponding to that first modal information piece is obtained based on at least one first proximity value corresponding to that first modal information piece. Finally, the student model is trained based on the comprehensive proximity value corresponding to each of the multiple first modal information pieces and a fourth proximity value corresponding to each of the multiple first modal information pieces.

[0184] According to embodiments of this application, a subset of first true direct correlation degrees corresponding to the first modal information may include at least one first true direct correlation degree corresponding to the first modal information. The first true direct correlation degree corresponding to the first modal information can characterize the true correlation degree of the direct correlation between the first modal information and other first modal information. Other first modal information can be any one of a plurality of first modal information other than the first modal information itself. For example, if the first modal information has a direct correlation relationship with the other first modal information, the first true direct correlation degree corresponding to the first modal information and the other first modal information can be characterized by a first predetermined identifier. If the first modal information does not have a direct correlation relationship with the other first modal information, the first true direct correlation degree corresponding to the first modal information and the other first modal information can be characterized by a second predetermined identifier. The first predetermined identifier and the second predetermined identifier can be configured according to actual business needs and are not limited here. For example, the first predetermined identifier can be "1". The second predetermined identifier can be "0". It should be noted that, as mentioned above, two modal information pieces with a direct correlation are of different modalities; that is, the direct correlation occurs between two modal information pieces of different modalities. Furthermore, the first true direct correlation degree between two modal information pieces of the same modality can be characterized by a second predetermined identifier.

[0185] As another implementation, a student model is trained based on at least one first proximity degree and at least one adjustable coefficient corresponding to each of the multiple first modal information pieces. For example, for a first modal information among the multiple first modal information pieces, and for a teacher model among at least one teacher model, the first proximity degree and adjustable coefficient corresponding to the first modal information, the student model, and the teacher model are multiplied to obtain a weighted proximity degree corresponding to the first modal information, the student model, and the teacher model. Thus, for the first modal information, since there is at least one teacher model, and each teacher model has a weighted proximity degree corresponding to it, the weighted proximity degree corresponding to the first modal information includes at least one. The student model is trained based on the at least one weighted proximity degree corresponding to each of the multiple first modal information pieces. The adjustable coefficient corresponding to the first modal information, the student model, and the teacher model can reflect the degree of contribution of the teacher model to processing the first modal information, and the degree of contribution of the teacher model among all teacher models.

[0186] As another implementation, the student model is trained based on at least one first proximity degree, at least one adjustable coefficient, and a subset of first correlation degree corresponding to each of the multiple first modal information.

[0187] As another implementation, the student model is trained based on at least one first proximity degree, at least one predetermined coefficient, and a subset of first correlation degree corresponding to each of the multiple first modal information.

[0188] As another implementation, when the first multimodal information is multimodal information that does not match the current task, and the second multimodal information is multimodal information that matches the current task, the student model is trained based on at least one first proximity degree corresponding to each of the multiple first modal information and a subset of second correlation degrees corresponding to each of the multiple second modal information.

[0189] According to embodiments of this application, the second association set corresponding to the student model can be obtained based on the second multimodal features corresponding to the student model. The second multimodal features corresponding to the student model can be obtained by inputting second multimodal information into the student model. The second multimodal information can include at least one second modal information corresponding to each of the multiple modalities. The second multimodal features can include second modal features corresponding to each of the multiple second modal information. The second association set can include a subset of second association degrees corresponding to each of the multiple second modal information. For a second modal information among the multiple second modal information, the subset of second association degrees corresponding to that second modal information can include at least one second association degree corresponding to that second modal information. The second association degree corresponding to that second modal information can characterize the degree of association between that second modal information and other second modal information. Other second modal information can be any one of the second multimodal information other than that second modal information. That is, the second correlation degree between the second modal information and any other second modal information among a plurality of second modal information can be determined, and at least one second correlation degree corresponding to the second modal information can be obtained.

[0190] Optionally, the second subset of correlation degrees may include at least one of the following: a second direct correlation degree subset and a second indirect correlation degree subset. The second direct correlation degree subset may include at least one second direct correlation degree. The second direct correlation degree can characterize the degree of correlation of the direct correlation between the second modality information and other second modality information. The second indirect correlation degree subset may include at least one second indirect correlation degree.

[0191] The second indirect correlation degree can characterize the degree of indirect correlation between the second modality information and other second modality information.

[0192] As described above, the second multimodal information may include multiple second modal information corresponding to the first modality and multiple second modal information corresponding to the second modality. Direct associations (i.e., intermodal relationships) occur between the second modal information corresponding to the first modality and the second modal information corresponding to the second modality. Indirect associations (i.e., intramodal relationships) occur between at least one of the following: between two second modal information corresponding to the first modality and between two second modal information corresponding to the second modality.

[0193] Based on this, the second direct correlation degree can characterize the degree of correlation between the second modal information and the first other second modal information. The first other second modal information can be any second modal information among multiple second modal information that has a different modality from the first second modal information. The second indirect correlation degree can characterize the degree of correlation between the second modal information and the second other second modal information. The second other second modal information can be any second modal information among multiple second modal information that has the same modality as the first second modal information.

[0194] According to embodiments of this application, a subset of second true direct correlation degrees corresponding to the second modality information may include at least one second true direct correlation degree corresponding to the second modality information. The second true direct correlation degree corresponding to the second modality information can characterize the true correlation degree of the direct correlation between the second modality information and other second modality information. Other second modality information can be any one of multiple second modality information other than the first second modality information. For example, if the second modality information has a direct correlation with the other second modality information, the second true direct correlation degree corresponding to the second modality information and the other second modality information can be characterized by a first predetermined identifier. If the second modality information does not have a direct correlation with the other second modality information, the second true direct correlation degree corresponding to the second modality information and the other second modality information can be characterized by a second predetermined identifier. The first predetermined identifier and the second predetermined identifier can be configured according to actual business needs and are not limited here. For example, the first predetermined identifier can be "1". The second predetermined identifier can be "0". It should be noted that, as mentioned above, two modal information pieces with a direct correlation are of different modalities; that is, the direct correlation occurs between two modal information pieces of different modalities. Furthermore, the second true direct correlation degree between two modal information pieces of the same modality can be characterized by a second predetermined identifier.

[0195] The following explains how to obtain the first correlation subset corresponding to the first modality information and the multimodal model in operation S1020.

[0196] (1) For obtaining the first correlation subset corresponding to the first modality information and the multimodal model directly based on modal features.

[0197] (1.1) For obtaining the first direct correlation subset directly based on modal features

[0198] For a first modal information among multiple first modal information, and for a first other first modal information among at least one first other first modal information, the first other first modal information can be any first modal information among multiple first modal information that has a different modality from the first modal information.

[0199] As one implementation, the similarity between the first modal feature corresponding to the first modal information and the multimodal model, and the similarity between the first other first modal information and the first modal feature corresponding to the multimodal model, is determined to obtain the similarity between the first modal information and the first other first modal information corresponding to the multimodal model. Based on the similarity between the first modal information corresponding to the multimodal model and at least one first other first modal information, a third comprehensive similarity corresponding to the first modal information and the multimodal model is obtained. Based on the similarity between the first modal information and the first other first modal information corresponding to the multimodal model, and the third comprehensive similarity corresponding to the first modal information and the multimodal model, a first direct correlation degree between the first modal information and the first other first modal information corresponding to the multimodal model is obtained.

[0200] For example, multiple first modal information includes first modal information VF1 corresponding to audio, and first modal information I1 and first modal information I2 corresponding to images. The first modal information I1 and the first modal information I2 can serve as first other first modal information corresponding to the first modal information VF1. By inputting the multiple first modal information into the m-th teacher model Tm, the first modal features R corresponding to the first modal information VF1 and the m-th teacher model Tm are obtained. Tm-VF1 The first modal feature R corresponding to the first modal information I1 and the m-th teacher model Tm Tm-I1 And the first modal feature R corresponding to the first modal information I2 and the m-th teacher model Tm. Tm-I2 .

[0201] Determine the first modal feature R Tm-VF1 With the first modal feature R Tm-I1 The similarity between them is obtained by calculating the similarity Sim between the first modal information VF1 and the first modal information I1 corresponding to the m-th teacher model Tm. Tm-VF1-I1 Similarly, the similarity Sim between the first modal information VF1 and the first modal information I2 corresponding to the m-th teacher model Tm is obtained. Tm-VF1-I2 Then, the similarity Sim... Tm-VF1-I1 Similarity Sim Tm-VF1-I2 Adding them together yields the third comprehensive similarity CSim3, which corresponds to the first modal information VF1 and the m-th teacher model Tm. Tm-VF1 Finally, the similarity Sim was determined. Tm-VF1-I1 Similarity to the third comprehensive similarity CSim3 Tm-VF1 The ratio between them yields the first direct correlation DC between the first modal information VF1 and the first modal information I1 corresponding to the m-th teacher model Tm. Tm-VF1-I1Similarly, the first direct correlation DC between the first modal information VF1 and the first modal information I2 corresponding to the m-th teacher model Tm is obtained. Tm-VF1-I2 Thus, the first direct correlation subset DCS corresponding to the first modal information VF1 and the m-th teacher model Tm is obtained. Tm-VF1 .

[0202] As another implementation, the similarity between the first modal feature corresponding to the first modal information and the multimodal model, and the first modal feature corresponding to the first other first modal information and the multimodal model is determined, thereby obtaining the first direct correlation between the first modal information corresponding to the multimodal model and the first other first modal information.

[0203] For example, similarity Sim can be used Tm-VF1-I1 The first direct correlation DC between the first modal information VF1 and the first modal information I1 corresponding to the m-th teacher model Tm is determined. Tm-VF1-I1 Similarity Sim Tm-VF1-I2 The first direct correlation DC between the first modal information VF1 and the first modal information I2 corresponding to the m-th teacher model Tm is determined. Tm-VF1-I2 .

[0204] (1.2) For obtaining the first indirect correlation subset directly based on modal features

[0205] For a first modal information among multiple first modal information, and for a second other first modal information among at least one second other first modal information, the second other first modal information may be any first modal information among multiple first modal information that has the same modality as the first modal information.

[0206] As one implementation, a first direct correlation subset corresponding to the first modal information and the multimodal model, and the similarity between them and the first direct correlation subset corresponding to the second other first modal information and the multimodal model are determined to obtain the similarity between the first modal information and the second other first modal information corresponding to the multimodal model. Optionally, the similarity between the first modal information and the second other first modal information corresponding to the multimodal model is obtained based on the first modal features corresponding to the first modal information and the multimodal model, the first modal features corresponding to the second other first modal information and the multimodal model, and the first modal features corresponding to at least one first associated modal information. A fourth comprehensive similarity corresponding to the first modal information and the multimodal model is obtained based on the similarity between the first modal information and at least one second other first modal information corresponding to the multimodal model. Based on the similarity between the first modal information corresponding to the multimodal model and the second other first modal information, and the fourth comprehensive similarity between the first modal information and the multimodal model, the first indirect correlation degree between the first modal information and the second other first modal information corresponding to the multimodal model is obtained.

[0207] For example, multiple first modal information also include first modal information VF2 and first modal information VF3 corresponding to the text. First modal information VF2 and first modal information VF3 can serve as second other first modal information corresponding to first modal information VF1. Based on obtaining the first direct correlation subset DCS... Tm-VF1 Similarly, the first direct correlation subset DCS corresponding to the first modal information VF2 and the m-th teacher model Tm is obtained. Tm-VF2 And the first direct correlation subset DCS corresponding to the first modal information VF3 and the m-th teacher model Tm. Tm-VF3 .

[0208] Determine the first direct correlation subset DCS Tm-VF1 With the first direct correlation subset DCS Tm-VF2 The similarity between them is obtained by calculating the similarity Sim between the first modal information VF1 and the first modal information VF2 corresponding to the m-th teacher model Tm. Tm-VF1-VF2 Similarly, the similarity Sim between the first modal information VF1 and the first modal information VF3 corresponding to the m-th teacher model Tm is obtained. Tm-VF1-VF3 Then, based on the similarity Sim... Tm-VF1-VF2 Similarity Sim Tm-VF1-VF3 The fourth comprehensive similarity CSim4 corresponding to the first modal information VF1 and the m-th teacher model Tm is obtained. Tm-VF1 Finally, the similarity Sim was determined. Tm-VF1-VF2Similarity to the fourth comprehensive similarity CSim4 Tm-VF1 The ratio between them yields the first indirect correlation IC between the first modal information VF1 and the first modal information VF2 corresponding to the m-th teacher model Tm. Tm-VF1-VF2 Similarly, the first indirect correlation IC between the first modal information VF1 and the first modal information VF3 corresponding to the m-th teacher model Tm is obtained. Tm-VF1-VF3 Therefore, the first indirect correlation subset ICS corresponding to the first modal information VF1 and the m-th teacher model Tm is obtained. Tm-VF1 .

[0209] As another implementation, the similarity between the first direct correlation subset corresponding to the first modal information and the multimodal model, and the first direct correlation subset corresponding to the second other first modal information and the multimodal model is determined, thereby obtaining the first indirect correlation between the first modal information corresponding to the multimodal model and the second other first modal information.

[0210] For example, similarity Sim can be used Tm-VF1-VF2 The first indirect correlation IC is determined to be the first modal information VF1 and the first modal information VF2 corresponding to the m-th teacher model Tm. Tm -VF1-VF2. Simulate the similarity. Tm -VF1-VF3 is determined as the first indirect correlation IC between the first modal information VF1 and the first modal information VF3 corresponding to the m-th teacher model Tm. Tm-VF1-VF3 .

[0211] As another implementation, a first indirect correlation degree between the first modal information corresponding to the multimodal model and the second other first modal information is obtained based on the first modal features corresponding to the first modal information and the multimodal model, the first modal features corresponding to the second other first modal information and the multimodal model, and the first modal features corresponding to at least one first associated modal information.

[0212] The first associated modal information can be a first modal information with context association function among multiple first modal information. The context association function can refer to the function used to associate the first modal information with the second other first modal information.

[0213] (2) For the indirect graph structure-based acquisition of the first correlation subset corresponding to the first modality information and the multimodal model.

[0214] Since the method proposed in this application for obtaining the first correlation subset corresponding to the first modality information and the multimodal model based on graph structure is based on obtaining the first-order proximity subset and the second-order proximity subset corresponding to the node based on graph structure, for ease of understanding, the following will first explain how to obtain the first-order proximity subset and the second-order proximity subset corresponding to the node based on graph structure.

[0215] First, we will explain how to obtain the first-order proximity subsets corresponding to nodes based on the graph structure.

[0216] For example, a graph structure can be represented by G = {V, E}. V can represent a set of nodes. V can include multiple nodes. For example, V = {v1, v2, ..., v...} i , ......, v H-1 v H}. v i A node i in V can be represented by a variable. H can represent the number of nodes. H can be an integer greater than 1. i can be an integer greater than or equal to 1 and less than or equal to H, i.e., i∈{1, 2, ..., H-1, H}. E can represent the set of edges.

[0217] With v i The corresponding first-order proximity subset can be determined according to the following formulas (1) to (3).

[0218] p1(v i v j ) = sim(u i u j (1)

[0219]

[0220] p1(·|v i )=[p1(v1|v i ), ......, p1(v j |v i ), ......, p1(v H |v i (3)

[0221] Among them, v j A node j in V can be represented. j∈{1,2,......,H-1,H}. k A node k in V can be represented. k∈{1,2,......,H-1,H}. i≠j. i≠k. i It can characterize v i The corresponding modal feature i. u j It can characterize v jThe corresponding modal feature j. p1(v i v j ) can characterize v i With v j The first similarity between them. sim(u i u j ) can characterize u i with u j The first similarity between them. p1(v i v k ) can characterize v i With v k The first similarity between them. p1(v j |v i ) can characterize v i With v j The first-order proximity between them. p1(v) k |v i ) can characterize v i With v k The first-order proximity between them. p1(·|v i ) can characterize v i The corresponding first-order proximity subset. It should be noted that if there is no edge connecting two nodes, the first-order proximity between them is zero. For example, if... Then p1(v i v j ) = 0. It can characterize v i With v j There are no edges connecting them.

[0222] The `sim()` function can include one of the following: cosine similarity function, modified cosine similarity function, Pearson correlation coefficient function, Spearman rank correlation coefficient function, Kendall rank correlation coefficient function, Dice coefficient function, Jacquard similarity function, Euclidean distance function, and Manhattan distance function. For example, `sim()` is a cosine similarity function, therefore...

[0223] Secondly, based on obtaining the first-order proximity subsets corresponding to nodes based on the graph structure, the following explains how to obtain the second-order proximity subsets corresponding to nodes based on the graph structure.

[0224] Based on formulas (1) to (3), and with v i The corresponding second-order proximity subsets can be determined according to the following formulas (7) to (9).

[0225] p2(v i v j )=sim(p1(·|v i), p1(·|v j (7)

[0226]

[0227] p2(·|v i )=[p2(v1|v i ), ......, p2(v j |v i ), ......, p2(v H |v i (9)

[0228] Among them, p2(v i v j ) can characterize v i With v j The third similarity between them. p2(v i v k ) can characterize v i With v k The third similarity between them. sim(p1(·|v i ), p1(·|v j )) can characterize p1(·|v i ) and p1(·|v j The similarity between p1(·|v) i ) can characterize v i The corresponding first-order proximity subset. p1(·|v j ) can characterize v j The corresponding first-order proximity subset. p2(v j |v i ) can characterize v i With v j The second-order proximity between them. p2(v) k |v i ) can characterize v i With v k The second-order proximity between them. p2(·|v i ) can characterize v i The corresponding second-order proximity subset.

[0229] Based on the above, the following will explain the method by which this application obtains the first set of correlation degree corresponding to the first modality information and the multimodal model based on the graph structure, namely the first direct correlation degree subset (i.e., the first-order proximity degree subset) and the first indirect correlation degree subset (i.e., the second-order proximity degree subset) corresponding to the first modality information and the multimodal model.

[0230] It should be noted that there is no first edge connection between two first modal information pieces of the same modality; therefore, the first direct correlation between two first modal information pieces of the same modality is zero. There may be a first edge connection between two first modal information pieces of different modalities.

[0231] First, we will explain in general how to obtain the first subset of correlation degrees corresponding to the first modality information and the multimodal model based on the graph structure.

[0232] Based on the structure of the first graph and the first multimodal features corresponding to the multimodal model, a first set of correlation degrees corresponding to the first multimodal information and the multimodal model is obtained. The first set of correlation degrees corresponding to the first multimodal information and the multimodal model may include the first correlation degree subsets corresponding to each of the multiple first modal information items corresponding to the multimodal model.

[0233] The first graph structure can be constructed based on the first multimodal information, that is, by representing the multiple first modal information included in the first multimodal information as their respective corresponding first nodes. Two first nodes with a direct relationship are connected by a first edge, thereby constructing the first graph structure corresponding to the first multimodal information. It can be understood that the first graph structure can include multiple first nodes and at least one first edge. A first node can represent first modal information. A first edge can be used to connect two first nodes with a direct relationship. The two first modal information nodes with a direct relationship have different modalities. It should be noted that the first graph structure constructed based on the first multimodal information can be understood as the initial graph structure. The first degree of association between two first nodes in the initial graph structure is not yet determined.

[0234] For example, the first graph structure can be represented using the same representation method as the graph structure described above. That is, the first graph structure can be represented by G = {V, E}. V can represent a set of nodes. V can include multiple first nodes. For example, V = {v1, v2, ..., v...} i , ......, v H-1 v H}. v i This can represent the first node i in V. i The first modality information i can be represented. H can represent the number of first nodes. H can be an integer greater than 1. i can be an integer greater than or equal to 1 and less than or equal to H, i.e., i∈{1,2,...,H-1,H}. E can represent the edge set. The edge set can include at least one first edge.

[0235] For a multimodal model among multiple multimodal models, the first correlation degree between two first nodes corresponding to the multimodal model in the first graph structure corresponding to the first multimodal information can be determined based on the first graph structure corresponding to the first multimodal information and the first multimodal features corresponding to the first multimodal information and the multimodal model. It should be noted that the first graph structure corresponding to the first multimodal information and the multimodal model can be called the target graph structure. The first correlation degree between two first nodes in the target graph structure has been determined.

[0236] Figure 11 An example schematic diagram of a first figure structure according to an embodiment of this application is shown.

[0237] like Figure 11 As shown, the first multimodal information may include multiple text information corresponding to text and multiple image information corresponding to images. Circle nodes can represent text information. Triangle nodes can represent image information. Figure 11 The first graph structure to the left of the middle arrow can be understood as the initial graph structure. Figure 11 The first graph structure to the right of the middle arrow can be understood as the target graph structure. In the initial graph structure, the first degree of association between the two first nodes is not yet determined. In the target graph structure, the first degree of association between the two first nodes has been determined.

[0238] It's important to note that since the nodes and edges of a graph structure can be understood as relationships between objects, this aligns perfectly with the essence of zero-shot learning. Zero-shot learning utilizes the mapping between known and unknown modal information to obtain a representation of the unknown modal information. Therefore, by obtaining the first correlation between the first modal information based on the first graph structure, these correlations can constrain the model parameters of the multimodal model, acting as a constraint term in the loss function, such as a regularization term, thus guiding the loss function and the direction of information propagation. In other words, by introducing the representation of the first graph structure during the training process of the student model, the correlations built based on the first graph structure constrain the loss function and the direction of information propagation. This not only alleviates the conflict between maintaining zero-shot learning capability and adapting to new information but also enhances the adaptability and stability of the multimodal model in continuous learning.

[0239] The following section will further explain how to obtain the first set of proximity subsets corresponding to the first modality information and the multimodal model based on graph structures, building upon the above description of obtaining the first set of proximity subsets corresponding to the first modality information and the multimodal model based on graph structures.

[0240] (2.1) For obtaining the first direct association subset based on the first graph structure

[0241] At least one first directly connected node pair is determined from the first graph structure. The first directly connected node pair may include two first nodes connected by a first edge. That is, a first directly connected node pair with a first edge connection is determined from the first graph structure. For each of the at least one first directly connected node pair, the similarity between the first modal features corresponding to the two first nodes in that first directly connected node pair is determined, thus obtaining a first similarity between the two first nodes in that first directly connected node pair. Therefore, a first similarity can be obtained between the two first nodes in each of the at least one first directly connected node pair. Optionally, the similarity may include one of the following: cosine similarity, modified cosine similarity, Pearson correlation coefficient, Spearman rank correlation coefficient, Kendall rank correlation coefficient, Dice coefficient, Jacquard similarity, Euclidean distance, and Manhattan distance, etc.

[0242] For example, based on the above formula (1), the first similarity between the two first nodes in the first direct-connected node pair can be determined according to the following formula (13).

[0243]

[0244] Among them, v i This can represent the first node i in V. i This can represent the first modal information i. i∈{1,2,......,H-1,H}. j This can represent node j in V. j This can represent the first modal information j. j∈{1,2,......,H-1,H}. i It can characterize v i The corresponding first modal feature i. u j It can characterize v j The corresponding first modal feature j. p1(v i v j ) can characterize v i With v j The first similarity between them. sim(u i u j ) can characterize u i with u j The similarity between them. (i, j) ∈ E can characterize v i With v j There is a first edge connecting them. (v) i v j ) can characterize including v i and v j The first directly connected node pair. It can characterize v i With v j There is no first edge connecting them.

[0245] Based on obtaining the first similarity between two first nodes in each of at least one first directly connected node pair, a first direct correlation subset corresponding to each of the multiple first modalities can be obtained according to the first similarity between the two first nodes in each of the at least one first directly connected node pair. For example, for a first node in at least one first directly connected node pair, at least one target first directly connected node pair including the first node can be determined from the at least one first directly connected node pair. Based on this, a first direct correlation subset corresponding to the first modalities represented by the first node is obtained according to the first similarity between the two first nodes in each of the at least one target first directly connected node pair.

[0246] The method for obtaining a subset of first direct association degrees corresponding to the first modality information represented by the first node based on the first similarity between two first nodes in each of at least one target first direct connected node pair can be achieved in the following way.

[0247] As one implementation, a first comprehensive similarity corresponding to a first node is obtained based on the first similarity between the two first nodes in each of at least one target first directly connected node pair. For each target first directly connected node pair in the at least one target first directly connected node pair, a second similarity between the two first nodes in the target first directly connected node pair is obtained based on the first comprehensive similarity corresponding to the first node and the first similarity between the two first nodes in the target first directly connected node pair. Based on the second similarity between the two first nodes in the target first directly connected node pair, a first direct correlation degree between the two first nodes in the target first directly connected node pair is obtained. For example, a first ratio between the first similarity between the two first nodes in the target first directly connected node pair and the first comprehensive similarity corresponding to the first node can be determined. This first ratio is determined as the second similarity between the two first nodes in the target first directly connected node pair. The second similarity between the two first nodes in the target first directly connected node pair can be determined as the first direct correlation degree between the two first nodes in the target first directly connected node pair.

[0248] For example, based on the above formulas (1), (3) and (13), the first direct correlation degree between the two first nodes in the first direct connection node pair of the target can be determined according to the following formula (14).

[0249]

[0250] Among them, p1(vj |v i ) can represent v in the first directly connected node pair of the target. i With v j The first direct correlation between them. p1(v) i v j ) can represent v in the first directly connected node pair of the target. i With v j The first similarity between them. p1(v i v k ) can represent v in the first directly connected node pair of the target. i With v k The first similarity between them. ∑ (i，k)∈E p1(v i v k ) can characterize v i The corresponding first comprehensive similarity.

[0251] It should be noted that the difference between formula (14) and formula (2) is that v k The ranges they belong to are different. In formula (14), v k It is V and v i There exists a first node of the first edge. In formula (2), v k It is V excluding v i Any node other than that.

[0252] Based on the above, a subset of the first direct correlation degree corresponding to the first modal information represented by the first node can be obtained according to the first direct correlation degree between the two first nodes of each of the at least one target first direct connected node pair.

[0253] For example, based on the above formula (14), the first direct correlation subset corresponding to the first modal information represented by the first node can be determined according to the following formula (15).

[0254] p1(·|v i )=[p1(v1|v i ), ......, p1(v j |v i ), ......, p1(v H |v i (15)

[0255] Among them, p1(·|v i ) can characterize v i The subset of the first direct correlation degree corresponding to the first modal information i represented.

[0256] It should be noted that, regarding the first direct correlation, since the student model learns the correlation between first-modal information from different first modalities, the correlation between first-modal information from different first modalities with matching relationships is stronger, while the correlation between first-modal information from different first modalities without matching relationships is weaker. Thus, the student model can learn first-modal features with good generalization ability. The student model's classification mechanism is based on the similarity between first-modal information from different first modalities, which is similar to the purpose of the first direct correlation. Therefore, in the above case, the first direct correlation can effectively align with the student model's classification mechanism. Furthermore, if a loss function term corresponding to the first direct correlation is added to the loss function, this term can serve as a constraint term. For example, the constraint term can include a regularization term; that is, the loss function term corresponding to the first direct correlation can serve as a regularization term.

[0257] (2.2) Regarding how to obtain the first indirect correlation subset based on the first graph structure

[0258] At least one pair of first inter-node connections is determined from the first graph structure. A pair of first inter-node connections may include two first nodes that are not connected by a first edge. That is, a pair of first inter-node connections that are not connected by a first edge is determined from the first graph structure. For each pair of first inter-node connections, a third similarity is obtained between the two first nodes in the pair based on the first association information corresponding to that pair.

[0259] To determine how to obtain the third similarity between the two first nodes in the first connection node pair based on the first association information corresponding to the first connection node pair, the following method can be used.

[0260] As one implementation, a third similarity between the two first nodes in a first linked node pair is obtained based on a subset of the first direct association degrees corresponding to each of the two first nodes in that first linked node pair. For example, the third similarity between the two first nodes in a first linked node pair is obtained by determining the similarity between the subsets of the first direct association degrees corresponding to each of the two first nodes in that first linked node pair. Thus, a third similarity can be obtained between the two first nodes in at least one first linked node pair. Optionally, the similarity may include one of the following: cosine similarity, modified cosine similarity, Pearson correlation coefficient, Spearman rank correlation coefficient, Kendall rank correlation coefficient, Dice coefficient, Jacquard similarity, Euclidean distance, and Manhattan distance, etc.

[0261] For example, based on the above formula (15), the third similarity between the two first nodes in the first pair of nodes can be determined according to the following formula (19).

[0262]

[0263] in, It can characterize v i With v j There is no first edge connecting them. (v) i v j ) can characterize including v i and v j The first pair of connected nodes. p2(v i v j ) can represent v in the first pair of connected nodes i With v j The third similarity between them. sim(p1(·|v i ), p1(·|v j )) can characterize p1(·|v i ) and p1(·|v j The similarity between p1(·|v) i ) can characterize v i The corresponding first direct association subset. p1(·|v j ) can characterize v j The corresponding first direct correlation subset.

[0264] It should be noted that in formula (19), the first indirect correlation degree can be the similarity between first nodes that do not have a first edge connection, identified based on neighborhood similarity. This application utilizes multimodal proximity established by a multimodal model to construct the first indirect correlation degree between first modal information of the same modality, that is, the proximity between first nodes of the same other modality can be defined by using the contextual correlation provided by the contextual modality. For example, the contextual modality can be text. Other modalities can be images. Optionally, the contextual modality can be images. Other modalities can be text. This approach differs from standard pairwise similarity. Standard pairwise similarity typically does not include multimodal information. Through the above approach provided by this application, the unique advantages of multimodal models in capturing cross-modal interactions are utilized, which can enhance the modeling of same-modal proximity.

[0265] As another implementation, a third similarity is obtained between the first nodes in the first connected node pair and the first modal information corresponding to each of the at least one first associated node, as well as the first modal features corresponding to each of the two nodes in the first connected node pair. The first associated node can represent the first node in the first graph structure that functions as a contextual link. The contextual link function can represent the function of linking the two first nodes in the first connected node pair. Thus, a third similarity can be obtained between the first nodes in each of the at least one first connected node pair.

[0266] For example, based on the fact that sim() is a cosine similarity function, formula (19) can be simplified to obtain formula (20).

[0267]

[0268] Wherein, p'2(v i v j ) can represent v in the first pair of connected nodes i With v j The third similarity between them. It can characterize U m The transpose of . It can characterize and The corresponding first modal feature. It can represent the first associated node m h . First mode information corresponding to the m-th mode It can characterize u i The transpose of H. m It can represent the number of first-mode information points corresponding to the m-th mode in the first multimodal information. H m It can be an integer greater than or equal to 1 and less than H. The m-th modality can refer to the modality corresponding to the first node that serves as the context association.

[0269] For example, in relation to v i and v j When the corresponding modality is image, and the modality corresponding to the first node that serves as the contextual linker is text, m can be represented by T. Therefore, H m It can represent the number of first modal information corresponding to the text in the first multimodal information.

[0270] It should be noted that formula (20) demonstrates that the first indirect correlation determination scheme (i.e., the integrated weighted similarity scheme) of this application is superior to the pairwise comparison based on the cosine similarity function, which can be expressed as u i ·um By integrating weighted similarity, the weights can be provided by the shared connections of the first node corresponding to the m-th modality, thus enabling a more comprehensive relationship analysis.

[0271] Based on obtaining a third similarity between two first nodes in each of at least one pair of first inter-node connections, a subset of first indirect correlation degrees corresponding to each of the multiple first modalities can be obtained according to the third similarity between two first nodes in each of the at least one pair of first inter-node connections. For example, for a first node in at least one pair of first inter-node connections, at least one target pair of first inter-node connections including that first node can be determined from the at least one pair of first inter-node connections. Based on this, a subset of first indirect correlation degrees corresponding to the first modalities represented by that first node can be obtained according to the third similarity between two first nodes in each of the at least one target pair of first inter-node connections.

[0272] To obtain a subset of the first indirect correlation degree corresponding to the first modality information represented by the first node based on the third similarity between two of the first nodes in at least one target first indirect connection node pair, the following method can be used.

[0273] As one implementation, a second comprehensive similarity corresponding to a first node is obtained based on a third similarity between two first nodes in each of at least one target first inter-node pair. For each target first inter-node pair, a fourth similarity is obtained based on the second comprehensive similarity corresponding to the first node and the third similarity between the two first nodes in the target first inter-node pair. A first indirect correlation degree is obtained based on the fourth similarity between the two first nodes in the target first inter-node pair. For example, a second ratio between the third similarity between the two first nodes in the target first inter-node pair and the second comprehensive similarity corresponding to the first node can be determined. This second ratio is determined as the fourth similarity between the two first nodes in the target first inter-node pair. The fourth similarity between the two first nodes in the target first inter-node pair can be determined as the first indirect correlation degree between the two first nodes in the target first inter-node pair. The first indirect correlation degree corresponds to the first modal information represented.

[0274] For example, as an implementation method, based on the above formula (19), the first indirect correlation degree between the two first nodes in the first indirect connection node pair of the target can be determined according to the following formula (21).

[0275]

[0276] Among them, p2(v j |v i ) can represent v in the first pair of connected nodes of the target. i With v j The first degree of indirect correlation between them. p2(v i v j ) can represent v in the first pair of connected nodes of the target. i With v j The third similarity between them. p2(v i v k ) can represent v in the first pair of connected nodes of the target. i With v k The third similarity between them. It can characterize v i The corresponding second comprehensive similarity.

[0277] For example, as another implementation, based on the above formula (20), the first indirect correlation degree between the two first nodes in the first indirect connection node pair of the target can be determined according to the following formula (22).

[0278]

[0279] Wherein, p'2(v j |v i ) can represent v in the first pair of connected nodes of the target. i With v j The first degree of indirect correlation between them. p'2(v i v j ) can represent v in the first pair of connected nodes of the target. i With v j The third similarity between them. p'2(v i v k ) can represent v in the first pair of connected nodes of the target. i With v k The third similarity between them. It can characterize v i The corresponding second comprehensive similarity.

[0280] It should be noted that the difference between formulas (21) and (22) and formula (8) lies in v k The ranges they belong to are different. In formulas (21) and (22), v... k It is V and v i There is no first node of the first edge between them. In formula (8), v k It is V excluding v i Any first node other than the one mentioned above.

[0281] Based on the above, a subset of the first indirect correlation degree corresponding to the first modal information represented by the first node can be obtained according to the first indirect correlation degree between two of the first nodes in each of the at least one target first indirect connection node pairs.

[0282] For example, based on the above formula (21) or formula (22), the first indirect correlation subset corresponding to the first modal information represented by the first node can be determined according to the following formula (23).

[0283] p2(·|v i )=[p2(v1|v i ), ......, p2(v j |v i ), ......, p2(v H |v i )] (twenty three)

[0284] Wherein, p2(·|v i ) can characterize v i The subset of indirect correlation degree corresponding to the first modality information i represented.

[0285] It should be noted that, for the first indirect correlation degree, the advantages of multimodal models in capturing cross-modal interactions can be effectively utilized to enhance the modeling of same-modal proximity.

[0286] The method proposed in this application for obtaining the first indirect correlation subset and the first indirect correlation subset corresponding to the first modality information and the multimodal model based on the first graph structure can capture intramodal and intermodal interactions.

[0287] The following explains how to obtain the first proximity degree corresponding to the first modality information, the student model, and the teacher model in operation S1030.

[0288] In the case where the first subset of correlation degree corresponding to the first modality information includes a first subset of direct correlation degree and a first subset of indirect correlation degree, as one implementation method, the proximity between the first subset of direct correlation degree corresponding to the first modality information and the student model, and the first subset of direct correlation degree corresponding to the first modality information and the teacher model can be determined to obtain a second proximity degree corresponding to the first modality information, the student model, and the teacher model. The proximity between the first subset of indirect correlation degree corresponding to the first modality information and the student model, and the first subset of indirect correlation degree corresponding to the first modality information and the teacher model can be determined to obtain a third proximity degree corresponding to the first modality information, the student model, and the teacher model. Based on the second and third proximity degrees corresponding to the first modality information, the student model, and the teacher model, the first proximity degree corresponding to the first modality information, the student model, and the teacher model is obtained.

[0289] In the case where the first correlation subset corresponding to the first modality information includes the first direct correlation subset, as an implementation method, the proximity between the first direct correlation subset corresponding to the first modality information and the student model, and the first direct correlation subset corresponding to the first modality information and the teacher model can be determined to obtain the first proximity corresponding to the first modality information, the student model and the teacher model.

[0290] In the case where the first correlation subset corresponding to the first modality information includes a first indirect correlation subset, as an implementation method, the proximity between the first indirect correlation subset corresponding to the first modality information and the student model, and the first indirect correlation subset corresponding to the first modality information and the teacher model can be determined to obtain the first proximity corresponding to the first modality information, the student model, and the teacher model.

[0291] The determination of the second proximity to the first modality information, the student model, and the teacher model can be achieved in the following way.

[0292] As one implementation, based on the first loss function sub-term, a second proximity degree corresponding to the first modality information, the student model, and the teacher model is obtained according to the first direct correlation degree subsets corresponding to the first modality information and the student model, and the first direct correlation degree subsets corresponding to the first modality information and the teacher model. The first loss function sub-term can be used to evaluate the proximity degree between the respective first direct correlation degree subsets of the student model and the teacher model corresponding to the same first modality information, that is, to evaluate the proximity degree between the student model and the teacher model. The first loss function sub-term can be configured according to actual business needs and is not limited here. For example, the first loss function sub-item may include one of the following: Cross Entropy (CE) loss function, KL (Kullback-Leibler) divergence loss function, JS (Jensen-Shannon) divergence loss function, Bulldozer (Wasserstein) distance loss function, cosine similarity loss function, modified cosine similarity loss function, Pearson correlation coefficient loss function, Spearman rank correlation coefficient loss function, Kendall rank correlation coefficient loss function, Dice coefficient loss function, Jacquard similarity loss function, Euclidean distance loss function, and Manhattan distance loss function, etc.

[0293] For example, based on formula (15), the second proximity corresponding to the first modality information, the student model and the teacher model can be determined according to the following formula (24).

[0294]

[0295] Among them, L PD1-Tm-S (v i ) can characterize v i The second proximity between the student model S and the teacher model Tm. It can characterize v i p1(·|v) corresponding to the teacher model Tm i ). It can characterize v i p1(·|v) corresponding to the student model S i p1(·|v i ) can characterize v i The corresponding first direct association subset. L CE () can characterize the cross-entropy loss function. m∈{1,2,......,M-1,M}. M can be an integer greater than or equal to 1.

[0296] The determination of the third proximity degree corresponding to the first modality information, the student model, and the teacher model can be achieved in the following way.

[0297] As one implementation, based on the second loss function sub-item, a third proximity degree corresponding to the first modality information, the student model, and the teacher model is obtained according to the first indirect correlation degree subsets corresponding to the first modality information and the student model, and the first indirect correlation degree subsets corresponding to the first modality information and the teacher model. The second loss function sub-item can be used to evaluate the proximity between the respective first indirect correlation degree subsets of the student model and the teacher model corresponding to the same first modality information, i.e., to evaluate the proximity between the student model and the teacher model. The second loss function sub-item can be configured according to actual business needs and is not limited here. For example, the second loss function sub-item can refer to the first loss function sub-item, which will not be elaborated further here.

[0298] For example, based on formula (23), the first indirect correlation subset corresponding to the first modality information, the student model and the teacher model can be determined according to the following formula (25).

[0299]

[0300] Among them, L PD2-Tm-S (v i ) can characterize v i The third proximity between the student model S and the teacher model Tm. It can characterize v i p2(·|v) corresponding to the teacher model Tm i ). It can characterize v i p2(·|v) corresponding to the student model S i p2(·|v i ) can characterize v i The corresponding first indirect correlation subset.

[0301] The following explains how to train the student model in operation S1040.

[0302] The determination of the fourth proximity degree corresponding to the first modality information can be achieved in the following way.

[0303] As one implementation, based on the second loss function term, a fourth proximity degree corresponding to the first modality information is obtained according to the first direct correlation subset and the first true direct correlation subset corresponding to the first modality information. The second loss function term can be used to evaluate the proximity between the first true direct correlation subset corresponding to the same first modality information and the first direct correlation subset of the student model. The second loss function term can be configured according to actual business needs and is not limited here. For example, the second loss function term can refer to the first loss function sub-term, which will not be elaborated further here.

[0304] For example, based on formula (15), the fourth proximity degree corresponding to the first modality information and the student model can be determined according to the following formula (26).

[0305]

[0306] Among them, L CE (v i ) can characterize v i The corresponding fourth proximity. It can characterize v i The corresponding first true direct correlation subset. It can characterize v i p1(·|v) corresponding to the student model S i p1(·|v i ) can characterize v i The corresponding first direct correlation subset.

[0307] The determination of at least one adjustable coefficient corresponding to each of the multiple first modal information can be achieved in the following way.

[0308] As one implementation, for a first modality among multiple first modality information, at least one distance corresponding to the first modality information is obtained based on the second association information corresponding to the first modality information. For example, for a teacher model in at least one teacher model, a first association subset corresponding to the first modality information and the student model is determined, as well as the distance between the first association subset corresponding to the first modality information and the teacher model, to obtain the distance corresponding to the first modality information, the student model, and the teacher model. The first association subset may be a first direct association subset. Optionally, the distance between the first modality feature corresponding to the first modality information and the student model, as well as the distance between the first modality feature corresponding to the first modality information and the teacher model, is determined to obtain the distance corresponding to the first modality information, the student model, and the teacher model. Based on this, for the first modality information, since the teacher model includes at least one, and the teacher model has a distance corresponding to the teacher model, the distance corresponding to the first modality information includes at least one. Based on the at least one distance corresponding to the first modality information, a comprehensive distance corresponding to the first modality information is obtained. For at least one teacher model, adjustable coefficients corresponding to the first modal information, the student model, and the teacher model are obtained based on the distances corresponding to the first modal information, the student model, and the teacher model, as well as the comprehensive distance corresponding to the first modal information. For example, the adjustable coefficients corresponding to the first modal information, the student model, and the teacher model can be obtained by determining the ratio between the distances corresponding to the first modal information, the student model, and the teacher model, and the comprehensive distance corresponding to the first modal information.

[0309] It should be noted that, for the first modal information among multiple first modal information, and for the teacher model among at least one teacher model, since the adjustable coefficients corresponding to the first modal information, the student model, and the teacher model can be determined based on at least one subset of first association degrees corresponding to the first modal information, the student model, and the teacher model, or can be determined based on at least one first modal feature corresponding to the first modal information, the student model, and the teacher model, the adjustable coefficients corresponding to the first modal information, the student model, and the teacher model can reflect the degree of contribution of the teacher model to processing the first modal information, and the degree of contribution of the teacher model among all teacher models. Therefore, based on the adjustable coefficients corresponding to the first modal information and the teacher model, the degree of learning that the first modal information should acquire from each teacher model can be determined, thereby determining the learning focus, so that for the first modal information, the student model focuses more on possessing abilities similar to which teacher model. The adjustable coefficients can characterize the degree of contribution. That is, the adjustable coefficients corresponding to the first modality information, the student model, and the teacher model can characterize the contribution of the teacher model among all teacher models in terms of its contribution to processing the first modality information.

[0310] For example, the larger the adjustable coefficient, the greater the contribution. At least one teacher model includes a teacher model with strong zero-shot learning ability and a teacher model with strong ability to adapt to new information. The teacher model with strong zero-shot learning ability can be a multimodal pre-trained model. The teacher model with strong ability to adapt to new information can be an older multimodal model. If the adjustable coefficient corresponding to the first modality information and the multimodal pre-trained model is greater than the adjustable coefficient corresponding to the first modality information and the older multimodal model, then for the first modality information, the student model will focus more on maintaining zero-shot learning ability. If the adjustable coefficient corresponding to the first modality information and the multimodal pre-trained model is less than the adjustable coefficient corresponding to the first modality information and the older multimodal model, then for the first modality information, the student model will focus more on adapting to new information.

[0311] For example, based on formula (15), the adjustable coefficients corresponding to the first modality information, the student model and the teacher model can be determined according to the following formula (27).

[0312]

[0313] Where, λ Tm-S (v i ) can characterize v i Adjustable coefficients corresponding to the student model S and the teacher model Tm. Can characterize and The distance between them. Can characterize and The distance between them. It can characterize v i p1(·|v) corresponding to the teacher model Tq i M can represent the number of teacher models. M can be an integer greater than 1. q can be an integer greater than or equal to 1 and less than or equal to M, i.e., q∈{1, 2, ..., M-1, M}.

[0314] The student model can be trained based on the first proximity, adjustable coefficient, and first correlation subset in the following way.

[0315] As one implementation, for each of the multiple first modal information types and for each teacher model in at least one teacher model, the first proximity degree and adjustable coefficient corresponding to the first modal information, the student model, and the teacher model are multiplied to obtain a first weighted proximity degree corresponding to the first modal information, the student model, and the teacher model. Based on at least one first weighted proximity degree corresponding to the first modal information, a first comprehensive weighted proximity degree corresponding to the first modal information is obtained. The proximity degree between the first direct correlation subset corresponding to the first modal information and the student model, and the first true direct correlation subset corresponding to the first modal information, is determined to obtain a fourth proximity degree corresponding to the first modal information. The student model is trained based on the first comprehensive weighted proximity degree and the fourth proximity degree corresponding to each of the multiple first modal information types.

[0316] For example, based on formulas (24) to (27), the first comprehensive weighted proximity degree and the fourth proximity degree corresponding to each of the multiple first modal information can be determined according to the following formulas (28) to (30).

[0317] L = L PD +L CE (28)

[0318]

[0319] Here, H represents the number of first modal information items included in the first multimodal information. H can be an integer greater than 1. i can be an integer greater than or equal to 1 and less than or equal to H, i.e., i∈{1, 2, ..., H-1, H}. M represents the number of teacher models. M can be an integer greater than 1. m can be an integer greater than or equal to 1 and less than or equal to M, i.e., m∈{1, 2, ..., M-1, M}. λ Tm-S (v i (L) PD1-Tm-S (v i )+LPD2-Tm-S (v i )) can characterize v i The corresponding first comprehensive weighted proximity. L CE (v i ) can characterize v i The corresponding fourth proximity.

[0320] For example, when M=2, formula (29) can be expressed as formula (31) below.

[0321]

[0322] The student model can be trained based on the first proximity, predetermined coefficient, and first correlation subset in the following way.

[0323] As one implementation, for each of the multiple first modal information types and for each teacher model in at least one teacher model, a first proximity score corresponding to the first modal information, the student model, and the teacher model is multiplied by a predetermined coefficient to obtain a second weighted proximity score corresponding to the first modal information, the student model, and the teacher model. Based on at least one second weighted proximity score corresponding to the first modal information, a second comprehensive weighted proximity score corresponding to the first modal information is obtained. The proximity scores between a first direct correlation subset corresponding to the first modal information and the student model, and a first true direct correlation subset corresponding to the first modal information, are determined to obtain a fourth proximity score corresponding to the first modal information. The student model is trained based on the second comprehensive weighted proximity score and the fourth proximity score corresponding to each of the multiple first modal information types.

[0324] For example, based on formulas (24) to (27), the second comprehensive weighted proximity degree and the fourth proximity degree corresponding to each of the multiple first modal information can be determined according to formulas (28) and (30) above, and formula (32) below.

[0325]

[0326] Where, λ Tm-S λ can represent the predetermined coefficients corresponding to the student model S and the teacher model Tm. Tm-S It can be a hyperparameter. λ Tm-S Configuration can be tailored to specific business needs; no restrictions are set here. For example, λ Tm-S It can be a number greater than 0 and less than or equal to 1. λ Tm-S (L PD1-Tm-S (v i )+L PD2-Tm-S (v i )) can characterize v i The corresponding second comprehensive weighted proximity.

[0327] The determination of the fifth proximity degree corresponding to the second modality information can be achieved in the following way.

[0328] As one implementation, based on the second loss function term, a fifth proximity degree corresponding to the second modality information is obtained according to the second direct correlation subset and the second true direct correlation subset corresponding to the second modality information. The second loss function term can be used to evaluate the proximity between the second true direct correlation subset and the second direct correlation subset corresponding to the same second modality information. The second loss function term can be configured according to actual business needs and is not limited here. For example, the second loss function term can refer to the first loss function sub-term, which will not be elaborated here.

[0329] For example, based on formula (15), the fifth proximity degree corresponding to the second modality information and the student model can be determined according to the following formula (33).

[0330]

[0331] Among them, L CE (v j ) can characterize v j The corresponding fifth degree of proximity. It can characterize v j The corresponding second true direct correlation subset. It can characterize v j p1(·|v) corresponding to the student model S j p1(·|v j ) can characterize v j The corresponding first direct association subset. j This can represent the second node j. j J can represent the second modal information j. J can represent the number of second modal information included in the second multimodal information. J can be an integer greater than 1. j can be an integer greater than or equal to 1 and less than or equal to J, i.e., j∈{1,2,...,J-1,J}.

[0332] The student model can be trained based on at least one first proximity degree corresponding to each of the multiple first modal information and a subset of second correlation degrees corresponding to each of the multiple second modal information in the following way.

[0333] As one implementation, for a second modality among multiple second modal information, the proximity between a second direct correlation subset and a second true direct correlation subset corresponding to that second modality information is determined, resulting in a fifth proximity score corresponding to that second modality information. A student model is trained based on at least one first proximity score corresponding to each of the multiple first modal information and the fifth proximity scores corresponding to each of the multiple second modal information. For example, for a first modality among multiple first modal information, a comprehensive proximity score corresponding to that first modality information is obtained based on at least one first proximity score. A first current multimodal model is trained based on the comprehensive proximity scores corresponding to each of the multiple first modal information and the fifth proximity scores corresponding to each of the multiple second modal information.

[0334] As another implementation, a student model is trained based on at least one first proximity degree and at least one adjustable coefficient corresponding to each of the multiple first modal information, and a second correlation subset corresponding to each of the multiple second modal information. For example, for a first modal information among the multiple first modal information, and for a teacher model among at least one teacher model, the first proximity degree and adjustable coefficient corresponding to the first modal information, the student model, and the teacher model are multiplied to obtain a first weighted proximity degree corresponding to the first modal information, the student model, and the teacher model. Based on at least one first weighted proximity degree corresponding to the first modal information, a first comprehensive weighted proximity degree corresponding to the first modal information is obtained. For a second modal information among the multiple second modal information, the proximity degree between a second direct correlation subset and a second true direct correlation subset corresponding to the second modal information is determined to obtain a fifth proximity degree corresponding to the second modal information. The student model is trained based on the first comprehensive weighted proximity degree corresponding to each of the multiple first modal information and the fifth proximity degree corresponding to each of the multiple second modal information.

[0335] For example, based on formulas (24) to (27), the first comprehensive weighted proximity degree corresponding to each of the multiple first modal information and the fifth proximity degree corresponding to each of the multiple second modal information can be determined according to formulas (28) to (29) above and formula (34) below.

[0336]

[0337] Among them, L CE (v j ) can characterize v j The corresponding fifth degree of proximity.

[0338] As another implementation, a student model is trained based on at least one first proximity degree and at least one predetermined coefficient corresponding to each of the multiple first modal information, and a second correlation degree subset corresponding to each of the multiple second modal information.

[0339] The student model can be trained based on the first proximity, the predetermined coefficient, and the second correlation subset in the following way.

[0340] As one implementation, for each of the multiple first modal information types and for each teacher model in at least one teacher model, a first proximity degree corresponding to the first modal information, the student model, and the teacher model is multiplied by a predetermined coefficient to obtain a second weighted proximity degree corresponding to the first modal information, the student model, and the teacher model. Based on at least one second weighted proximity degree corresponding to the first modal information, a second comprehensive weighted proximity degree corresponding to the first modal information is obtained. For each of the multiple second modal information types, a second direct association subset corresponding to the second modal information and the student model, and the proximity degree between them and a second true direct association subset corresponding to the second modal information are determined to obtain a fifth proximity degree corresponding to the second modal information. Based on the second comprehensive weighted proximity degrees corresponding to each of the multiple first modal information types and the fifth proximity degrees corresponding to each of the multiple second modal information types, a student model is trained.

[0341] For example, the second comprehensive weighted proximity degree corresponding to each of the multiple first modal information and the fifth proximity degree corresponding to the multiple second modal information can be determined according to the above formulas (28), (32) and (34).

[0342] It should be noted that a similar approach to obtaining the first set of association degrees corresponding to the student model can be used to obtain the second set of association degrees. For example, this can be achieved directly based on modal features, or indirectly based on graph structure. The method of obtaining the second set of association degrees indirectly based on graph structure will be explained below. For the method of obtaining the second set of association degrees directly based on modal features, please refer to the relevant section above; it will not be repeated here.

[0343] The specific implementation for indirectly obtaining the second correlation subset corresponding to the second modality information and the student model based on graph structure is as follows.

[0344] Based on the structure of the second graph and the second multimodal features corresponding to the student model, a second set of correlation degrees corresponding to the second multimodal information and the student model is obtained. This second set of correlation degrees can include subsets of second correlation degrees corresponding to each of the multiple second modal information features corresponding to the student model.

[0345] The second graph structure can be constructed based on the second multimodal information, that is, by representing each of the multiple second modal information included in the second multimodal information as a corresponding second node. Two second nodes with a direct relationship are connected by a second edge, thereby constructing the second graph structure corresponding to the second multimodal information. It can be understood that the second graph structure can include multiple second nodes and at least one second edge. Second nodes can represent second modal information. Second edges can be used to connect two second nodes with a direct relationship. The two directly related second modal information have different modalities. It should be noted that the second graph structure constructed based on the second multimodal information can be understood as the initial graph structure. The second degree of association between two second nodes in the initial graph structure is not yet determined.

[0346] Based on the second graph structure corresponding to the second multimodal information and the second multimodal features corresponding to the second multimodal information and the student model, the second correlation degree between the two second nodes corresponding to the student model in the second graph structure corresponding to the second multimodal information can be determined. It should be noted that the second graph structure corresponding to the second multimodal information and the student model can be called the target graph structure. The second correlation degree between the two second nodes in the target graph structure has been determined.

[0347] The following evaluation assesses the performance of the student model obtained using the training method for the multimodal model described in the embodiments of this application, in order to illustrate the performance of the student model in this application.

[0348] Performance evaluation metrics can include Transfer, Avg., and Last. Transfer characterizes the average performance of the tasks in the upper right triangle of the matrix. It can be calculated by averaging the performance of each task and making it equal to the weight of each dataset. Transfer measures the extent to which zero-shot learning ability is preserved. Avg. characterizes the model's average accuracy across all datasets. Last characterizes the average performance of all sequence tasks after continuous learning. Higher values ​​for Transfer, Avg., and Last indicate better model performance.

[0349] Table 1 schematically illustrates the performance evaluation results of student models obtained based on various technical solutions of the embodiments of this application. "×" indicates "not adopted," and "√" indicates "adopted." Therefore, in Table 1, the first row refers to "student models obtained based on the dual-teacher model combined with adjustable coefficients," the second row refers to "student models obtained based on the dual-teacher model combined with adjustable coefficients based on direct correlation," the third row refers to "student models obtained based on the dual-teacher model combined with adjustable coefficients based on indirect correlation," and the fourth row refers to "student models obtained based on the dual-teacher model combined with adjustable coefficients based on correlation."

[0350] As shown in Table 1, the student model obtained by combining the dual-teacher model based on correlation degree with the adjustable coefficient scheme has the best performance evaluation results. The performance evaluation results of the student model obtained by combining the dual-teacher model based on direct correlation degree with the adjustable coefficient scheme and the student model obtained by combining the dual-teacher model based on indirect correlation degree with the adjustable coefficient scheme are also close to the best.

[0351] 1st-order 2st-order Transfer Avg. Last × × 44.6 55.9 77.3 √ × 69.5 76.8 84.8 × √ 66.6 73.9 83.1 √ √ 69.8 76.9 85.1

[0352] Table 1

[0353] Table 2 schematically illustrates the performance evaluation results of the student models obtained by each technical solution based on the embodiments of this application. In Table 2, the first row refers to the "student model obtained by the first teacher model scheme based on correlation degree", the second row refers to the "student model obtained by the second teacher model scheme based on correlation degree", the third row refers to the "student model obtained by the dual teacher model based on correlation degree combined with the predetermined coefficient scheme", and the fourth row refers to the "student model obtained by the dual teacher model based on correlation degree combined with the adjustable coefficient scheme".

[0354] As shown in Table 2, the student model obtained by combining the correlation-based dual-teacher model with the adjustable coefficient scheme achieves the best performance on Transfer and Avg., and is also close to the best on Last. The performance evaluation indicators of student models obtained by other schemes are also close to or reach the best.

[0355] Scheme Transfer Avg. Last First Teacher Model Only 69.5 76.8 84.8 Second Teacher Model Only 69.4 76.9 85.4 The pre-determined coefficient is 0.5+ for the dual-teacher model. 69.1 76.9 85.4 Adjustable coefficient + dual-teacher model 69.8 76.9 85.1

[0356] Table 2

[0357] Table 3 schematically illustrates the performance evaluation results of various models based on the MTIL (Multi-domain Task Incremental Learning) benchmark. Order I refers to evaluating the model performance according to the alphabetical order of the datasets. The alphabetical order is as follows: Aircraft, Caltech101, CIFAR100, DTD, EuroSAT, Flowers, Food, MNIST, OxfordPet, StanfordCars, and SUN397. Order II refers to evaluating the model performance according to a random order of the data. The random order is as follows: StanfordCars, Food, MINST, OxfordPet, Flowers, SUN397, Aircraft, Caltech101, DTD, EuroSAT, and CIFAR100.

[0358] "Ours" refers to the student model obtained by combining a correlation-based dual-teacher model with an adjustable coefficient scheme as described in the embodiments of this application. "Zero-Shot" refers to a "multimodal pre-trained model," such as the initial CLIP model. "ContinualFT" refers to a model obtained by fine-tuning "Zero-Shot" in continuous learning. "LwF" refers to a "Learning without Forgetting" model. "LwF-VR" refers to a "Learning without Forgetting via Replayed Vocabularies" model. "iCaRL" refers to an "Incremental Classifier and Representation Learning" model. "WiSE-FT" can refer to an "ensembling the Weights of the Zero-Shot and Fine-Tuned" model. "ZSCL" refers to a "Zero-Shot in Continual Learning" model. "Δ" can refer to the difference between the performance evaluation metrics of "other models" and the performance evaluation metrics of "Zero-Shot."

[0359] As can be seen from Table 3, "Ours" has the best performance evaluation results.

[0360]

[0361] Table 3

[0362] Table 4 schematically illustrates the performance evaluation results of each model in Order I of Table 3 on various datasets. "Fine-Tuning" refers to fine-tuning directly on each dataset, which can serve as an upper bound to prevent catastrophic forgetting. As can be seen from Table 4, "Ours" achieves the best or near-best performance evaluation metrics on various datasets.

[0363]

[0364]

[0365] Table 4

[0366] Figure 12 A flowchart illustrating an information processing method according to an embodiment of this application is shown.

[0367] like Figure 12 As shown, the method 1200 may include operations S1210 to S1220.

[0368] In operation S1210, the third modality information is input into the trained student model to obtain the third modality features.

[0369] In operation S1220, processing information is obtained based on the third modal characteristics.

[0370] According to embodiments of this application, the trained student model can be obtained by using the training method for multimodal models described in embodiments of this application.

[0371] According to embodiments of this application, the modality of the third modality information can be configured according to actual business needs, and is not limited herein. For example, the modality of the third modality information can be one of text, image, and video. The information processing method can be applied to various downstream tasks. For example, downstream tasks can include at least one of the following: computer vision tasks, natural language processing tasks, and speech processing tasks. Computer vision tasks can include at least one of the following: visual classification tasks, visual retrieval tasks, object recognition tasks, visual segmentation tasks, visual question answering tasks, image description tasks, visual object detection tasks, object tracking tasks, cross-modal retrieval tasks, multimodal matching tasks, video summarization tasks, emotion recognition tasks, and person re-identification tasks. Natural language processing tasks can include at least one of the following: text classification tasks, text retrieval tasks, text translation tasks, and dialogue tasks. Speech processing tasks can include at least one of the following: speech recognition tasks, etc.

[0372] Taking visual retrieval as an example, the modality of the third modality information can be text. After obtaining the third modality features, the similarity between the third modality features and the respective first candidate modality features of multiple first candidate modality information can be determined, resulting in multiple fifth similarity scores. Based on these fifth similarity scores, the first candidate modality information matching the third modality information is determined from among the multiple first candidate modality information. The modality of the first candidate modality information can be an image. Thus, image information retrieval based on text information is achieved.

[0373] Taking information recommendation as an example, after obtaining the third modality feature, the similarity between the third modality feature and the respective second candidate modality features of multiple second candidate modality information can be determined, resulting in multiple sixth similarities. Based on these sixth similarities, second candidate modality information matching the third modality information is selected from the multiple second candidate modality information. This second candidate modality information matching the third modality information is then used as the recommendation information. The modality of the second candidate modality information can be different from that of the third modality information.

[0374] It should be noted that, because trained student models have good zero-shot learning capabilities and the ability to adapt to new information, applying trained student models to downstream tasks can provide more accurate, reliable, and efficient multimodal services, thereby meeting the needs of the target audience.

[0375] To facilitate understanding of the technical solutions of this application, the embodiments of this application will be combined with Figure 13 An exemplary system architecture for training methods, information processing methods, and apparatus of the multimodal model applicable to the examples of this application is described. It should be noted that... Figure 13 The figures shown are merely examples of system architectures applicable to embodiments of this application, intended to help those skilled in the art understand the technical content of this application. The positional relationships between the devices, components, modules, etc., shown in the figures do not constitute any limitation. For example, in Figure 13 In this context, the data storage system 1350 is an external memory relative to the execution device 1310. In other cases, the data storage system 1350 may also be placed within the execution device 1310.

[0376] Figure 13 The illustration schematically depicts an exemplary system architecture for training methods, information processing methods, and apparatus for multimodal models that can be applied according to embodiments of this application.

[0377] First, how to base it on Figure 13 The exemplary architecture shown illustrates a method for training multimodal models.

[0378] like Figure 13 As shown, the data acquisition device 1360 can be used to acquire first multimodal information. Furthermore, the data acquisition device 1360 can also be used to acquire second multimodal information.

[0379] After acquiring the first and second multimodal information, the data acquisition device 1360 can store the first and second multimodal information in the database 1330. The training device 1320 can train the model based on the data maintained in the database 1330 to obtain a trained student model 1301. The trained student model 1301 can realize the information processing function provided in this application.

[0380] For example, training device 1320 can be used to input first multimodal information into multiple multimodal models respectively, obtaining first multimodal features corresponding to each of the multiple multimodal models. For each multimodal model, a first association degree set corresponding to the multimodal model is determined based on the first multimodal features corresponding to the multimodal model. Based on the first association degree sets corresponding to each of the multiple multimodal models, at least one first proximity degree corresponding to each of the multiple first modal information is obtained. Based on the at least one first proximity degree corresponding to each of the multiple first modal information, a student model is trained.

[0381] For example, training device 1320 can also be used to input first multimodal information into multiple multimodal models respectively to obtain first multimodal features corresponding to each of the multiple multimodal models. For each multimodal model, a first association degree set corresponding to the multimodal model is determined based on the first multimodal features corresponding to the multimodal model. Based on the first association degree sets corresponding to each of the multiple multimodal models, at least one first proximity degree corresponding to each of the multiple first modal information is obtained. Second multimodal information is input into a student model to obtain second multimodal features corresponding to the student model. Based on the second multimodal features corresponding to the student model, a second association degree set corresponding to the student model is obtained. Based on at least one first proximity degree corresponding to each of the multiple first modal information and the second association degree subset corresponding to each of the multiple second modal information, the student model is trained.

[0382] It should be noted that the data maintained in database 1330 may not all come from the data acquisition device 1360; it may also be received from other devices. It should also be noted that training device 1320 may not necessarily train the student model entirely based on the data maintained in database 1330; it may also obtain training samples from the cloud or other devices for model training. The above description should not be construed as limiting the embodiments of this application.

[0383] The trained student model 1301, obtained by training the training device 1320, can be applied to different systems or devices; for example, it can be applied to... Figure 13 The execution device 1310 can be a terminal device, server, edge computing device, dedicated AI hardware, or cloud computing platform, etc.

[0384] For example, the terminal device may include at least one of the following: mobile phone, tablet computer, laptop computer, augmented reality (AR), virtual reality (VR), mixed reality (MR), extended reality (XR), wearable device, smart robot, and in-vehicle terminal, etc. Furthermore, the execution device 1310 may also be a server. The server may be a server that provides various services.

[0385] As one implementation approach, a lightweight version of the trained student model 1301 can be applied to a smartphone, enabling the smartphone to handle multimodal tasks. For example, multimodal tasks could include visual retrieval and information recommendation.

[0386] For example, a server can be a cloud server, also known as a cloud computing server or cloud host. It is a host product in the cloud computing service system, which solves the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability.

[0387] As one approach, for scenarios involving large amounts of data processing or complex tasks, the trained student model 1201 can be applied to the server to utilize the computing power provided by the server.

[0388] For example, an optimized and compressed trained student model can be applied to edge computing devices. Edge computing devices can include, for instance, smart cameras and drones, to enable real-time multimodal information processing.

[0389] For example, the trained student model 1301 can be applied to dedicated AI hardware. Dedicated AI hardware may include one of the following: GPU (Graphics Processing Unit) and TPU (Tensor Processing Unit), to achieve faster multimodal information processing speed and lower latency.

[0390] For example, the trained student model 1201 can also be deployed as a cloud service on a cloud computing platform and an API (Application Programming Interface) can be provided to objects for them to call.

[0391] It should also be noted that the trained student model 1301 can be provided to the object in the following manner.

[0392] As one implementation, an API based on the trained student model 1301 can be provided to objects. This allows objects to use the trained student model 1301 by calling the API, enabling the application of the trained student model 1301 to downstream tasks.

[0393] As an alternative implementation, customized services can be provided for the trained student model 1301.

[0394] As an alternative implementation, the trained student model 1301 can be integrated into a hardware device to provide plug-and-play multimodal interaction capabilities. For example, the hardware device could include one of the following: a smart camera or a smart assistant.

[0395] Next, regarding how to base it on Figure 13 The exemplary architecture shown is used to illustrate the information processing method.

[0396] exist Figure 13 In this embodiment, device 1310 configures input / output (I / O) interface 1312 for data exchange with external devices. Users can input data to I / O interface 1312 through client device 1340, and this data may include third modal information in this embodiment. It should be noted that the third modal information can be input in any manner.

[0397] Optionally, preprocessing modules 1313 and 1314 can be used to preprocess the input data received from I / O interface 1312. Calculation module 1311 is used to perform student-model-based information processing operations on the data input from preprocessing modules 1313 and 1314 or from I / O interface 1312 according to the aforementioned student model 1301.

[0398] For example, the calculation module 1311 can input the third modality information into the trained student model 1301 to obtain the third modality features. Based on the third modality features, processing information is obtained.

[0399] During the preprocessing of input data by the execution device 1310, or during the execution of information processing operations based on the student model by the calculation module 1311 of the execution device 1310, the execution device 1310 can call the data, code, etc. of the data storage system 1350 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 1350.

[0400] Finally, I / O interface 1312 returns the processed information to client device 1340 for use by the user.

[0401] exist Figure 13 In the scenario shown, the user can manually specify the data to be input into the execution device 1310. For example, by operating the interface provided by the I / O interface 1312, the user can manually input the third modality information. Alternatively, the client device 1340 can automatically input data into the I / O interface 1312 and obtain the results. If the client device 1340 requires user authorization to automatically input data, the user can set the appropriate permissions in the client device 1340. The user can view the results output by the execution device 1310 on the client device 1340; the specific presentation format can be at least one of display, sound, and motion. The client device 1340 can also act as a data acquisition end, collecting data such as... Figure 13 The input data of the input I / O interface 1312 and the third modality features output by the trained student model 1301 are used as new training samples and stored in the database 1330.

[0402] It should be noted that the entity executing the training method for multimodal models and the entity executing the information processing method can be the same or different. For example, regarding the case where the entities executing the training method are different, please refer to the above. Figure 13 The execution entity for the multimodal model training method is the training device 1320. The execution entity for the information processing method is the execution device 1310. In cases where the execution entities are the same, the execution entity for both the multimodal model training method and the information processing method can be, for example, [other entities]. Figure 13 The training equipment 1320, or all of them are Figure 13 The execution device is 1330. Furthermore, the execution subject can also be other devices, which are not limited here. Correspondingly, when the training device 1320 executes the multimodal model training method, the multimodal model training apparatus can be provided in the training device 1320. When the execution device 1310 executes the multimodal model training method, the multimodal model training apparatus can be provided in the execution device 1310. When other devices execute the multimodal model training method, the multimodal model training apparatus can be provided in other devices. When the execution device 1310 executes the information processing method, the information processing apparatus can be provided in the execution device 1310. When the training device 1220 executes the information processing method, the information processing apparatus can be provided in the training device 1320. When other devices execute the information processing method, the information processing apparatus can be provided in other devices.

[0403] Based on the same concept as the aforementioned embodiment of a multimodal model training method, this application also provides a multimodal model training apparatus 1400. This multimodal model training apparatus 1400 can be deployed on any computing-capable device, equipment, platform, or device cluster to implement the multimodal model training method provided in this application embodiment. The multimodal model training apparatus 1400 includes units or modules for implementing various operations in the multimodal model training method.

[0404] Figure 14 A block diagram of a training apparatus for a multimodal model according to an embodiment of this application is shown schematically.

[0405] like Figure 14 As shown, the training device 1400 for the multimodal model may include a first obtained mode 1410, a first determined module 1420, a second obtained module 1430, and a training module 1440.

[0406] The first acquisition module 1410 is used to input the first multimodal information into multiple multimodal models respectively to obtain the first multimodal features corresponding to each of the multiple multimodal models. The first multimodal information includes at least one first modal information corresponding to each of the multiple modalities. The multiple multimodal models include a student model and at least one teacher model.

[0407] The first determining module 1420 is used to determine a first correlation set corresponding to the multimodal model based on the first multimodal features corresponding to the multimodal model. The first correlation set includes at least one first correlation degree corresponding to each of the multiple first modal information pieces. The first correlation degree characterizes the degree of correlation between the first modal information and other first modal information. The other first modal information pieces are any one of the first multimodal information pieces other than the first modal information itself.

[0408] The second acquisition module 1430 is used to determine at least one first proximity degree corresponding to each of the multiple first modal information based on the first association degree set corresponding to each of the multiple multimodal models. The first proximity degree characterizes the degree of proximity between the student model and the teacher model.

[0409] Training module 1440 is used to train a student model based on at least one first proximity corresponding to each of the multiple first modal information.

[0410] The multimodal model training apparatus 1400 according to the embodiments of this application can correspond to the execution of the multimodal model training method described in the embodiments of this application, and the above and other operations and / or functions of each module in the multimodal model training apparatus 1400 are respectively the corresponding processes of the multimodal model training method of the embodiments of this application, which will not be described again here for the sake of brevity.

[0411] Based on the same concept as the aforementioned embodiment of the information processing method, this application also provides an information processing apparatus 1500. This apparatus 1500 can be deployed on any computing-capable device, equipment, platform, or device cluster to execute and implement the information processing method provided in this application. The information processing apparatus 1500 includes units or modules for implementing various operations within the information processing method.

[0412] Figure 15 A block diagram of an information processing apparatus according to an embodiment of this application is shown schematically.

[0413] like Figure 15 As shown, the information processing device 1500 may include a third acquisition module 1510 and a fourth acquisition module 1520.

[0414] The third acquisition module 1510 is used to input the third modality information into the trained student model to obtain the third modality features.

[0415] The fourth acquisition module 1520 is used to obtain processing information based on the third modality features.

[0416] According to embodiments of this application, the trained student model can be trained using a training device for a multimodal model according to embodiments of this application.

[0417] The information processing apparatus 1500 according to the embodiments of this application can be used to execute the information processing method described in the embodiments of this application. The above and other operations and / or functions of each module in the information processing apparatus 1500 are respectively the corresponding flow of the information processing method of the embodiments of this application. For the sake of brevity, they will not be described again here.

[0418] This application also provides a computing device, including at least one processor, a memory, and a communication interface, wherein the processor is used to execute the training method and information processing method of the multimodal model of this application.

[0419] Figure 16 A block diagram of a computing device according to an embodiment of this application is shown as an example.

[0420] like Figure 16 As shown, the computing device 1600 includes at least one processor 1601, a memory 1602, and a communication interface 1603. The processor 1601, memory 1602, and communication interface 1603 are communicatively connected, which can be achieved via a wired (e.g., bus) or wireless connection. The communication interface 1603 is used to send and / or receive data from other devices. The memory 1602 stores computer instructions, which the processor 1601 executes to perform the multimodal model training method and information processing method described in the aforementioned method embodiments.

[0421] It should be understood that, in the embodiments of this application, the processor 1601 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0422] The memory 1602 may include read-only memory and random access memory, and provides instructions and data to the processor 1601. The memory 1602 may also include non-volatile random access memory.

[0423] The memory 1602 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0424] It should be understood that the computing device 1600 according to the embodiments of this application can execute the training method and information processing method of the multimodal model implemented in the embodiments of this application. For a detailed description of the implementation of the above methods, please refer to the above text. For the sake of brevity, it will not be repeated here.

[0425] Embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer instructions are executed by a processor, the aforementioned method is implemented.

[0426] An embodiment of this application provides a chip including at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the method mentioned above.

[0427] Embodiments of this application provide a computer program or computer program product that includes instructions that, when executed, cause a computer to perform the methods mentioned above.

[0428] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0429] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, processor-executed software modules, or a combination of both. The software modules can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0430] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for training a multi-modal model, comprising: inputting first multi-modal information into a plurality of multi-modal models respectively to obtain first multi-modal features corresponding to the plurality of multi-modal models respectively, wherein the first multi-modal information comprises at least one first modal information corresponding to a plurality of modalities respectively, and the plurality of multi-modal models comprises a student model and at least one teacher model; determining, for the multi-modal model, a first correlation degree set corresponding to the multi-modal model according to the first multi-modal features corresponding to the multi-modal model, wherein the first correlation degree set comprises at least one first correlation degree corresponding to the plurality of first modal information respectively, and the first correlation degree represents a correlation degree of a correlation relationship between the first modal information and other first modal information, the other first modal information being any one of the first multi-modal information except the first modal information; obtaining at least one first closeness corresponding to the plurality of first modal information respectively according to the first correlation degree set corresponding to the plurality of multi-modal models respectively, wherein the first closeness represents a closeness between the student model and the teacher model; and training the student model according to the at least one first closeness corresponding to the plurality of first modal information respectively.

2. The method of claim 1, wherein, The determining, for the multi-modal model, the first correlation degree set corresponding to the multi-modal model according to the first multi-modal features corresponding to the multi-modal model comprises: obtaining the first correlation degree set corresponding to the multi-modal model according to a first graph structure and the first multi-modal features corresponding to the multi-modal model; wherein the first graph structure is constructed according to the first multi-modal information, the first graph structure comprises a plurality of first nodes and at least one first edge, the first node represents the first modal information, and the first edge is used to connect two first nodes having a direct correlation relationship, the modalities of the two first modal information having the direct correlation relationship being different.

3. The method of claim 1 or 2, wherein, The first correlation degree set comprises a first correlation degree subset corresponding to the plurality of first modal information respectively, and the first correlation degree subset comprises at least one of a first direct correlation degree subset and a first indirect correlation degree subset; wherein the first direct correlation degree subset comprises at least one first direct correlation degree, and the first direct correlation degree represents a correlation degree of a direct correlation relationship between the first modal information corresponding to the multi-modal model and other first modal information; wherein the first indirect correlation degree subset comprises at least one first indirect correlation degree, and the first indirect correlation degree represents a correlation degree of an indirect correlation relationship between the first modal information corresponding to the multi-modal model and other first modal information.

4. The method of claim 3, wherein, The first multi-modal features corresponding to the multi-modal model comprise first modal features corresponding to the plurality of first modal information respectively, and in a case where the first correlation degree subset comprises the first direct correlation degree subset, the obtaining the first correlation degree set corresponding to the multi-modal model according to the first graph structure and the first multi-modal features corresponding to the multi-modal model comprises: determining at least one first direct node pair from the first graph structure, wherein the first direct node pair comprises two first nodes connected by the first edge; determining, for the first direct node pair, a similarity between first modal features respectively corresponding to the two first nodes in the first direct node pair, to obtain a first similarity between the two first nodes in the first direct node pair, wherein the first modal features correspond to the multi-modal model; and obtaining, according to the first similarity between the two first nodes in each of the at least one first direct node pair, a first direct correlation degree subset respectively corresponding to the plurality of first modal information.

5. The method of claim 4, wherein, The obtaining, according to the first similarity between the two first nodes in each of the at least one first direct node pair, a first direct correlation degree subset respectively corresponding to the plurality of first modal information, comprises: for the first node in the at least one first direct node pair, determining at least one target first direct node pair from the at least one first direct node pair, wherein the target first direct node pair comprises the first node; and obtaining, according to the first similarity between the two first nodes in each of the at least one target first direct node pair, a first direct correlation degree subset respectively corresponding to the first modal information represented by the first node.

6. The method of claim 5, wherein, The obtaining, according to the first similarity between the two first nodes in each of the at least one target first direct node pair, a first direct correlation degree subset respectively corresponding to the first modal information represented by the first node, comprises: obtaining, according to the first similarity between the two first nodes in each of the at least one target first direct node pair, a first comprehensive similarity corresponding to the first node; for the target first direct node pair, obtaining, according to the first comprehensive similarity corresponding to the first node and the first similarity between the two first nodes in the target first direct node pair, a second similarity between the two first nodes in the target first direct node pair; obtaining, according to the second similarity between the two first nodes in the target first direct node pair, a first direct correlation degree between the two first nodes in the target first direct node pair; and obtaining, according to the first direct correlation degree between the two first nodes in each of the at least one target first direct node pair, a first direct correlation degree subset respectively corresponding to the first modal information represented by the first node.

7. The method according to any one of claims 3 to 6, wherein, The first multi-modal features corresponding to the multi-modal model comprise first modal features respectively corresponding to the plurality of first modal information, and in a case where the first correlation degree set comprises the first indirect correlation degree subset, the obtaining, according to the first graph structure and the first multi-modal features corresponding to the multi-modal model, a first correlation degree set corresponding to the multi-modal model, comprises: determining at least one first indirect node pair from the first graph structure, wherein the first indirect node pair comprises two first nodes not connected by the first edge; obtaining, according to first association information corresponding to the first inter-connected node pair, a third similarity between the two first nodes in the first inter-connected node pair; and obtaining, according to the third similarity between the two first nodes in each of the at least one first inter-connected node pair, a first indirect association degree subset corresponding to each of the plurality of first modal information; The first association information corresponding to the first inter-connected node pair includes one of the following: a first direct association degree subset corresponding to each of the two first nodes in the first inter-connected node pair; a first modal feature corresponding to each of the at least one first association node and a first modal feature corresponding to each of the two first nodes in the first inter-connected node pair, the first association node representing a first node in the first graph structure that functions as a context association function, the context association function representing a function of associating the two first nodes in the first inter-connected node pair.

8. The method of claim 7, wherein, The obtaining, according to the third similarity between the two first nodes in each of the at least one first inter-connected node pair, a first indirect association degree subset corresponding to each of the plurality of first modal information, includes: for a first node in the at least one first inter-connected node pair, determining at least one target first inter-connected node pair from the at least one first inter-connected node pair, wherein the target first inter-connected node pair includes the first node; and obtaining, according to the third similarity between the two first nodes in each of the at least one target first inter-connected node pair, a first indirect association degree subset corresponding to the first modal information represented by the first node.

9. The method of claim 8, wherein, The obtaining, according to the third similarity between the two first nodes in each of the at least one target first inter-connected node pair, a first indirect association degree subset corresponding to the first modal information represented by the first node, includes: obtaining, according to the third similarity between the two first nodes in each of the at least one target first inter-connected node pair, a second comprehensive similarity corresponding to the first node; for the target first inter-connected node pair, obtaining, according to the second comprehensive similarity corresponding to the first node and the third similarity between the two first nodes in the target first inter-connected node pair, a fourth similarity between the two first nodes in the target first inter-connected node pair; obtaining, according to the fourth similarity between the two first nodes in the target first inter-connected node pair, a first indirect association degree between the two first nodes in the target first inter-connected node pair; and obtaining, according to the first indirect association degree between the two first nodes in each of the at least one target first direct connection node pair, a first indirect association degree subset corresponding to the first modal information represented by the first node.

10. The method according to any one of claims 1 to 9, wherein, The first association degree set includes a first association degree subset corresponding to each of the plurality of first modal information; The obtaining, according to the first association degree set corresponding to each of the plurality of multi-modal models, at least one first proximity degree corresponding to each of the plurality of first modal information, includes: for the first modal information, for the teacher model, determine a first closeness degree corresponding to the first modality information, the student model and the teacher model according to the first closeness degree corresponding to the first modality information, the student model and the teacher model.

11. The method of claim 10, wherein, In a case where the first closeness degree corresponding to the first modality information, the student model and the teacher model is determined according to the first closeness degree corresponding to the first modality information, the student model and the teacher model, the method further comprises: determining a first closeness degree corresponding to the first modality information, the student model and the teacher model according to the first closeness degree corresponding to the first modality information, the student model and the teacher model. determining a first closeness degree corresponding to the first modality information, the student model and the teacher model according to the first closeness degree corresponding to the first modality information, the student model and the teacher model. determining a first closeness degree corresponding to the first modality information, the student model and the teacher model according to the first closeness degree corresponding to the first modality information, the student model and the teacher model.

12. The method of any one of claims 1-11, wherein, determining a first closeness degree corresponding to the first modality information, the student model and the teacher model according to the first closeness degree corresponding to the first modality information, the student model and the teacher model. training the student model according to the at least one first closeness degree corresponding to each of the plurality of first modality information and at least one adjustable coefficient. wherein the at least one adjustable coefficient corresponding to each of the first modality information comprises an adjustable coefficient corresponding to the first modality information, the student model and the teacher model, and the adjustable coefficient corresponding to the first modality information, the student model and the teacher model represents a contribution degree of the teacher model in all the teacher models in terms of a contribution degree of processing the first modality information. wherein the adjustable coefficient corresponding to the first modality information is determined according to a comprehensive distance corresponding to the first modality information and a distance, the comprehensive distance corresponding to the first modality information is determined according to at least one distance corresponding to the first modality information, and the at least one distance corresponding to the first modality information is determined according to second association information corresponding to the first modality information. wherein the at least one distance corresponding to the first modality information is determined according to the second association information corresponding to the first modality information, and the second association information corresponding to the first modality information comprises one of: The at least one distance corresponding to the first modality information is obtained according to a first correlation degree subset corresponding to the first modality information and the student model, and a first correlation degree subset corresponding to the first modality information and each of the at least one teacher model. The at least one distance corresponding to the first modality information is obtained according to a first modality feature corresponding to the first modality information and the student model, and a first modality feature corresponding to the first modality information and each of the at least one teacher model.

13. The method of claim 12, wherein, For the teacher model, the at least one distance corresponding to the first modality information is obtained according to a first correlation degree subset corresponding to the first modality information and the student model, and a first correlation degree subset corresponding to the first modality information and each of the at least one teacher model, including: The distance corresponding to the first modality information, the student model and the teacher model is obtained according to a distance between a first correlation degree subset corresponding to the first modality information and the student model, and a first correlation degree subset corresponding to the first modality information and the teacher model.

14. The method of claim 12 or 13, wherein, For the teacher model, the at least one distance corresponding to the first modality information is obtained according to a first modality feature corresponding to the first modality information and the student model, and a first modality feature corresponding to the first modality information and each of the at least one teacher model, including: The distance corresponding to the first modality information, the student model and the teacher model is obtained according to a distance between a first modality feature corresponding to the first modality information and the student model, and a first modality feature corresponding to the first modality information and the teacher model.

15. The method of any one of claims 12-14, wherein, The student model is trained according to at least one first closeness corresponding to each of the plurality of first modality information and at least one adjustable coefficient, including: For the first modality information, For the teacher model, The first closeness corresponding to the first modality information, the student model and the teacher model is multiplied by the adjustable coefficient to obtain a weighted closeness corresponding to the first modality information, the student model and the teacher model; and The student model is trained according to at least one weighted closeness corresponding to each of the plurality of first modality information.

16. The method of any one of claims 1-11, wherein, The student model is trained according to at least one first closeness corresponding to each of the plurality of first modality information, including: The student model is trained according to a first correlation degree subset corresponding to each of the plurality of first modality information and at least one first closeness.

17. The method of any one of claims 1-11, wherein, In a case where the first multi-modal information is multi-modal information that does not match the current task, the student model is trained according to at least one first closeness corresponding to each of the plurality of first modality information, including: inputting second multi-modal information into the student model to obtain second multi-modal features corresponding to the student model, wherein the second multi-modal information comprises at least one second modal information corresponding to each of the plurality of modalities, and the second multi-modal features comprise second modal features corresponding to each of the plurality of second modal information; obtaining a second correlation degree set corresponding to the student model according to the second multi-modal features corresponding to the student model, wherein the second correlation degree set comprises a second correlation degree subset corresponding to each of the plurality of second modal information, and the second correlation degree subset corresponding to the second modal information comprises at least one second correlation degree corresponding to the second modal information, wherein the second correlation degree corresponding to the second modal information represents a correlation degree between the second modal information and other second modal information, and the other second modal information is any one of the second modal information other than the second modal information in the second multi-modal information; and training the student model according to at least one first proximity degree corresponding to each of the plurality of first modal information, and the second correlation degree subset corresponding to each of the plurality of second modal information; wherein the second multi-modal information is multi-modal information matched with the current task.

18. The method of claim 17, wherein, The training of the student model according to at least one first proximity degree corresponding to each of the plurality of first modal information, and the second correlation degree subset corresponding to each of the plurality of second modal information, comprises: training the student model according to at least one first proximity degree corresponding to each of the plurality of first modal information, at least one adjustable coefficient, and the second correlation degree subset corresponding to each of the plurality of second modal information; wherein the at least one adjustable coefficient corresponding to the first modal information comprises an adjustable coefficient corresponding to the first modal information, the student model, and the teacher model, and the adjustable coefficient corresponding to the first modal information, the student model, and the teacher model represents a contribution degree of the teacher model in all the teacher models in terms of a contribution degree of processing the first modal information; wherein the adjustable coefficient corresponding to the first modal information is obtained according to a comprehensive distance corresponding to the first modal information and a distance, the comprehensive distance corresponding to the first modal information is obtained according to at least one distance corresponding to the first modal information, and the at least one distance corresponding to the first modal information is obtained according to second correlation information corresponding to the first modal information; wherein the at least one distance corresponding to the first modal information is obtained according to the second correlation information corresponding to the first modal information, comprising one of the following: the at least one distance corresponding to the first modal information is obtained according to a first correlation degree subset corresponding to the first modal information and the student model, and a first correlation degree subset corresponding to the first modal information and the at least one teacher model; The at least one distance corresponding to the first modality information is obtained according to the first modality feature corresponding to the first modality information and the student model, and the first modality feature corresponding to the first modality information and each of the at least one teacher model.

19. The method of claim 17 or 18, wherein, The second correlation degree set corresponding to the student model is obtained according to the second multi-modality feature corresponding to the student model, including: The second correlation degree set corresponding to the student model is obtained according to the second multi-modality feature corresponding to the student model and the second graph structure; The second graph structure is constructed according to the second multi-modality information, and the second graph structure includes a plurality of second nodes and at least one second edge, the second node represents the second modality information, and the second edge is used to connect two second nodes having a direct correlation relationship, and the two second modality information having the direct correlation relationship are different in modality.

20. The method of any one of claims 1-19, wherein, In the case that the at least one teacher model includes one, the teacher model is one of the following: a multi-modality pre-training model and an old multi-modality model; In the case that the at least one teacher model includes two, the two teacher models include the multi-modality pre-training model and the old multi-modality model. The old multi-modality model is a multi-modality model corresponding to a previous task of the current task.

21. An information processing method, comprising: inputting third modality information into a trained student model to obtain third modality features; and obtaining processing information according to the third modality features; The trained student model is trained by the method according to any one of claims 1-20.

22. A multi-modality model training device, comprising: a first obtaining module configured to input first multi-modality information into a plurality of multi-modality models respectively to obtain first multi-modality features corresponding to the plurality of multi-modality models respectively, wherein the first multi-modality information includes at least one first modality information corresponding to a plurality of modalities, and the plurality of multi-modality models include a student model and at least one teacher model; a first determining module configured to determine, for the multi-modality model, a first correlation degree set corresponding to the multi-modality model according to the first multi-modality feature corresponding to the multi-modality model, wherein the first correlation degree set includes at least one first correlation degree corresponding to each of the plurality of first modality information, the first correlation degree represents a correlation degree of a correlation relationship between the first modality information and any one of the other first modality information in the first multi-modality information; a second obtaining module configured to obtain at least one first proximity degree corresponding to each of the plurality of first modality information according to the first correlation degree set corresponding to each of the plurality of multi-modality models, wherein the first proximity degree represents a proximity degree between the student model and the teacher model; and a training module configured to train the student model according to the at least one first proximity degree corresponding to each of the plurality of first modality information.

23. An information processing device, comprising: a third obtaining module, configured to input the third modality information into the trained student model to obtain third modality features; and a fourth obtaining module, configured to obtain processing information according to the third modality features. The trained student model is trained by using the device according to claim 22.

24. A computing device, comprising a memory and a processor; The memory stores instructions, when the instructions are executed by the processor, the method according to any one of claims 1-21 is implemented.

25. A computer readable storage medium, having stored thereon computer program instructions, when the computer instructions are executed by a processor, the method according to any one of claims 1-21 is implemented.

26. A computer program product, comprising a computer program, when the computer program is executed by a processor, the method according to any one of claims 1-21 is implemented.

27. A chip, comprising at least one processor and a communication interface, the processor is configured to execute the method according to any one of claims 1-21.

Images