Model training method and device, equipment, storage medium and program product

Abstract

The application provides a model training method and device, equipment, a storage medium and a program product. The method comprises: clustering text samples and image samples in a multi-modal data set to obtain a data set of sample categories and category centers; extracting features of a text-image sample pair in the data set by using a preset multi-modal model to obtain fused features of the text-image sample pair, and determining a multi-modal loss based on the fused features; updating a historical category center set of the model according to the category centers of the sample categories, determining a clustering loss of the text-image sample pair based on the updated category center set; and converging the preset multi-modal model according to the multi-modal loss and the clustering loss to obtain a trained multi-modal model. In this embodiment, when the multi-modal model learns new knowledge, the category centers in the updated category center set can be regarded as anchor points, which prevents feature drift in the continuous learning process and improves the performance and robustness of the multi-modal model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, device, storage medium, and program product. Background Technology

[0002] In the current field of machine learning and artificial intelligence, multimodal models have become an important research direction. Multimodal models can process and fuse data from different modalities, such as images and text, thus performing well in many application scenarios.

[0003] In practical applications, multimodal models can be continuously trained to adapt to constantly changing network data, thereby improving their ability to process new data. However, due to the diversity and complexity of the model's input data, multimodal models are prone to degradation and knowledge forgetting during continuous training, affecting model performance and robustness. Summary of the Invention

[0004] This invention provides a model training method, apparatus, device, storage medium, and program product to solve the problem in related technologies where multimodal models are prone to model degradation and knowledge forgetting during continuous training, affecting model performance and robustness.

[0005] In a first aspect, embodiments of this application provide a model training method, including:

[0006] Obtain a multimodal dataset, which includes at least one text sample and an image sample corresponding to the text sample;

[0007] Cluster the text samples and image samples to obtain a dataset of at least one sample category and the category centers of the sample categories. The dataset includes at least one pair of text and image samples.

[0008] A pre-defined multimodal model is used to fuse features of the dataset to obtain fused features of image-text sample pairs, and at least one multimodal loss is determined based on the fused features.

[0009] Based on the category centers of the sample categories, the historical category center set of the preset multimodal model is updated, and based on the updated category center set, the clustering loss of the image and text sample pairs is determined.

[0010] Based on the multimodal loss and clustering loss, the preset multimodal model is converged to obtain the trained multimodal model.

[0011] Secondly, embodiments of this application provide a model training apparatus, including:

[0012] The acquisition unit is used to acquire a multimodal data set, which includes at least one text sample and an image sample corresponding to the text sample.

[0013] Clustering unit, used to cluster text samples and image samples to obtain a data set of at least one sample category and the category center of the sample category, the data set including at least one image-text sample pair;

[0014] The extraction unit is used to perform feature fusion on the dataset using a preset multimodal model to obtain the fused features of the image-text sample pairs, and to determine at least one multimodal loss based on the fused features;

[0015] The update unit is used to update the historical set of category centers of the preset multimodal model according to the category centers of the sample categories, and to determine the clustering loss of the image and text sample pairs based on the updated set of category centers.

[0016] The training unit is used to converge the preset multimodal model based on the multimodal loss and clustering loss to obtain the trained multimodal model.

[0017] In one embodiment, the update unit is specifically used to combine the category centers in the updated category center set to obtain the classification weights of the classification network in the preset multimodal model; based on the classification weights and the fused features, the classification network is used to predict the category of the image-text sample pair to obtain the predicted category; and the clustering loss of the image-text sample pair is determined according to the predicted category and the sample category.

[0018] In one embodiment, the updating unit is further configured to obtain category information corresponding to the category centers in the historical category center set; and based on the category information, add the category centers to the historical category center set to obtain the updated category center set.

[0019] In one embodiment, the historical category center set includes historical category centers of at least one historical category. The updating unit is further configured to: when category information indicates that the sample category does not contain a historical category, add the category center of the sample category to the historical category center set to obtain an updated category center set; when category information indicates that the sample category contains a historical category, filter out category centers other than historical categories from the sample category, and add the filtered category centers to the historical category center set to obtain an updated category center set.

[0020] In one embodiment, the historical category center set includes historical category centers of at least one historical category. The updating unit is further configured to: when category information indicates that a sample category contains a historical category, filter out the target historical category contained in the sample category from the historical categories; filter out the category center corresponding to the target historical category from the category centers of the sample category to obtain the current category center; replace the historical category center of the target historical category with the current category center in the historical category center set to obtain a replaced category center set; and add the category centers of the sample category other than the current category center to the replaced category center set to obtain an updated category center set.

[0021] In one embodiment, the clustering unit is specifically used to cluster text samples and image samples to obtain an initial data set of at least one initial sample category; determine the category relationship between the initial sample categories based on the initial data set; and merge the initial sample categories according to the category relationship and the initial data set to obtain a data set of at least one sample category and a category center of the sample category.

[0022] In one embodiment, the clustering unit is further configured to: extract at least one text word from the text sample, and cluster the text sample based on the text word to obtain a set of text samples of at least one initial sample category; filter out the current image sample corresponding to the text sample in the text sample set from the image sample; and add the current image sample to the text sample set to obtain an initial data set of the initial sample category.

[0023] In one embodiment, the clustering unit is further configured to: merge the initial sample categories according to the category relationship to obtain at least one sample category; merge the initial data set based on the sample categories, and construct at least one image-text sample pair in the merged data to obtain a data set of sample categories; and calculate the category center of the sample categories based on the data set of sample categories.

[0024] In one embodiment, the extraction unit is specifically used to: determine the image-text matching loss of the image-text sample pair based on the fused features; determine the text masking loss of the image-text sample pair based on the fused features and a preset text mask; and use at least one of the image-text matching loss and the text masking loss as a multimodal loss.

[0025] In one embodiment, the extraction unit is further configured to: add a preset image mask to the image sample in the image-text sample pair to obtain a masked image sample; based on the fused features, use a multimodal coding network in a preset multimodal model to restore the masked image sample to obtain a restored image sample; determine the image masking loss of the image-text sample pair according to the restored image sample and the image sample; and use at least one of the image-text matching loss, text masking loss and image masking loss as the multimodal loss.

[0026] In one embodiment, the extraction unit is further specifically used to: predict the image-text matching information of the image-text sample pair using a multimodal coding network in a preset multimodal model based on the fused features, and obtain the predicted image-text matching information; identify the image-text matching relationship of the image-text sample pair in the multimodal dataset, and obtain the image-text matching label; and determine the image-text matching loss of the image-text sample pair based on the predicted image-text matching information and the image-text matching label.

[0027] In one embodiment, the extraction unit is further specifically used to add a preset text mask to the text sample in the image-text sample pair to obtain a masked text sample; based on the fused features, use a multimodal coding network in a preset multimodal model to restore the text of the masked text sample to obtain a restored text sample; and determine the text mask loss of the image-text sample pair based on the restored text sample and the text sample.

[0028] In one embodiment, the extraction unit is further specifically used to extract features from the dataset using a preset multimodal model to obtain text features of text samples and image features of image samples; and to fuse the text features of text samples in the image-text sample pair with the image features of the corresponding image samples to obtain the fused features of the image-text sample pair.

[0029] In one embodiment, the model training device further includes a retrieval unit, which is used to perform image-text retrieval on the dataset based on the text features of text samples and the image features of image samples after feature fusion using a preset multimodal model, to obtain an image-text retrieval loss. The training unit is used to converge the preset multimodal model according to the multimodal loss, clustering loss, and image-text retrieval loss to obtain a trained multimodal model.

[0030] In one embodiment, the retrieval unit is specifically configured to: calculate the similarity between the text features of a text sample and the image features of at least one image sample to obtain a first similarity set of text samples; calculate the similarity between the image features of an image sample and the text features of at least one text sample to obtain a second similarity set of image samples; determine a first retrieval loss of text samples based on the first similarity set, and determine a second retrieval loss of image samples based on the second similarity set; and fuse the first retrieval loss and the second retrieval loss to obtain an image-text retrieval loss.

[0031] In one embodiment, the training unit is specifically used to: fuse multimodal loss, clustering loss, and image / text retrieval loss to obtain the target loss corresponding to the multimodal dataset; obtain incremental learning parameters corresponding to at least one incremental learning method; and incrementally update the preset multimodal model according to the incremental learning parameters and the target loss to obtain the trained multimodal model.

[0032] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to perform the steps in the above-described model training method.

[0033] Fourthly, embodiments of this application provide a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute the steps in the above-described model training method.

[0034] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps in the above-described model training method.

[0035] In one of the solutions provided by the aforementioned model training methods, devices, equipment, storage media, and program products, a multimodal dataset is acquired, which includes at least one text sample and a corresponding image sample; the text samples and image samples are clustered to obtain a dataset of at least one sample category and category centers of the sample categories, the dataset including at least one image-text sample pair; a preset multimodal model is used to perform feature fusion on the dataset to obtain fused features of the image-text sample pair, and at least one multimodal loss is determined based on the fused features; the historical category center set of the preset multimodal model is updated according to the category centers of the sample categories, and the clustering loss of the image-text sample pair is determined based on the updated category center set; the preset multimodal model is converged according to the multimodal loss and the clustering loss to obtain the trained multimodal model. In this embodiment, during the training of the multimodal model, the category centers of the image and text samples are first obtained by clustering, and then the historical category center set of the model is updated based on this to obtain the clustering loss. On the basis of the multimodal loss function, a multimodal superclass clustering loss function can be introduced, so that the image and text multimodal features belonging to the same large class can always be clustered in the corresponding category center during the continuous learning process of the model. This improves the ability of the multimodal model to learn new knowledge from complex and diverse new data and improves the training effect of the multimodal model. Moreover, when the multimodal model learns new knowledge, the category centers in the updated category center set can be regarded as anchor points to prevent feature drift during the continuous learning process. This allows the multimodal model to minimize the forgetting of old categories while continuously receiving new knowledge, thereby improving the performance and robustness of the multimodal model. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of an application scenario of the model training method in one embodiment of the present invention;

[0038] Figure 2 This is a schematic flowchart of a model training method in one embodiment of the present invention;

[0039] Figure 3 yes Figure 2 A schematic diagram of the implementation process of step S20;

[0040] Figure 4 This is a schematic diagram illustrating a process for determining the category relationship of an initial data set in one embodiment of the present invention;

[0041] Figure 5 , Figure 6 This is a schematic diagram illustrating a process for determining sample categories and their data sets in one embodiment of the present invention;

[0042] Figure 7 This is a schematic diagram of a pre-defined multimodal model in one embodiment of the present invention;

[0043] Figure 8 yes Figure 2 A schematic diagram of the implementation process of step S30;

[0044] Figure 9 This is another structural schematic diagram of a preset multimodal model in one embodiment of the present invention;

[0045] Figure 10 yes Figure 2 A schematic diagram of the implementation process of step S40;

[0046] Figure 11 This is another flowchart illustrating the model training method in one embodiment of the present invention;

[0047] Figure 12 This is a schematic diagram of an update process for a historical category center set in one embodiment of the present invention;

[0048] Figure 13 This is a schematic diagram of a superclass tree in one embodiment of the present invention;

[0049] Figure 14 This is another structural schematic diagram of a preset multimodal model in one embodiment of the present invention;

[0050] Figure 15 This is another flowchart illustrating the model training method in one embodiment of the present invention;

[0051] Figure 16 This is a schematic diagram of a data processing process in a content management system using a multimodal model, according to an embodiment of the present invention.

[0052] Figure 17 This is a schematic diagram of a data processing process in which a multimodal model is applied to a search system according to an embodiment of the present invention;

[0053] Figure 18 This is a schematic diagram of a data processing process in which a multimodal model is applied to an intelligent customer service system according to an embodiment of the present invention;

[0054] Figure 19 This is a schematic diagram of a model training device in one embodiment of the present invention;

[0055] Figure 20 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. It should also be understood that, as used in this specification and the appended claims, the term "and / or" refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0058] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0059] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0060] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0061] To facilitate understanding of the technical solutions of this application, some terms and concepts involved in the embodiments of this application will be explained below:

[0062] 1. A multimodal model is a machine learning model capable of processing and understanding multiple types of data, such as text, images, and audio. The core of a multimodal model lies in its ability to process multiple data types simultaneously. Compared to ordinary models that are trained and predicted only on a single type of data, multimodal models, by integrating information from different modalities, can more comprehensively understand and generate complex multimodal content, thus improving the model's overall capabilities.

[0063] 2. Continual Training is a neural network model training method that refers to continuing to train the model with new data during actual application of the model after it has been pre-trained and possibly post-trained, so as to continuously update and improve the model's performance.

[0064] 3. Super Clustering is an advanced clustering technique used to extract deep relationships between multimodal data and group the data (clustering) based on these relationships. This allows multimodal data to be grouped into higher-level sets based on the grouping results, thereby better capturing the complex structure and relationships of the data.

[0065] 4. Multimodal Super Clustering Loss (MSC) is a loss function used to measure and optimize clustering results in multimodal clustering. It represents the clustering loss while preserving multimodal features during continuous learning. The goal of MSC is to ensure that features from different modalities are compatible in the shared space and can reflect the true distribution of the data.

[0066] Multimodal models can process and fuse data from different modalities, such as images and text, thus performing well in many application scenarios. In practical applications, to adapt to continuously changing network data, multimodal models can be continuously trained to improve their ability to process new data.

[0067] However, due to the diversity and complexity of the model input data, multimodal models often face several challenges during continuous training. For example, when continuously training multimodal models, the methods used in related technologies to process the model input data are unable to handle the complex structure of multimodal data, struggle to capture the diverse and complex data features, and fail to effectively converge the model, leading to model degradation. In traditional continuous model training methods, although the model can learn new knowledge on new data, the model trained on new data will also forget the previously learned knowledge, causing model performance degradation and reducing model performance and robustness.

[0068] To address the aforementioned problems, this application provides a model training method, apparatus, device, storage medium, and program product. The method involves acquiring a multimodal dataset, including at least one text sample and corresponding image samples; clustering the text and image samples to obtain a dataset of at least one sample category and category centers for each category, with the dataset including at least one image-text sample pair; fusing features of the dataset using a pre-defined multimodal model to obtain fused features for the image-text sample pair, and determining at least one multimodal loss based on these fused features; updating the historical category center set of the pre-defined multimodal model according to the category centers of the sample categories, and determining the clustering loss for the image-text sample pair based on the updated category center set; and converging the pre-defined multimodal model based on the multimodal loss and the clustering loss to obtain the trained multimodal model. This method effectively solves the problems of model degradation and knowledge forgetting during continuous training of multimodal models. In the training process of multimodal models, on the one hand, by first clustering to obtain the category centers of image and text samples, and then updating the model's historical category center set based on these centers to obtain the clustering loss, a multimodal superclass clustering loss function can be introduced on the basis of the multimodal loss function. This ensures that image and text multimodal features belonging to the same major category can always cluster in the corresponding category centers during the model's continuous learning process, improving the multimodal model's ability to learn new knowledge from complex and diverse new data and enhancing the training effect of the multimodal model. On the other hand, when the multimodal model learns new knowledge, the category centers in the updated category center set can be regarded as anchor points to prevent feature drift during continuous learning. This allows the multimodal model to minimize the forgetting of old categories while continuously receiving new knowledge, improving the performance and robustness of the multimodal model.

[0069] The model training method provided in this embodiment of the invention can be applied to, for example... Figure 1 In the application scenario shown. See also Figure 1 As shown, terminal device 101 communicates with server 102 via a network. This network can be, but is not limited to, a local area network (LAN), a metropolitan area network (MAN), or a wide area network (WAN).

[0070] Server 102 can be a backend server of a preset platform. Server 102 can call the multimodal model provided in this embodiment to provide related functions to the preset platform, such as intelligent voice assistant, image recognition, natural language processing, etc. The preset platform can be, but is not limited to, social media platforms, audio sharing platforms, video sharing platforms, news media platforms, and knowledge sharing platforms. Relevant users can access the preset platform through terminal device 101 to perform data searches, knowledge Q&A, and related operations on the content of the preset platform, such as publishing, searching, browsing, and sharing content.

[0071] Users log in to the preset platform using terminal device 101 to perform relevant data operations, such as enabling functions like intelligent voice assistants, image recognition, and natural language processing. The preset platform receives or generates new data. During this process, server 102 calls the trained multimodal model (i.e., the preset multimodal model) to perform relevant data processing tasks based on the received or generated new data. In addition, server 102 also uses new multimodal data (such as text, images, and voice data) from the preset platform to continuously train the multimodal model, ensuring that the multimodal model can adapt to continuously changing new data.

[0072] During the continuous training of the multimodal model, server 102 can acquire new data (i.e., new multimodal data) to form a multimodal dataset, which includes at least one text sample and its corresponding image sample. The server then clusters the text and image samples in the multimodal dataset to obtain a dataset of at least one sample category and its category centers. The dataset includes at least one image-text sample pair. A pre-defined multimodal model (i.e., the untrained multimodal model or a previously trained multimodal model) is used to fuse features in the dataset to obtain fused features of the image-text sample pair. Based on these fused features, at least one multimodal loss is determined. The historical category center set of the pre-defined multimodal model is updated according to the category centers of the sample categories, and the clustering loss of the image-text sample pair is determined based on the updated category center set. The pre-defined multimodal model is trained based on the multimodal loss and the clustering loss to achieve convergence, resulting in a trained multimodal model. This process addresses the problems of model degradation and knowledge forgetting during continuous training of the multimodal model.

[0073] In this embodiment, during the multimodal model training process, the category centers of image and text samples are first obtained through clustering. Then, the historical category center set of the model is updated based on these centers to obtain the clustering loss. This allows the introduction of a multimodal superclass clustering loss function on top of the existing multimodal loss function. This ensures that multimodal features belonging to the same major category are consistently clustered within their corresponding category centers during the model's continuous learning process. This enhances the multimodal model's ability to learn new knowledge from complex and diverse new data, thereby improving the training effect. Furthermore, when the multimodal model learns new knowledge, the category centers in the updated category center set can be considered as anchor points, preventing feature drift during continuous learning. This allows the multimodal model to minimize the forgetting of old categories while continuously receiving new knowledge, improving the performance and robustness of the multimodal model.

[0074] The terminal device 101 may include, but is not limited to, smartphones (such as Android phones, iOS phones, etc.), tablet computers, portable personal computers, mobile internet devices (MIDs), smart TVs, in-vehicle devices, portable wearable devices (such as head-mounted devices, wristbands), and other smart devices. The server may be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers. In one embodiment, as... Figure 2 As shown, a model training method is provided, and the application of this method in a model training device is illustrated as an example. The model training device can be... Figure 1 The server in the system. The model training method specifically includes the following steps:

[0075] S10: Obtain a multimodal data set, which includes at least one text sample and an image sample corresponding to the text sample.

[0076] The model training device can acquire a multimodal dataset, which is new data used for training a preset multimodal model. This multimodal dataset includes at least one text sample and a corresponding image sample; that is, the multimodal dataset includes one or more sets of training data samples, each set of training data samples including one text sample and a corresponding image sample.

[0077] The image sample corresponding to the text sample can be an image sample related to the text sample, that is, an image sample that has an association with the text sample. For example, the image sample can have the same source as the text sample, such as text and image input by the same user at the same time, which can serve as the text sample and its corresponding image sample; or an image and text appearing in the same document or appearing in the same document and in close proximity can serve as the text sample and its corresponding image sample. In other embodiments, the image sample corresponding to the text sample can also be text and image that have an association after being manually annotated or annotated by an annotation model.

[0078] S20: Cluster the text samples and image samples to obtain a dataset of at least one sample category and the category center of the sample category.

[0079] After acquiring the multimodal dataset, the model training device clusters the text and image samples in the multimodal dataset to obtain a dataset of at least one sample category and the category center of that sample category.

[0080] The dataset includes at least one image-text sample pair, which can be either a positive or negative sample pair. A positive sample pair consists of a text sample and its corresponding image sample, while a negative sample pair consists of the text sample and another image sample from the dataset, which is any image sample in the dataset other than the one corresponding to the text sample. The cluster center of a sample category is determined by the cluster center of the dataset for that category, i.e., the mean (centroid) of all samples in the dataset for that category, and is used to represent the central location of the dataset for that category.

[0081] Among them, sample category refers to dividing data samples into different groups or categories based on their characteristics or attributes; different types of data correspond to different categories, which are determined according to the platform to which the data belongs.

[0082] For example, taking a social media platform as an example, the multimodal data set obtained from the platform, and its corresponding data category, that is, the sample category, can include current events, entertainment, Q&A, travel, etc., and can be further subdivided according to actual needs, which will not be elaborated here.

[0083] S30: Use a preset multimodal model to perform feature fusion on the dataset to obtain the fused features of the image and text sample pairs, and determine at least one multimodal loss based on the fused features.

[0084] After obtaining the dataset of sample categories and the category centers of the sample categories, the model training device uses a preset multimodal model to extract features from the data in the dataset to obtain the fused features of the image-text sample pairs, and determines at least one multimodal loss of the image-text sample pairs based on the fused features of the image-text sample pairs.

[0085] After obtaining the dataset of sample categories and the category centers of each category, the model training device inputs the image-text sample pairs into a pre-defined multimodal model. The model extracts features from the text samples within the image-text sample pairs to obtain text features, and extracts features from the image samples within the image-text sample pairs to obtain image features. These text features and image features are then fused to obtain the fused features of the image-text sample pairs. Based on these fused features, the model training device calculates the multimodal loss of the image-text sample pairs using a pre-defined loss function.

[0086] S40: Update the historical set of category centers of the preset multimodal model based on the category centers of the sample categories, and determine the clustering loss of the image-text sample pairs based on the updated set of category centers.

[0087] Simultaneously, the model training device acquires a set of historical category centers for a pre-defined multimodal model. This set comprises the category centers of different historical data sets used to train the model; specifically, it includes the category centers of multiple historical data sets from different categories. During the initial training of the pre-defined multimodal model, this set of historical category centers can be pre-labeled multiple category centers.

[0088] After determining the multimodal loss based on the fused features of image-text sample pairs, the model training device updates the historical set of category centers of the preset multimodal model according to the category centers of the sample categories, obtains the updated set of category centers, and determines the clustering loss of image-text sample pairs based on the updated set of category centers.

[0089] In this process, the category centers of the sample categories obtained in step S2, which are also the category centers of the data set obtained by the new data classification, can be directly added to the historical category centers of the preset multimodal model to obtain the updated category center set. Then, based on the updated category center set, the multimodal superclass clustering loss function is used to determine the clustering loss of the image and text sample pairs.

[0090] S50: Based on the multimodal loss and clustering loss, the preset multimodal model is converged to obtain the trained multimodal model.

[0091] After determining the multimodal loss and clustering loss for the image-text sample pairs, the model training device converges the preset multimodal model based on the multimodal loss and clustering loss to obtain the trained multimodal model.

[0092] The model training device can fuse multimodal loss and clustering loss to obtain the target loss corresponding to the multimodal dataset. For example, the multimodal loss and clustering loss can be directly added together, which is simple and intuitive; or the multimodal loss and clustering loss can be weighted and summed using different weights to obtain the target loss. Different weights can be set as needed to make the model biased towards different losses, which better meets user needs. Then, based on the target loss, it is determined whether the preset multimodal model meets the convergence condition. If the preset multimodal model does not meet the convergence condition, gradient backpropagation is used to continue training the preset multimodal model using the multimodal dataset, that is, to continue feature extraction and loss calculation until it is determined that the preset multimodal model meets the convergence condition. If the preset multimodal model meets the convergence condition, it means that the preset multimodal model has converged, and the converged model parameters are output to obtain the trained multimodal model.

[0093] Specifically, when the target loss is less than or equal to the calibrated loss value, or when the number of training iterations of the preset multimodal model reaches the calibrated number of training iterations, the preset multimodal model is determined to meet the convergence condition; conversely, when the target loss is greater than the calibrated loss value, and the number of training iterations of the preset multimodal model has not reached the calibrated number of training iterations, the preset multimodal model is determined to not meet the convergence condition.

[0094] In this embodiment, during the multimodal model training process, the category centers of image and text samples are first obtained through clustering. Then, the historical category center set of the model is updated based on these centers to obtain the clustering loss. This allows the introduction of a multimodal superclass clustering loss function on top of the existing multimodal loss function. This ensures that multimodal features belonging to the same major category are consistently clustered within their corresponding category centers during the model's continuous learning process. This enhances the multimodal model's ability to learn new knowledge from complex and diverse new data, thereby improving the training effect. Furthermore, when the multimodal model learns new knowledge, the category centers in the updated category center set can be considered as anchor points, preventing feature drift during continuous learning. This allows the multimodal model to minimize the forgetting of old categories while continuously receiving new knowledge, improving the performance and robustness of the multimodal model.

[0095] In one embodiment, such as Figure 3 As shown, step S20, which involves clustering text samples and image samples to obtain a dataset of at least one sample category and the category centers of that category, specifically includes the following steps:

[0096] S21: Cluster the text samples and image samples to obtain an initial data set of at least one initial sample category.

[0097] After acquiring the multimodal dataset, the model training device can use a preset clustering algorithm (such as K-means clustering, mean-shift clustering, density-based clustering, or Gaussian mixture model-based clustering) to cluster the text and image samples in the multimodal dataset, thereby obtaining at least one initial dataset, with each initial dataset corresponding to an initial sample category.

[0098] S22: Based on the initial dataset, determine the class relationships between the initial sample classes.

[0099] After clustering text and image samples to obtain an initial dataset and its initial sample categories, the model training device can determine the category relationships between the initial sample categories based on the initial dataset. These category relationships characterize the correlation between two initial sample categories, and include both correlated and uncorrelated relationships.

[0100] The model training device calculates the mean of all samples in the initial dataset to obtain the centroid of the initial dataset, which is also the cluster center. Then, based on the cluster centers of two initial datasets, it calculates the correlation value (e.g., similarity) between the two datasets. Based on the correlation value, it determines the class relationship between the initial sample categories corresponding to the two datasets, with high accuracy. Specifically, if the correlation value between two initial datasets is greater than or equal to a predetermined threshold, the class relationship between the initial sample categories corresponding to each dataset is considered class-related; if the correlation value is less than the predetermined threshold, the class relationship between the initial sample categories corresponding to each dataset is also considered class-related. Furthermore, the class relationship can also be directly represented by the correlation value: the higher the correlation value between two initial sample categories, the closer the two categories are; conversely, the lower the correlation value, the closer the two categories are.

[0101] In other embodiments, a preset similarity model can be used to calculate the similarity of samples in any two initial datasets. The average of the similarities between the samples in the two initial datasets is then used as the category similarity between the initial sample categories corresponding to the two initial datasets, thus obtaining the category relationship between the two initial sample categories. Alternatively, a pre-defined relational knowledge graph, which includes relationships between different categories, can be used to determine the category relationship between each initial sample category. The category relationship can be determined directly based on the clustering results, which is simple and intuitive. In the relational knowledge graph, the edge between two adjacent types (i.e., nodes) represents the correlation between two adjacent initial sample categories, that is, the category relationship between two adjacent initial sample categories.

[0102] S23: Based on the class relationships and the initial dataset, the model training device can merge the initial sample classes to obtain a dataset of at least one sample class and the class center of the sample class.

[0103] After determining the class relationships between the initial sample categories, the initial sample categories are merged according to these relationships to obtain at least one sample category. Then, the initial datasets corresponding to this sample category are merged to obtain the dataset for that merged sample category. Next, the cluster centers of this dataset are calculated to obtain the class centers of the sample categories corresponding to that dataset, thus obtaining the datasets for each sample category and the class centers for each sample category. By reclassifying the clustering results of multimodal data to group it into higher-level classes, the complex structure of multimodal data can be effectively captured, enhancing the model's ability to understand and process the data.

[0104] Specifically, based on the class relationships between the initial sample classes, initial sample classes with a correlation greater than a threshold (or class relationships that are class-related) are merged to obtain a merged sample class. The initial datasets corresponding to each merged initial sample class are then merged to obtain the dataset of the merged sample class. Next, the mean of all samples in the dataset is calculated to obtain the cluster center of the dataset, which is also the class center of the sample class corresponding to the dataset. By traversing all initial sample classes, datasets of one or more sample classes are obtained, along with the class centers of each sample class in the dataset.

[0105] For example, such as Figure 4 As shown, both initial dataset 1 and initial dataset 2 contain multiple samples, represented by black dots in the figure. The cluster center A of initial dataset 1 can be obtained by calculating the mean of the multiple samples in initial dataset 1; similarly, the cluster center B of initial dataset 2 can be obtained by calculating the mean of the multiple samples in initial dataset 2. The correlation value between initial dataset 1 and initial dataset 2 is obtained by calculating the correlation value between cluster centers A and B. If this correlation value is greater than or equal to the calibrated threshold L, then the class relationship between the two initial sample categories corresponding to initial dataset 1 and initial dataset 2 is determined to be class correlation.

[0106] Since the class relationship between the two initial sample categories corresponding to initial dataset 1 and initial dataset 2 is class-related, such as Figure 5 As shown, the initial sample category "smartphone" from initial dataset 1 and the initial sample category "tablet" from initial dataset 2 can be merged to obtain a single sample category that can represent both initial sample categories: "smartphone" and "tablet": "smart device". Then, initial datasets 1 and 2 are merged to form a single dataset M for the sample category "smart device". Finally, the mean of all samples in dataset M is calculated to obtain the cluster center C of dataset M, which is the category center of the sample category "smart device".

[0107] In this embodiment, the initial sample category 1, the initial sample category 2, and the sample category are merely illustrative examples. In other embodiments, the initial sample category and the sample category may be other categories, which will not be elaborated here.

[0108] In this embodiment, by clustering text samples and image samples, an initial data set of at least one initial sample category is obtained; based on the initial data set, the category relationships between the initial sample categories are determined; according to the category relationships and the initial data set, the initial sample categories are merged to obtain a data set of at least one sample category and the category centers of the sample categories. Utilizing superclass clustering technology, multimodal data can be efficiently grouped and represented, effectively capturing the complex structure of multimodal data, improving the accuracy and consistency of data representation, enhancing the model's ability to understand and process data, and thus improving model performance.

[0109] In one embodiment, step S21, which involves clustering text samples and image samples to obtain an initial data set for at least one initial sample category, specifically includes the following steps:

[0110] S211: Extract at least one text word from the text sample, and cluster the text samples based on the text word to obtain a set of text samples with at least one initial sample category.

[0111] After acquiring the multimodal dataset, the model training device can perform word segmentation and keyword extraction on the text samples in the multimodal dataset to extract at least one text word from each text sample. Then, based on the text words extracted from each text sample, the model training device clusters the text samples in the multimodal dataset to classify text samples with the same or similar meanings into the same set, thereby obtaining a set of text samples with at least one initial sample category.

[0112] One approach is to use the synonym clustering technique based on the WordNet vocabulary database. By extracting text words from each text sample, the text samples in the multimodal dataset are clustered to obtain a set of text samples with at least one initial sample category.

[0113] S212: Filter out the current image sample corresponding to the text sample in the text sample set from the image samples.

[0114] After obtaining the initial set of text samples for each sample category, the model training device selects the current image sample corresponding to the text sample in the text sample set from the image samples in the multimodal dataset.

[0115] S213: Add the current image sample to the text sample set to obtain the initial data set of the initial sample category.

[0116] After obtaining the current image sample corresponding to the text sample in the text sample set, the model training device adds the current image sample corresponding to the text sample to the text sample set in which it belongs, and obtains the initial data set of the initial sample category.

[0117] In this embodiment, at least one text word is extracted from the text samples, and the text samples are clustered based on the text words to obtain a set of text samples with at least one initial sample category. The current image sample corresponding to the text sample in the text sample set is selected from the image samples. The current image sample is added to the text sample set to obtain an initial data set for the initial sample category. By extracting text words from the text samples, text samples with the same or similar meanings are classified into the same set to obtain a set of text samples with at least one initial sample category. Then, the image sample corresponding to the text sample is selected and added to its respective text sample set. The clustering process is simple and the clustering effect is good.

[0118] In one embodiment, step S23, which involves merging the initial sample categories based on the category relationships and the initial data set to obtain a data set of at least one sample category and the category center of the sample category, specifically includes the following steps:

[0119] S231: Based on the category relationship, merge the initial sample categories to obtain at least one sample category.

[0120] After determining the class relationships between the initial sample classes, the model training device merges the class-related initial sample classes according to these relationships to obtain at least one sample class. This sample class can represent the merged multiple initial sample classes.

[0121] Specifically, when the class relationship between two initial sample classes indicates that the class relationship is correlated (i.e., the correlation value of the two initial sample classes is greater than or equal to a threshold), the two initial sample classes are merged to obtain a single sample class that can represent the two initial sample classes. When the class relationship between two initial sample classes indicates that the class relationship is uncorrelated (i.e., the correlation value of the two initial sample classes is less than a threshold), the two initial sample classes are not merged.

[0122] For example, such as Figure 5 As shown, the initial sample category 1 of the initial dataset 1 is "smartphone" and the initial sample category 2 is "tablet". Since the correlation value between initial sample category 1 and initial sample category 2 is greater than a threshold, meaning their category relationship is class-related, the initial sample category 1 and initial sample category 2 are merged to obtain a single sample category that can represent both initial sample categories. This sample category can be "smart device". Alternatively, it can also be "technology product".

[0123] S232: Based on the sample category, merge the initial dataset and construct at least one image-text sample pair from the merged data to obtain the dataset of the sample category.

[0124] The model training device, based on the merged sample categories, merges the initial datasets corresponding to the initial sample categories to obtain merged data. It then constructs image-text pairs (positive sample pairs) with the corresponding image samples from the merged data, obtaining at least one image-text pair to form the dataset for that sample category. Furthermore, the text sample can be combined with other image samples from the merged data to form negative sample pairs, resulting in a dataset containing multiple image-text pairs for each sample category.

[0125] For example, such as Figure 6 As shown, initial data set 1 includes at least one text sample A1 and its corresponding image sample B1; initial data set 2 includes at least one text sample A2 and its corresponding image sample B2; initial sample category 1 of initial data set 1 is "smartphone", and initial sample category 2 of initial data set 1 is "tablet computer". After merging initial sample category 1 and initial sample category 2 into the sample category "smart device" according to the category relationship, the model training device merges initial data set 1 and initial data set 2 corresponding to this sample category; in the merged data, text sample A1 and its corresponding image sample B1 are constructed into a text-image sample pair (i.e., a positive sample pair), and text sample A2 and its corresponding image sample B2 are constructed into a text-image sample pair (i.e., a positive sample pair), resulting in the data set M of the sample categories.

[0126] S233: Calculate the class center of the sample class based on the dataset of sample classes.

[0127] After obtaining the dataset of sample categories, the model training device calculates the cluster centers of the sample categories based on the dataset. That is, it calculates the mean of multiple samples in the dataset of a sample category to obtain the cluster center of the dataset, which is then used as the cluster center of that sample category.

[0128] In this embodiment, based on category relationships, initial sample categories are merged to obtain at least one sample category; based on the sample categories, the initial datasets are merged, and at least one image-text sample pair is constructed from the merged data to obtain a dataset of sample categories; based on the dataset of sample categories, the category centers of the sample categories are calculated. By merging categories according to the category relationships of the initial sample categories to obtain sample categories, and then merging the initial datasets corresponding to the initial sample categories involved in the merging, the dataset of sample categories can be obtained. Finally, the category centers are calculated. This clarifies the implementation process of super-clustering technology. The clustering process is simple and intuitive, and the clustered datasets can accurately represent the complex structure of multimodal data, improving the model's data processing capabilities.

[0129] In one embodiment, step S30, which involves using a preset multimodal model to perform feature fusion on the dataset to obtain the fused features of the image-text sample pairs, specifically includes the following steps:

[0130] S301: Use a preset multimodal model to extract features from the dataset to obtain text features of text samples and image features of image samples.

[0131] S302: Fuse the text features of the text sample in the image-text sample pair with the image features of the corresponding image sample to obtain the fused features of the image-text sample pair.

[0132] Specifically, the preset multimodal model includes a text encoding network, an image encoding network, and a multimodal encoding network. After obtaining the dataset of sample categories and the category centers of the sample categories, the model training device inputs the image-text sample pairs (which can be positive or negative sample pairs) from the dataset into the preset multimodal model. The preset multimodal model extracts features from the text samples in the image-text sample pairs to obtain the text features, and also extracts features from the image samples in the image-text sample pairs to obtain the image features. Then, using the multimodal encoding network in the preset multimodal model, the text features of the text sample and the image features of the corresponding image sample are fused to obtain the fused features of the image-text sample pair. Based on these fused features, the model training device calculates the multimodal loss of the image-text sample pair using a preset loss function.

[0133] The pre-defined multimodal models include text coding networks, image coding networks, and multimodal coding networks. For example... Figure 7 As shown, both text encoding networks and image encoding networks include self-attention networks and feedforward neural networks; multimodal encoding networks (multimodal encoders in the example) include self-attention networks, cross-attention networks, and feedforward neural networks.

[0134] After obtaining the dataset of sample categories and the category centers of the sample categories, the model training device inputs the image-text sample pairs from the dataset into a preset multimodal model. The text samples of the image-text sample pairs are sequentially passed through the self-attention network and feedforward neural network of the text encoding network to extract text features, thus obtaining the text features of the text samples. The image samples of the image-text sample pairs are sequentially passed through the self-attention network and feedforward neural network of the image encoding network to extract image features, thus obtaining the image features of the image samples. Then, the text features of the text samples and the image features of the image samples are input into the multimodal encoding network. The text features and image features are processed through the self-attention network, cross-attention network and feedforward neural network to fuse and obtain the fused features of the image-text sample pairs.

[0135] In other embodiments, the preset multimodal model may include an encoding network, through which features are extracted from text samples and their corresponding image samples, and the extracted features are fused to obtain the fused features of the text-image sample pair.

[0136] In this embodiment, a preset multimodal model is used to extract features from the dataset, obtaining text features of text samples and image features of image samples. The text features of the text samples in the image-text sample pair are then fused with the image features of the corresponding image samples to obtain the fused features of the image-text sample pair. By extracting the key features of the text samples and their corresponding image samples separately, and then fusing the two features to obtain the fused features of the image-text sample pair, the mutual interference of features caused by directly using the same coding network for multimodal feature extraction and fusion is reduced, thus improving the accuracy of the fused features.

[0137] In one embodiment, such as Figure 8 As shown, in step S30, which involves determining at least one multimodal loss based on the fused features, the specific steps include:

[0138] S31: Determine the image-text matching loss for the image-text sample pair based on the fused features.

[0139] In this process, after obtaining the fused features of the text-image sample pairs in the dataset, the model training device can use a first neural network model to predict the matching relationship between the text samples and image samples in the text-image sample pair based on these fused features. Specifically, it predicts whether the text samples and image samples in the text-image sample pair match (i.e., whether they are related), thus obtaining the predicted text-image matching information. Then, it calculates the loss between the predicted text-image matching information and the text-image matching label of the text-image sample pair, obtaining the text-image matching loss of the text-image sample pair. The first neural network model can be a pre-defined multimodal model or other multimodal models used to predict text-image matching relationships.

[0140] The image-text matching label of the image-text sample pair is annotated during the construction of the image-text sample pair. It is used to characterize the image-text matching relationship between the text sample and the image sample in the image-text sample pair, that is, a binary classification label indicating whether the text sample and the image sample are related. The image-text matching label can be either a positive sample pair or a negative sample pair.

[0141] S32: Based on the fused features and the preset text mask, determine the text mask loss of the image-text sample pair.

[0142] After obtaining the fused features of the image-text sample pairs in the dataset, the model training device can also determine the text mask loss of the image-text sample pairs based on the fused features and the preset text mask.

[0143] The model training device adds a preset text mask at any position in the text sample of the image-text sample pair to obtain a masked text sample. Then, based on the fused features of the image-text sample pair, a second neural network model is used to perform text restoration processing on the masked text sample to obtain the restored text. Finally, the loss between the restored text and the text sample is calculated to obtain the text mask loss of the image-text sample pair. The first neural network model can be a preset multimodal model or other text prediction models used to restore the masked text.

[0144] S33: Use at least one of the image-text matching loss and the text masking loss as the multimodal loss.

[0145] After obtaining the image-text matching loss and text masking loss for the image-text sample pair, the model training device uses at least one of the image-text matching loss and text masking loss as the multimodal loss. That is, the image-text matching loss and text masking loss can be used together as the multimodal loss for the image-text sample pair; the image-text matching loss of the image-text sample pair can be used as the multimodal loss for the image-text sample pair; or the text masking loss of the image-text sample pair can be used as the multimodal loss for the image-text sample pair.

[0146] In this embodiment, the model training device determines the image-text matching loss of the image-text sample pair based on the fused features; and determines the text masking loss of the image-text sample pair based on the fused features and a preset text mask. At least one of the image-text matching loss and the text masking loss is used as the multimodal loss, improving the diversity of the multimodal loss. The fused features of the image-text sample pair are used to calculate multiple losses. Through a multi-task learning strategy, the model comprehensively considers the multidimensional feature information of both image and text modal data during the learning process. By combining the image-text matching loss and the text masking loss, the multimodal model's ability to understand the deep semantic relationship between images and text is improved, thereby achieving more efficient and accurate multimodal learning.

[0147] In one embodiment, step S31, which is to determine the image-text matching loss of the image-text sample pair based on the fused features, specifically includes the following steps:

[0148] S311: Based on the fused features, the multimodal coding network in the preset multimodal model is used to predict the image-text matching information of the image-text sample pairs, and the predicted image-text matching information is obtained.

[0149] After obtaining the fused features of the image-text sample pairs in the dataset, the model training device can use the multimodal coding network in the preset multimodal model to predict the matching relationship (i.e. whether they are related) between the text samples and image samples in the image-text sample pairs based on the fused features of the image-text sample pairs. In other words, it can predict the image-text matching information of the image-text sample pairs and obtain the predicted image-text matching information of the image-text sample pairs.

[0150] The multimodal coding network includes a multimodal encoder and a first prediction network. The multimodal encoder fuses the text image features of the text sample and the image features of the image sample to obtain fused features. The first prediction network predicts the matching relationship between the text sample and the image sample in the image-text sample pair based on the fused features, thus obtaining the predicted image-text matching information of the image-text sample pair.

[0151] S312: Identify the image-text matching relationship of image-text sample pairs in a multimodal dataset and obtain image-text matching labels.

[0152] After obtaining the predicted image-text matching information of the image-text sample pair, the model training device identifies the image-text matching relationship of the image-text sample pair in the multimodal dataset, that is, the matching relationship between the text sample and the image sample in the image-text sample pair, and obtains the image-text matching label.

[0153] The image-text matching label includes two labels: positive sample pairs and negative sample pairs. A positive sample pair indicates that the image-text matching relationship of the image-text sample pair is a match (i.e., relevant), which can be represented by 1; a negative sample pair indicates that the image-text matching relationship of the image-text sample pair is a mismatch (i.e., irrelevant), which can be represented by 0.

[0154] S313: Based on the predicted image-text matching information and image-text matching labels, determine the image-text matching loss of the image-text sample pair.

[0155] After obtaining the predicted image-text matching information and image-text matching labels for the image-text sample pairs, the loss of the predicted image-text matching information and image-text matching labels is calculated to obtain the image-text matching loss for the image-text sample pairs.

[0156] The cross-entropy loss function can be used to calculate the loss for predicting image-text matching information and image-text matching labels, thus obtaining the image-text matching loss for the image-text sample pair. This image-text matching loss is expressed by the following formula:

[0157] L itm =E (I,T)～D H(y itm ,p itm (I,T));

[0158] Among them, L itm Let represent the image-text matching loss for the image-text sample pair; I represent the image features of the image samples in the image-text sample pair in dataset D; T represent the text features of the text samples in the image-text sample pair; H represent the cross-entropy loss function; y itm The image-text matching label represents the image-text sample pair, that is, the binary label indicating whether the text sample and the image sample in the image-text sample pair match; p itm (I,T) represents the predicted image-text matching information for the image-text sample pair; E represents the mathematical expectation of the loss.

[0159] In other embodiments, other loss functions, such as the KL divergence function, can be used to calculate the loss of predicting image-text matching information and image-text matching labels, thereby obtaining the image-text matching loss of image-text sample pairs, which will not be elaborated here.

[0160] In this embodiment, based on the fused features, a multimodal coding network in a pre-defined multimodal model is used to predict the image-text matching information of image-text sample pairs, thus obtaining the predicted image-text matching information. The image-text matching relationship of the image-text sample pairs is identified in the multimodal dataset, resulting in image-text matching labels. Based on the predicted image-text matching information and the image-text matching labels, the image-text matching loss of the image-text sample pairs is determined. Directly using a pre-defined multimodal model to predict image-text matching information and obtain the image-text matching loss can improve the model's ability to judge the matching relationships between multimodal data, thereby enhancing the model's ability to understand the semantic correspondence between cross-modal inputs, thus improving the model's processing ability for multimodal data and ultimately improving model performance.

[0161] In one embodiment, step S32, namely determining the text mask loss of the image-text sample pair based on the fused features and the preset text mask, specifically includes the following steps:

[0162] S321: Add a preset text mask to the text sample in the image-text sample pair to obtain the masked text sample.

[0163] After obtaining the fused features of the image-text sample pairs in the dataset, the model training device acquires a preset text mask. This preset text mask can be pre-stored text mask information or text mask information randomly generated by the model training device.

[0164] After obtaining a preset text mask, the model training device adds the preset text mask to the text sample in the image-text sample pair to obtain a masked text sample. The preset text mask can be added at any one or more locations within the text sample. In other embodiments, a preset recognition model can be used to identify key information in the text sample, obtaining at least one key information location. After obtaining the preset text mask, a preset text mask is added to at least one key information location in the text sample to obtain a masked text sample. This allows for subsequent reconstruction of the masked text sample from the key information mask, increasing the model's semantic understanding and reconstruction capabilities of the text sample.

[0165] S322: Based on the fused features, the multimodal coding network in the preset multimodal model is used to restore the text sample after masking, and the restored text sample is obtained.

[0166] The multimodal coding network comprises a multimodal encoder, a first prediction network, and a second prediction network. The multimodal encoder fuses the image features of the text sample and the image features of the image sample to obtain fused features. The first prediction network predicts the matching relationship between the text sample and the image sample in the image-text sample pair based on the fused features, obtaining predicted image-text matching information. The second prediction network restores the masked text sample based on the fused features, obtaining the restored text sample.

[0167] After obtaining the masked text sample, the model training device uses the second prediction network of the multimodal coding network in the preset multimodal model, based on the fused features of the image-text sample pair, to restore the text from the masked text sample, thus obtaining the restored text sample. That is, the fused features of the image-text sample pair and the masked text sample are used as input data to the multimodal coding network, enabling the multimodal coding network to predict the masked text information in the masked text sample, thereby obtaining the predicted text information, which is the restored text sample.

[0168] S323: Determine the text mask loss of the image-text sample pair based on the restored text sample and the text sample.

[0169] After obtaining the restored text samples, the model training device determines the text mask loss of the image-text sample pair based on the restored text samples and the loss of the text samples.

[0170] The cross-entropy loss function can be used to calculate the loss between the restored text sample and the text sample, thus obtaining the text mask loss for the image-text sample pair. The text mask loss for this image-text sample pair is expressed by the following formula:

[0171]

[0172] Among them, L mlm denoted as the text masking loss of the image-text sample pair; I represents the image feature of the image sample in the image-text sample pair in the dataset D; y represents the masked text features, that is, the features of the text sample after masking the image-text sample against the Chinese text sample; H represents the cross-entropy loss function; msk This represents the textual features of the image-text sample compared to the Chinese text sample; denoted by , represents the features of the restored text sample, that is, the features of the predicted text sample obtained after restoring the text sample after masking the image-text sample; E represents the mathematical expectation of the loss.

[0173] In other embodiments, based on the position of the preset text mask in the masked text sample, the text information at the same position in the text sample is determined to obtain the original text information, and the text information at the same position in the restored text sample is determined to obtain the restored text information (i.e., the predicted text information); the cross-entropy loss function is used to calculate the loss of the original text information and the restored text information to obtain the text mask loss of the image-text sample pair. The text mask loss of the image-text sample pair is expressed by the following formula:

[0174]

[0175] Among them, L mlm denoted as the text masking loss of the image-text sample pair; I represents the image feature of the image sample in the image-text sample pair in the dataset D; H represents the features of the text sample after masking the image-text sample with respect to the Chinese text sample; H represents the cross-entropy loss function; y msk Features representing the original text information of a masked text sample; E represents the features of the restored text information, that is, the features of the predicted text information obtained after restoring the masked text information of the image-text sample to the Chinese text sample; E represents the mathematical expectation of the loss.

[0176] In this embodiment, the use of the cross-entropy loss function to calculate the text masking loss of the image-text sample pair is merely illustrative; in other embodiments, other loss functions, such as the KL divergence function, can also be used to calculate the text masking loss of the image-text sample pair, which will not be elaborated here.

[0177] In this embodiment, a preset text mask is added to the text sample in the image-text sample pair to obtain the masked text sample. Based on the fused features, a multimodal coding network in a preset multimodal model is used to restore the text of the masked text sample, resulting in the restored text sample. The text masking loss of the image-text sample pair is determined based on the restored text sample and the original text sample. Directly using a preset multimodal model to restore the masked text sample and determine the text masking loss can enhance the model's ability to understand the structure and semantics of natural language. It also guides the model to use image information to fill in language gaps, achieving cross-modal information supplementation, thereby improving the model's ability to process multimodal data and ultimately improving model performance.

[0178] In one embodiment, while determining the image-text matching loss and text masking loss of the image-text sample pair, the image masking loss of the image-text sample pair can also be determined based on the fused features, the image samples in the image-text sample pair, and a preset image mask. Then, at least one of the image-text matching loss, text masking loss, and image masking loss is used as the multimodal loss. That is, in step S30, a preset multimodal model is used to perform feature fusion on the dataset to obtain the fused features of the image-text sample pair, specifically including the following steps:

[0179] S31: Determine the image-text matching loss for the image-text sample pair based on the fused features.

[0180] The specific process for determining the image-text matching loss is described in steps S311 to S313 above, and will not be repeated here.

[0181] S32: Based on the fused features and the preset text mask, determine the text mask loss of the image-text sample pair.

[0182] The specific process for determining the image-text matching loss is described in steps S321 to S323 above, and will not be repeated here.

[0183] S33: Determine the image mask loss of the image-text sample pair based on the fused features, image samples in the image-text sample pair, and the preset image mask.

[0184] Specifically, the process of determining the image masking loss for image-text sample pairs includes the following steps:

[0185] S331: Add a preset image mask to the image sample in the image-text sample pair to obtain the masked image sample.

[0186] After obtaining the fused features of the image-text sample pairs in the dataset, the model training device acquires a preset image mask. This preset image mask can be pre-stored image mask information or randomly generated image mask information by the model training device. The preset image mask can be a mosaic.

[0187] After obtaining a preset image mask, the model training device adds the preset image mask to the image samples in the image-text sample pair to obtain masked image samples. The preset image mask can be added at any one or more locations within the image sample. In other embodiments, a preset recognition model can be used to identify key information in the image sample, obtaining at least one key information location in the image sample. After obtaining the preset image mask, a preset image mask is added to at least one key information location in the image sample to obtain a masked image sample. This allows for subsequent reconstruction of the masked image sample of the key information mask, increasing the model's visual understanding and reconstruction capabilities of the image samples.

[0188] S332: Based on the fused features, the multimodal coding network in the preset multimodal model is used to restore the masked image samples to obtain the restored image samples.

[0189] The multimodal coding network includes a multimodal encoder, a first prediction network, a second prediction network, and a third prediction network. The third prediction network is used to restore the masked image samples based on the fused features of the image-text sample pairs, thus obtaining the restored image samples.

[0190] After obtaining the masked image samples, the model training device uses the second prediction network of the multimodal coding network in the preset multimodal model, based on the fused features of the image-text sample pairs, to restore the masked image samples and obtain the restored image samples. That is, the fused features of the image-text sample pairs and the masked image samples are used as input data for the multimodal coding network, so that the multimodal coding network predicts the masked image information in the masked image samples, and obtains the predicted image information, that is, the restored image samples.

[0191] S333: Determine the image mask loss of the image-text sample pair based on the restored image sample and the image sample.

[0192] After obtaining the restored image samples, the model training device determines the image mask loss of the image-text sample pair based on the restored image samples and the loss of the image samples.

[0193] The cross-entropy loss function can be used to calculate the loss between the restored image sample and the image sample, thus obtaining the image mask loss for the image-text sample pair. This image mask loss for the image-text sample pair is expressed by the following formula:

[0194]

[0195] Among them, L mim The image masking loss represents the image-text sample pair; T represents the text features of the text samples in the image-text sample pair in the dataset D. H represents the features of the image samples after masking the image samples in the image-text sample pair; H represents the cross-entropy loss function; y msk This represents the image features of the image samples in the image-text sample pair; E represents the features of the restored image sample, that is, the features of the predicted image sample obtained after restoring the image sample after masking the image sample in the image-text sample pair; E represents the mathematical expectation of the loss.

[0196] In other embodiments, based on the position of the preset image mask in the masked image sample, image information at the same position in the image sample is determined to obtain the original image information, and image information at the same position in the restored image sample is determined to obtain the restored image information (i.e., the predicted image information); the cross-entropy loss function is used to calculate the loss between the original image information and the restored image information to obtain the image mask loss of the image-text sample pair. The image mask loss of this image-text sample pair is expressed by the following formula:

[0197]

[0198] Among them, L mim The image mask loss represents the image-text sample pair; T represents the text feature of the text sample in the image-text sample pair (I,T) in the dataset D; H represents the features of the image samples after masking the image samples in the image-text sample pair; H represents the cross-entropy loss function; y msk Features representing the original text information of the image samples in an image-text sample pair after they have been masked; E represents the features of the restored image information, that is, the features of the predicted image information obtained after restoring the masked image information of the image samples in the image-text sample pair; E represents the mathematical expectation of the loss.

[0199] In this embodiment, the use of the cross-entropy loss function to calculate the image mask loss of the image-text sample pair is merely an illustrative example; in other embodiments, other loss functions, such as the Dice loss function, can also be used to calculate the image mask loss of the image-text sample pair, which will not be elaborated here.

[0200] In this embodiment, a preset image mask is added to the image sample in the image-text sample pair to obtain a masked image sample. Based on the fused features, a multimodal coding network in a preset multimodal model is used to restore the masked image sample, resulting in a restored image sample. Based on the restored image sample and the image sample, the image masking loss of the image-text sample pair is determined. By occluding a portion of the image and directly using a preset multimodal model to restore the masked image sample to obtain the image masking loss, the model can be guided to infer the missing content of the masked image sample from the text, strengthening the model's visual understanding and semantic association capabilities, thereby improving the model's ability to process multimodal data and ultimately improving model performance.

[0201] S34: Use at least one of the image-text matching loss, text masking loss, and image masking loss as the multimodal loss.

[0202] After obtaining the image-text matching loss, text masking loss, and image masking loss for the image-text sample pairs, the model training device uses at least one of these losses as the multimodal loss. For example, the image-text matching loss, text masking loss, and image masking loss can all be directly used as multimodal losses so that the preset multimodal model can be converged based on the image-text matching loss, text masking loss, image masking loss, and clustering loss, resulting in a trained multimodal model and improving model performance.

[0203] In this embodiment, the image-text matching loss of the image-text sample pair is determined based on the fused features; the text masking loss of the image-text sample pair is determined based on the fused features and a preset text mask; the image masking loss of the image-text sample pair is determined based on the fused features, image samples in the image-text sample pair, and a preset image mask; at least one of the image-text matching loss, text masking loss, and image masking loss is used as the multimodal loss. By adding an image masking loss to the image-text matching loss and text masking loss, the diversity of multimodal losses is increased, enhancing the multimodal model's capabilities in semantic alignment, language understanding, and visual reasoning, thereby achieving a more comprehensive, robust, and adaptable multimodal understanding and generation effect.

[0204] In one embodiment, step S40, which involves determining the clustering loss of image-text sample pairs based on the updated set of category centers, specifically includes the following steps:

[0205] S401: Combine the category centers in the updated category center set to obtain the classification weights of the classification network in the preset multimodal model.

[0206] Among them, such as Figure 9As shown, the preset multimodal model includes a text encoding network, an image encoding network, a multimodal encoding network, and a classification network. Both the text encoding network and the image encoding network include a self-attention network and a feedforward neural network; the multimodal encoding network (the multimodal encoder in the model) includes a self-attention network, a cross-attention network, and a feedforward neural network.

[0207] After obtaining the updated set of class centers, the model training device combines the class centers in the updated set to obtain the classification weights of the classification network in the preset multimodal model.

[0208] In one embodiment, the updated set of category centers can be directly used as the classification weights. Each classification weight includes multiple weight values, and each weight value corresponds to a category center in the updated set of category centers. Alternatively, the category centers in the updated set of category centers can be activated to obtain classification weights that include at least one weight value.

[0209] S402: Based on classification weights and fused features, a classification network is used to predict the category of image-text sample pairs to obtain the predicted category.

[0210] After obtaining the classification weights of the classification network, the model training device uses the classification network to predict the category of the image-text sample pair based on the classification weights and the fused features, thus obtaining the predicted category. Specifically, the classification weights and fused features are input into the classification network, which multiplies the classification weights with the fused features to obtain the target features of the image-text sample pair, and then uses these target features to predict the category of the image-text sample pair, thus obtaining the predicted category of the image-text sample pair.

[0211] The target features can be represented by the following formula:

[0212] z msc (I,T)=W*z(I,T);

[0213] W = [w1, w2, w3, ...];

[0214] Among them, z msc (I,T) represents the target feature of the image-text sample pair, which is the feature obtained by multiplying the classification weight and the fused feature; z(I,T) represents the fused feature of the image-text sample pair; W represents the classification weight, which is a set of weights including multiple weight values ​​w1, w2, w3, ...; each weight value is the category center of the updated category center set.

[0215] S403: Determine the clustering loss for image-text sample pairs based on the predicted category and the sample category.

[0216] After obtaining the predicted categories of the image-text sample pairs, the model training device calculates the loss values ​​of the predicted categories of the image-text sample pairs and the sample categories of the dataset to which the image-text sample pairs belong, thus obtaining the clustering loss of the image-text sample pairs.

[0217] The clustering loss for image-text sample pairs is expressed by the following formula:

[0218] L msc =E (I,T) ~ D H(y msc ,p msc (I,T));

[0219] Among them, L msc The clustering loss for image-text sample pairs is represented by T; the text features of the text samples in the image-text sample pairs in dataset D are represented by I; the image features of the image samples in the image-text sample pairs are represented by H; and the cross-entropy loss function is represented by y. msc This represents the sample category of the image-text sample pair, that is, the sample category corresponding to the data set D; P msc (I,T) represents the predicted category of the image-text sample pair, which is the predicted category obtained by using a classification network based on classification weights and fused features; E represents the mathematical expectation of the loss.

[0220] In this embodiment, the category centers in the updated category center set are combined to obtain the classification weights of the classification network in the preset multimodal model. Based on the classification weights and the fused features, the classification network is used to predict the category of the image-text sample pair to obtain the predicted category. The clustering loss of the image-text sample pair is determined according to the predicted category and the sample category. By combining the category centers in the updated category center set to construct the classification weights of the classification network in the preset multimodal model, the model's category representation ability is made to better fit the true distribution of samples. The fused features of the classification network for the image-text sample pair are used to predict the category, and the error between the predicted category and the true sample category is calculated to obtain the clustering loss of the image-text sample pair. This can optimize the clustering distribution of multimodal features in the semantic space, enhance the model's ability to distinguish category semantics, avoid feature confusion, and enable the model to adapt to new categories during training while maintaining the stability of old categories.

[0221] In one embodiment, such as Figure 10 As shown, step S40 involves updating the historical category center set of the preset multimodal model based on the category centers of the sample categories. This specifically includes the following steps:

[0222] S41: Obtain the category information corresponding to the category center in the historical category center set.

[0223] After acquiring the multimodal dataset, the model training device acquires the historical category center set and obtains the category information corresponding to the category centers in the historical category center set.

[0224] The historical category center set includes at least one historical category center (old category center), meaning that the historical category center set includes at least one historical category center, and each historical category center corresponds to one historical category. In other words, the category information corresponding to the category centers in the historical category center set includes the historical category corresponding to at least one historical category center.

[0225] S42: Based on category information, add the category centers to the historical category center set to obtain the updated category center set.

[0226] After obtaining a dataset of at least one sample class and the class center of the sample class, the model training device determines whether the class center of the sample class can be added to the historical class center set based on the class information corresponding to the class center in the historical class center set. If it is determined that it can be added, the class center of the sample class is added to the historical class center set to obtain the updated class center set.

[0227] In this embodiment, the category information corresponding to the category centers in the historical category center set is obtained; based on the category information, the category centers are added to the historical category center set to obtain the updated category center set, avoiding duplicate categories in the historical category center set, improving the accuracy of the data in the updated category center set, and thus improving the accuracy of the clustering loss.

[0228] In one embodiment, the historical category center set includes historical category centers of at least one historical category. Step S42, which involves adding category centers to the historical category center set based on category information to obtain an updated category center set, specifically includes the following steps:

[0229] S421: When the category information indicates that the sample category does not contain a historical category, add the category center of the sample category to the historical category center set to obtain the updated category center set.

[0230] The set of historical category centers may include historical category centers of at least one historical category.

[0231] After obtaining a dataset of at least one sample category and the category center of the sample category, when the category information corresponding to the category center in the historical category center set indicates that the sample category does not contain a historical category, that is, when the sample category does not contain a historical category corresponding to the category center in the historical category center set, the model training device directly adds the category center of the sample category to the historical category center set to obtain the updated category center set.

[0232] S422: When the category information indicates that the sample category contains a historical category, filter out the category centers other than the historical category in the sample category, and add the filtered category centers to the historical category center set to obtain the updated category center set.

[0233] When the category information indicates that the sample category contains a historical category, that is, when the sample category contains a historical category corresponding to the category center in the historical category center set, the model training device filters out the category centers other than the historical categories in the sample category and adds the filtered category centers to the historical category center set to obtain the updated category center set.

[0234] In one embodiment, the historical category center set includes at least one preset category center and at least one historical category center for a historical category. The preset category centers are reserved for updating the category centers in the preset multimodal model. The preset category centers are obtained through random initialization; during continuous training of the multimodal model, the sample categories and their category centers are determined based on new data, and the preset category centers are updated accordingly; otherwise, they retain their randomly initialized values, ensuring that the data dimension of the model's latest category center set remains unchanged, thus guaranteeing the convergence speed and stability of the model training.

[0235] In other words, when the category information indicates that the sample category does not contain a historical category, the preset category centers in the historical category center set are updated to the category centers of the sample category, resulting in an updated category center set. When the category information indicates that the sample category contains a historical category, category centers other than historical categories are selected from the sample category, and the preset category centers in the historical category center set are updated to the selected category centers, resulting in an updated category center set.

[0236] The number of preset category centers to be updated is determined based on the number of sample categories. For example, when there is only one sample category, a preset category center in the historical category center set is updated to be the category center of that sample category, resulting in an updated category center set.

[0237] In this embodiment, when the category information indicates that the sample category does not contain a historical category, the category center of the sample category is added to the historical category center set to obtain an updated category center set; when the category information indicates that the sample category contains a historical category, the category centers other than the historical categories are filtered out from the sample category, and the filtered category centers are added to the historical category center set to obtain an updated category center set. This avoids duplicate data in the model's category center set and ensures the accuracy of the model's latest category center set.

[0238] In one embodiment, the historical category center set includes historical category centers of at least one historical category. Step S42, which involves adding category centers to the historical category center set based on category information to obtain an updated category center set, specifically includes the following steps:

[0239] S4201: When the category information indicates that the sample category contains a historical category, filter out the target historical category contained in the sample category from the historical categories.

[0240] After obtaining a dataset of at least one sample category and the category center of the sample category, when the category information indicates that the sample category contains a historical category, that is, when the sample category does not contain the historical category corresponding to the category center in the historical category center set, the model training device filters out the target historical category contained in the sample category from the historical categories.

[0241] S4202: Filter out the category center corresponding to the target historical category from the category centers of the sample category to obtain the current category center.

[0242] S4203: In the historical category center set, replace the historical category center of the target historical category with the current category center to obtain the replacement category center set.

[0243] S4204: Add the category centers of the sample category other than the current category center to the replacement category center set to obtain the updated category center set.

[0244] After filtering out the target historical category contained in the sample category from the historical categories, the model training device filters out the category center corresponding to the target historical category from the category center of the sample category to obtain the current category center. In the historical category center set, the historical category center of the target historical category is replaced with the current category center to obtain the replacement category center set. Then, the category centers of the sample category other than the current category center are added to the replacement category center set to obtain the updated category center set.

[0245] In one embodiment, when the historical category center set includes preset category centers, the model training device replaces the historical category centers of the target historical category with the current category centers in the historical category center set to obtain a replaced category center set, and then updates the preset category centers in the replaced category center set with the selected category centers to obtain an updated category center set.

[0246] The number of preset category centers updated is determined based on the number of categories in the sample categories excluding historical categories. For example, if there is a sample category other than historical categories, one preset category center in the replacement category center set will be updated with a category center selected from the sample categories, resulting in the updated category center set.

[0247] In this embodiment, when the category information indicates that the sample category contains a historical category, the target historical category contained in the sample category is selected from the historical categories; the category center corresponding to the target historical category is selected from the category centers of the sample category to obtain the current category center; in the historical category center set, the historical category center of the target historical category is replaced with the current category center to obtain the replaced category center set; the category centers other than the current category center are added to the replaced category center set to obtain the updated category center set. This not only updates the model's category center set with newly added sample categories but also replaces the category centers of historical categories, improving the accuracy of the model's latest category center set. While ensuring that old categories are not forgotten, it enhances the model's ability to learn new categories, thereby improving model performance.

[0248] In one embodiment, taking a preset multimodal model of an instant messaging platform as an example, the specific process of the model training method provided in this application embodiment is described in detail. Figure 11 The diagram illustrates the model training method in this embodiment, which specifically includes the following steps:

[0249] S01: Obtain a multimodal data set from an instant messaging platform.

[0250] After receiving a training instruction for a pre-defined multimodal model, the model training device acquires a multimodal data set from the instant messaging platform. This multimodal data set includes text samples and their corresponding image samples from the instant messaging platform. The text and image samples can be data sent by users in the instant messaging platform's chat window or social channels. Text samples can also be text information converted from voice data sent by users in the instant messaging platform's chat window.

[0251] S02: Perform super-class clustering on text and image samples in the multimodal dataset to obtain sample categories, as well as the dataset of sample categories and category centers.

[0252] After acquiring the multimodal dataset, the model training device clusters the text and image samples in the multimodal dataset to obtain at least one sample category, as well as the dataset of each sample category and the category center of each sample category.

[0253] S03: Update the historical set of category centers of the preset multimodal model based on the category centers of the sample categories to obtain the current set of category centers.

[0254] The model training device updates the historical set of class centers of the preset multimodal model based on the class centers of the sample classes to obtain the current set of class centers.

[0255] See the update process of the historical category center set. Figure 12 .like Figure 12 As shown, the data processing procedure during the update of the historical category center set is as follows:

[0256] 1. Extract text words from text samples in a multimodal dataset.

[0257] 2. Perform super-class clustering on text samples based on text words to obtain a super-class tree.

[0258] The model training device extracts the text words from the text samples and performs entity recognition based on these words to determine the entity names of the text samples. Then, it clusters the entity names of each text sample to obtain initial sample categories. It determines the category relationships between these initial categories and clusters them based on these relationships, merging related initial categories to obtain the final sample categories. The superclass tree includes the extracted text words and entity names from the text samples, as well as the initial sample categories obtained from clustering and the sample categories obtained from further clustering.

[0259] For the structure and data of the superclass tree, see [link / reference]. Figure 13 .like Figure 13 As shown, the model training device extracts text words from text sample 1. The extracted text words include feather, gold, and short-tailed eagle. The entity name identified from the text words of text sample 1 is eagle; the entity names identified from other text samples can also be canary, turtle, and shark.

[0260] 3. Determine the sample categories of the multimodal dataset, as well as the dataset and category centers of the sample categories, based on the superclass tree.

[0261] The model training device can take the total category of the superclass tree (i.e. the category at the top) as the sample category, combine the text samples and corresponding image samples under the sample category to obtain the data set of the sample category, and calculate the mean of all text samples in the data set to obtain the category center of the sample type.

[0262] 4. Update the category center of the new sample category to the historical category center set to obtain the current category center set.

[0263] The historical category center set includes historical category centers of historical categories and multiple preset category centers. The preset category centers are category centers with randomly set initial values. The preset category centers are used to reserve category center update positions for the next model training.

[0264] When the historical categories in the historical category center set do not include the sample category, it indicates that the sample category is a new sample category. The model training device updates the category center of the new sample category to the position of the preset category center in the historical category center set, thus obtaining the current category center set of the preset multimodal model. Figure 12 As shown, it is possible to retain the historical category centers while adding new category centers, thus mitigating the problem of forgetting old knowledge in the model.

[0265] S04: Use a preset multimodal model to perform feature fusion on the dataset to obtain the fused features of the image and text sample pairs.

[0266] A pre-defined multimodal model is used to extract features from the image-text sample pairs in the dataset, obtaining the text features of the text samples and the image features of the image samples. The image features and text images are then fused to obtain the fused features of the image-text sample pairs.

[0267] like Figure 14 As shown, the preset multimodal model includes a text encoding network, an image encoding network, a multimodal encoding network, and a classification network. The multimodal encoding network includes a multimodal encoder and a prediction network. The prediction network includes a first prediction network, a second prediction network, and a third prediction network; the multimodal encoder includes a self-attention network, a cross-attention network, and a feedforward neural network. Both the text encoding network and the image encoding network include a self-attention network and a feedforward neural network.

[0268] Among them, such as Figure 13 As shown, the model training device inputs image-text sample pairs from the dataset into a preset multimodal model: A text encoding network is used to extract features from the text samples in the image-text sample pairs, yielding text features; and an image encoding network is used to extract features from the image samples in the image-text sample pairs, yielding image features. Both text and image samples are processed through a self-attention network and a feedforward neural network to extract corresponding text and image features. The multimodal encoder in the multimodal encoding network then fuses the extracted text and image features to obtain the fused features of the image-text sample pairs.

[0269] S05: Based on the fused features, determine the image-text matching loss, text masking loss, and image masking loss for the image-text sample pair.

[0270] Among them, after obtaining the fused features of the image-text sample pairs, such as Figure 13As shown, the model training device uses the first prediction network in the multimodal coding network to predict the image-text matching relationship of the image-text sample pairs based on the fused features, and obtains the predicted image-text matching information; based on the fused features and the features of the masked text samples, the second prediction network is used to restore the masked text samples, and obtain the restored text samples; based on the fused features and the features of the masked image samples, the third prediction network is used to restore the masked image samples, and obtain the restored image samples.

[0271] Then, the model training device uses the cross-entropy loss function to calculate the loss between the true matching relationship of the image-text sample pair and the predicted image-text matching information, thus obtaining the image-text matching loss; it calculates the loss between the restored text sample and the original text sample, thus obtaining the text mask loss; and it calculates the loss between the restored image sample and the original image sample, thus obtaining the image mask loss.

[0272] S06: Determine the clustering loss for image-text sample pairs based on the fused features and the current set of category centers.

[0273] In this process, after obtaining the fused features of the image-text sample pairs, the model training device uses the current set of category centers as the classification weights of the classification network, and then, as follows... Figure 13 As shown, the model training device inputs the classification weights and fused features into the classification network. The classification network multiplies the classification weights with the fused features to obtain the target features of the image-text sample pair, and then predicts the category of the image-text sample pair based on the target features, thus obtaining the predicted category of the image-text sample pair.

[0274] Then, the model training device uses the cross-entropy loss function to calculate the predicted category and the loss of the sample category of the image-text sample pair, and obtains the clustering loss of the image-text sample pair.

[0275] S07: The target loss is obtained by weighted summation of the image-text matching loss, text masking loss, image masking loss, and clustering loss.

[0276] The model training device uses corresponding weights to perform a weighted summation of the image-text matching loss, text masking loss, image masking loss, and clustering loss to obtain the target loss of the preset multimodal model. The weights of the image-text matching loss, text masking loss, and image masking loss can be the same. In other embodiments, the weights of the image-text matching loss, text masking loss, and image masking loss can also be different, and their weights are configured according to requirements.

[0277] S08: If the convergence condition is not met based on the target loss, continue training the preset multimodal model until the convergence condition is met, so as to obtain the trained multimodal model.

[0278] After obtaining the target loss, the model training device determines whether the target loss is greater than the calibrated loss value. If it is less than or equal to the calibrated loss value, the preset multimodal model is determined to meet the convergence condition; otherwise, if it is greater than the calibrated loss value, the preset multimodal model is determined not to meet the convergence condition. If the convergence condition is not met, the preset multimodal model continues to be trained, i.e., steps S04 to S07 are executed to continue feature extraction and loss calculation until the convergence condition is met, and the converged model parameters are output to obtain the trained multimodal model.

[0279] In this embodiment, during the multimodal model training process, the category centers of image and text samples are first obtained through clustering. Then, the historical category center set of the model is updated based on these centers to obtain the clustering loss. This allows the introduction of a multimodal superclass clustering loss function on top of the existing multimodal loss function. This ensures that multimodal features belonging to the same major category are consistently clustered within their corresponding category centers during the model's continuous learning process. This enhances the multimodal model's ability to learn new knowledge from complex and diverse new data, thereby improving the training effect. Furthermore, when the multimodal model learns new knowledge, the category centers in the updated category center set can be considered as anchor points, preventing feature drift during continuous learning. This allows the multimodal model to minimize the forgetting of old categories while continuously receiving new knowledge, improving the performance and robustness of the multimodal model.

[0280] In one embodiment, after step S30 or step S301, that is, after performing feature fusion (extraction) on the dataset using a preset multimodal model, the method further includes the following steps:

[0281] S60: Based on the text features of text samples and the image features of image samples, perform image-text retrieval on the dataset to obtain the image-text retrieval loss.

[0282] After extracting features from the dataset using a pre-defined multimodal model to obtain text features of text samples and image features of image samples, the model training device performs image-text retrieval on the text samples and image samples in the dataset based on the text features of the text samples and the image features of the image samples, in order to obtain the image-text retrieval loss.

[0283] The model training device can retrieve image samples in the dataset based on at least one of the text features of text samples and the image features of image samples to obtain image retrieval loss, and perform image-text retrieval on text samples in the dataset based on at least one of the image features of image samples to obtain image-text retrieval loss.

[0284] Specifically, after obtaining the image-text retrieval loss for image-text sample pairs, the pre-defined multimodal model is converged based on the multimodal loss and clustering loss to obtain the trained multimodal model, including:

[0285] S50: Based on the multimodal loss, clustering loss, and image / text retrieval loss, the preset multimodal model is converged to obtain the trained multimodal model.

[0286] After obtaining the clustering loss, multimodal loss, and image-text retrieval loss for the image-text sample pairs, the model training device converges the preset multimodal model based on the multimodal loss, clustering loss, and image-text retrieval loss to obtain the trained multimodal model.

[0287] The model training device can fuse the image and text retrieval loss, multimodal loss, and clustering loss to obtain the target loss corresponding to the multimodal dataset. For example, the image and text retrieval loss, multimodal loss, and clustering loss can be directly added together, which is simple and intuitive; or different weights can be used to perform a weighted sum of the image and text retrieval loss, multimodal loss, and clustering loss to obtain the target loss. Different weights can be set as needed to make the model biased towards different losses, which better meets user needs. Then, based on the target loss, it is determined whether the preset multimodal model meets the convergence condition. If the preset multimodal model does not meet the convergence condition, gradient backpropagation is used to continue training the preset multimodal model using the multimodal dataset, that is, steps S10 to S60 are executed until it is determined that the preset multimodal model meets the convergence condition. If the preset multimodal model meets the convergence condition, it means that the preset multimodal model has converged, and the converged model parameters are output to obtain the trained multimodal model.

[0288] In this embodiment, after using a preset multimodal model to perform feature fusion on the dataset to obtain the fused features of image-text sample pairs, the multimodal loss is determined based on the fused features. Then, based on the text features of the text samples and the image features of the image samples, image-text retrieval is performed on the dataset to obtain the image-text retrieval loss. Simultaneously, the historical category center set of the preset multimodal model is updated according to the category centers of the sample categories to determine the clustering loss of the image-text sample pairs. Finally, based on the multimodal loss, clustering loss, and image-text retrieval loss, the preset multimodal model is converged. This allows for simultaneous training of the text encoding network, image encoding network, and multimodal encoding network, improving model training effectiveness and addressing the issues of forgetting old knowledge and performance degradation during continuous training, thereby enhancing model performance.

[0289] On the one hand, the image-text retrieval loss function can train the model to map semantically related image and text sample pairs to similar positions in the feature space, while distancing unrelated sample pairs. This encourages the model to learn and discern the true matching relationship between images and text, thereby significantly improving the model's retrieval accuracy for cross-modal information. On the other hand, the introduction of a multimodal superclass clustering loss function ensures that multimodal features belonging to the same large category are consistently clustered in their corresponding category centers during the model's continuous learning process. This enhances the multimodal model's ability to learn new knowledge from complex and diverse new data. Moreover, when the multimodal model learns new knowledge, the category centers in the updated category center set can be considered as anchor points, preventing feature drift during continuous learning. This allows the multimodal model to minimize the forgetting of old categories while continuously receiving new knowledge, improving the performance and robustness of the multimodal model.

[0290] In one embodiment, step S60, which involves performing image-text retrieval on the dataset based on text and image features to obtain the image-text retrieval loss, specifically includes the following steps:

[0291] S61: Calculate the similarity between the text features of a text sample and the image features of at least one image sample to obtain the first similarity set of the text samples.

[0292] After extracting features from the dataset using a pre-defined multimodal model to obtain the text features of the Chinese text sample and the image features of the image sample, the model training device calculates the similarity between the text features of the image sample and the image features of at least one image sample in the dataset, thus obtaining a first similarity set for the text sample. This first similarity set includes the similarity between the text sample and at least one image sample in the dataset.

[0293] S62: Calculate the similarity between the image features of the image sample and the text features of at least one text sample to obtain a second similarity set of the image samples.

[0294] Simultaneously, the model training device calculates the similarity between the image features of the image sample in the image-text sample pair and the text features of at least one text sample in the dataset, obtaining a second similarity set of image samples. The first similarity set includes the similarity between the text sample and at least one image sample in the dataset.

[0295] S63: Based on the first similarity set, determine the first retrieval loss for text samples, and based on the second similarity set, determine the second retrieval loss for image samples.

[0296] After obtaining the first similarity set and the second similarity set, the model training device can determine the first retrieval loss for text samples based on the first similarity set, and determine the second retrieval loss for image samples based on the second similarity set.

[0297] Specifically, based on the similarity between the text sample and the first similarity set, the softmax function is used to predict the image sample in the dataset that matches the text sample, thus obtaining the predicted matching image sample for the text sample. The image sample corresponding to the text sample in the image-text pair is then determined, and the loss between this predicted matching image sample and the image sample is calculated, resulting in the first retrieval loss for the text sample. This first retrieval loss is expressed by the following formula:

[0298]

[0299] L t =E (I,T) ~ D H(y t2i (T),p t2i (T));

[0300] Where T represents the text features of the image-text samples to the Chinese text samples in the dataset D; I m p represents the image feature of the m-th image sample in the dataset D; t2i (T) represents the predicted matching image sample features of the text feature T; τ represents the temperature coefficient, used to adjust the smoothness of the data distribution of the softmax function; s(T,I) m () represents the text features T of the text sample and the image features I of the image sample in the dataset D. m The similarity between them; M represents the number of sample pairs in the dataset D, that is, the number of image samples; y t2i (T) represents the image feature of the image sample that truly matches the text feature T; L t Let represent the first retrieval loss for text sample T; exp represents the exponential function; H represents the cross-entropy loss function; and E represents the expected value of the loss.

[0301] Specifically, based on the similarity between the image sample and the second similarity set, the softmax function is used to predict the text sample that matches the image sample in the dataset, thus obtaining the predicted matching text sample for the image sample. The text sample corresponding to the image sample in the image-text pair is then determined, and the loss between this predicted matching text sample and the actual loss is calculated to obtain the second retrieval loss for the image sample. This second retrieval loss for the image sample is expressed by the following formula:

[0302]

[0303] Li =E (I,T) ~ D [H(y i2t (I),p i2t (I));

[0304] Where I represents the image features of the image samples in the image-text sample pairs in the dataset D; T m p represents the text feature of the m-th text sample in dataset D; i2t (I) represents the features of the predicted matching text sample of image feature I; τ represents the temperature coefficient, used to adjust the smoothness of the data distribution of the softmax function; s(I,T) m () represents the image feature I and the text feature T of the text sample in the dataset D. m The similarity between them; M represents the number of sample pairs in the dataset D, that is, the number of text samples; y i2t (I) represents the text features of the text sample that truly matches image feature I; L i Let represent the second retrieval loss for image sample I; exp represents the exponential function; H represents the cross-entropy loss function; and E represents the expected value of the loss.

[0305] S64: Combine the first retrieval loss and the second retrieval loss to obtain the image and text retrieval loss.

[0306] After determining the first retrieval loss for the text samples and the second retrieval loss for the image samples in the image-text sample pair, the model training device fuses the first retrieval loss and the second retrieval loss to obtain the image-text retrieval loss for the image-text sample pair.

[0307] The image-text retrieval loss for this image-text sample pair is expressed by the following formula:

[0308]

[0309] Among them, L itc H represents the image-text retrieval loss for the image-text sample pair; H represents the cross-entropy loss function; T represents the text features of the text samples in the image-text sample pair in dataset D; I represents the image features of the image samples in the image-text sample pair in dataset D; H(y t2i (T),p t2i (T) represents the first retrieval loss; H(y) represents the first retrieval loss. i2t (I),p i2t (I) represents the second retrieval loss; p t2i (T) represents the features of the predicted matching image sample for the text feature T; p i2t (I) represents the predicted matching text sample features of image feature I; E represents the mathematical expectation of the loss.

[0310] In this embodiment, the similarity between the text features of a text sample and the image features of at least one image sample is calculated to obtain a first similarity set of text samples; the similarity between the image features of an image sample and the text features of at least one text sample is calculated to obtain a second similarity set of image samples; based on the first similarity set, a first retrieval loss for text samples is determined, and based on the second similarity set, a second retrieval loss for image samples is determined; the first and second retrieval losses are fused to obtain the image-text retrieval loss. By calculating the bidirectional similarity between text samples and image samples separately, constructing a bidirectional retrieval loss function, and fusing them to obtain the image-text retrieval loss, more accurate image-text alignment and cross-modal matching are achieved, improving the retrieval performance and feature discrimination ability of the multimodal model in image-text retrieval tasks.

[0311] In one embodiment, step S50, which involves converging the preset multimodal model based on the multimodal loss, clustering loss, and image / text retrieval loss to obtain the trained multimodal model, specifically includes the following steps:

[0312] S51: The multimodal loss, clustering loss, and image / text retrieval loss are fused to obtain the target loss corresponding to the multimodal data set.

[0313] After obtaining the multimodal loss, clustering loss, and image-text retrieval loss for the image-text sample pairs, the model training device obtains the weights of the multimodal loss, clustering loss, and image-text retrieval loss, respectively, and then performs a weighted sum of the multimodal loss, clustering loss, and image-text retrieval loss based on each weight to obtain the target loss corresponding to the multimodal data set.

[0314] The multimodal loss includes image-text matching loss, text masking loss, and image masking loss.

[0315] The target loss for the multimodal dataset can be determined using the following formula:

[0316] L total =λ1L itc +λ2L msc +λ3(L itm +L mlm +L mim );

[0317] Among them, L itm L represents the image-text matching loss for image-text sample pairs; mlm L represents the text masking loss of this image-text sample pair; mim L represents the image masking loss of the image-text sample pair; msc L represents the clustering loss of the image-text sample pair; itcλ1 represents the image-text retrieval loss for the image-text sample pair; λ2 and λ3 represent the weights of the image-text retrieval loss, the clustering loss, and the multimodal loss, respectively.

[0318] S52: Obtain the incremental learning parameters corresponding to at least one incremental learning method.

[0319] Simultaneously, the model training device can also acquire at least one incremental learning parameter corresponding to the incremental learning method. The incremental learning parameter can be determined according to the incremental learning method. The incremental learning parameter may include at least one parameter such as the old class weight preservation strategy, knowledge distillation factor, class center constraint, sample resampling strategy, and dynamic / static weight balance.

[0320] Incremental learning methods can include regularization-based incremental learning, sample replay-based incremental learning, and dynamic network-based incremental learning.

[0321] S53: Based on the incremental learning parameters and the target loss, the preset multimodal model is incrementally updated to obtain the trained multimodal model.

[0322] After obtaining the incremental learning parameters and the target loss, the model training device incrementally updates the preset multimodal model based on the incremental learning parameters and the target loss to obtain the trained multimodal model.

[0323] The model training device can determine whether the preset multimodal model meets the convergence condition based on the target loss. If the preset multimodal model does not meet the convergence condition, the gradient backpropagation method is used to continue to incrementally update the preset multimodal model using the multimodal data set until it is determined that the preset multimodal model meets the convergence condition. If the preset multimodal model meets the convergence condition, it means that the preset multimodal model has converged, and the converged model parameters are output to obtain the trained multimodal model.

[0324] This model training scheme integrates multimodal loss, clustering loss, and image-text retrieval loss to obtain the target loss corresponding to the multimodal dataset. It also acquires incremental learning parameters corresponding to at least one incremental learning method. Based on the incremental learning parameters and the target loss, the pre-defined multimodal model is incrementally updated to obtain the trained multimodal model. This model training scheme optimizes data by fusing multimodal loss with image retrieval loss supervised by image-text comparison, and combining clustering loss based on category center updates. Incremental learning parameters are then introduced, and the model is continuously trained using incremental learning. This reduces the forgetting of old knowledge in the model, effectively adapts to new categories or samples, and improves the processing capability for new data and new data categories. This enables the continuous evolution and efficient expansion of the multimodal model, thereby improving model performance.

[0325] In one embodiment, taking the application of a multimodal model in an instant messaging platform as an example, the specific process of the model training method provided in this application embodiment is described in detail.

[0326] The backend server of this instant messaging platform can invoke a multimodal model to provide different functions for users within the platform. For example, the instant messaging platform may include an image recognition system, a natural language processing system, and intelligent customer service (such as an intelligent voice assistant), etc. In this case, the backend server can invoke the multimodal model to perform image recognition tasks, natural language processing tasks, and user intent recognition tasks based on the multimodal data input by the user.

[0327] Users can log in to the instant messaging platform via their terminal devices and enable relevant functions to perform information searches or knowledge Q&A based on multimodal data. The backend server of the instant messaging platform can call the multimodal model to perform information searches and matching based on the multimodal data input by the user, and provide the user with corresponding search data and response data.

[0328] Meanwhile, during the operation of the instant messaging platform, the backend server of the instant messaging platform can store different types of new data received or generated in the instant messaging platform to form a multimodal data set. After receiving the multimodal model training instruction, the backend server of the instant messaging platform obtains the multimodal data set to execute the model training method provided in the embodiments of this application to train the multimodal model currently running on the instant messaging platform (i.e., the preset instant messaging platform) to ensure that the multimodal model in the instant messaging platform can adapt to continuously changing new data.

[0329] like Figure 15 The diagram illustrates the model training method in this embodiment, which specifically includes the following steps:

[0330] S1: Obtain a multimodal data set from an instant messaging platform.

[0331] Text samples and image samples can be data sent by users in the chat window of an instant messaging platform or on social channels.

[0332] S2: Perform super-class clustering on text and image samples in the multimodal dataset to obtain sample categories, as well as the dataset of sample categories and category centers.

[0333] S3: Update the historical set of category centers of the preset multimodal model based on the category centers of the sample categories to obtain the current set of category centers.

[0334] This includes the following steps:

[0335] S31: Determine whether the sample category is the historical category corresponding to the historical category center set. If not, proceed to step S32; if yes, proceed to step S33.

[0336] S32: Add the category center of the sample category to the historical category center set.

[0337] S33: Replace the corresponding historical category center in the historical category center set with the category center of the sample category.

[0338] The update process for the historical category center set is detailed below. Figure 12 .like Figure 12 As shown, the data processing procedure during the update of the historical category center set is as follows:

[0339] 1. Extract text words from text samples in a multimodal dataset.

[0340] 2. Perform super-class clustering on text samples based on text words to obtain a super-class tree.

[0341] The super-clustering process is described above and will not be repeated here. Figure 13 As shown, the model training device extracts text words from text sample 1. The extracted text words include feather, gold, and short-tailed eagle. The entity name identified from the text words of text sample 1 is eagle; the entity names identified from other text samples can also be canary, turtle, and shark.

[0342] 3. Determine the sample categories of the multimodal dataset, as well as the dataset and category centers of the sample categories, based on the superclass tree.

[0343] 4. Update the category center of the new sample category to the historical category center set to obtain the current category center set.

[0344] When the historical categories in the historical category center set do not include the sample category, it indicates that the sample category is a new sample category. The model training device updates the category center of the new sample category to the position of the preset category center in the historical category center set, thus obtaining the current category center set of the preset multimodal model. Figure 12 As shown, it is possible to retain the historical category centers while adding new category centers, thus mitigating the problem of forgetting old knowledge in the model.

[0345] S4: Extract the features of the text and image sample pairs in the dataset to obtain text features and image features, and fuse them to obtain the fused features of the text and image sample pairs.

[0346] The process involves using a pre-defined multimodal model to extract features from text-image sample pairs in the dataset, obtaining text features for text samples and image features for image samples. These image and text features are then fused to obtain the fused features of the text-image sample pairs. The specific steps include:

[0347] S41: Extract text features from Chinese text samples in image-text sample pairs using a text encoding network;

[0348] S42: Extract image features from image samples in image-text sample pairs using an image coding network;

[0349] S43: Use a multimodal encoder to fuse image features and text features to obtain fused features.

[0350] The feature extraction and fusion processes are described above and will not be repeated here.

[0351] S5: Perform image-text retrieval on the dataset based on text features and image features, and obtain the image-text retrieval loss.

[0352] This includes the following steps:

[0353] S51: Calculate the feature similarity between text samples and image samples to determine the first similarity set of text samples;

[0354] S52: Calculate the feature similarity between image samples and text samples to determine the second similarity set of image samples;

[0355] S53: Determine the first retrieval loss based on the first similarity set, and determine the second retrieval loss based on the second similarity set;

[0356] S54: The first retrieval loss and the second retrieval loss are fused together to obtain the image and text retrieval loss.

[0357] The specific processing steps are described above and will not be repeated here.

[0358] S6: Based on the fused features, determine the image-text matching loss, text masking loss, and image masking loss for the image-text sample pair.

[0359] Specifically, the steps include the following:

[0360] S61: Determine the image-text matching loss of the image-text sample pair based on the fused features;

[0361] S62: Based on the fused features and the preset text mask, determine the text mask loss of the image-text sample pair;

[0362] S63: Based on the fused features, image samples in the image-text sample pair, and the preset image mask, determine the image mask loss of the image-text sample pair.

[0363] The determination process for the image-text matching loss, text masking loss, and image masking loss is described above and will not be repeated here.

[0364] S7: Determine the clustering loss for image-text sample pairs based on the fused features and the current set of category centers.

[0365] Specifically, the steps include the following:

[0366] S71: Use the updated set of category centers as the classification weights, and each category center in the updated set of category centers is the weight value in the classification weights.

[0367] S72: The classification weights are fused with the fused features to obtain the target features. Based on the target features, a classification network is used to predict the category of the image-text sample pair to obtain the predicted category.

[0368] S73: Calculate the loss for the predicted category and the sample category to obtain the clustering loss for the image-text sample pairs.

[0369] The specific processing steps are described above and will not be repeated here.

[0370] S8: The target loss is obtained by weighted summation of the image-text matching loss, text masking loss, image masking loss, and clustering loss.

[0371] The model training device uses corresponding weights to perform a weighted summation of the image-text matching loss, text masking loss, image masking loss, and clustering loss to obtain the target loss of the preset multimodal model. The weights for the image-text matching loss, text masking loss, and image masking loss can be the same or different, and each weight is configured according to requirements.

[0372] S9: If the convergence condition is not met based on the target loss, continue training the preset multimodal model until the convergence condition is met, so as to obtain the trained multimodal model.

[0373] After obtaining the target loss, the model training device determines whether the target loss is greater than the calibrated loss value. If it is less than or equal to the calibrated loss value, the preset multimodal model is determined to meet the convergence condition; otherwise, if it is greater than the calibrated loss value, the preset multimodal model is determined not to meet the convergence condition. If the convergence condition is not met, training of the preset multimodal model continues, i.e., steps S04 to S08 are executed to continue feature extraction and loss calculation until the convergence condition is met. The converged model parameters are then output to obtain the trained multimodal model. The trained multimodal model includes a text encoding network, an image encoding network, a multimodal encoder in the multimodal encoding network, and a fully connected network. This fully connected network is calibrated according to the model's application scenario; for example, it could be a classifier for content classification, a generative network for generating images, text, or graphics, or a relevance prediction network for relevance prediction.

[0374] In this embodiment, during the multimodal model training process, the category centers of image and text samples are first obtained through clustering. Then, the historical category center set of the model is updated based on these centers to obtain the clustering loss. This allows the introduction of a multimodal superclass clustering loss function on top of the existing multimodal loss function. This ensures that multimodal features belonging to the same major category are consistently clustered within their corresponding category centers during the model's continuous learning process. This enhances the multimodal model's ability to learn new knowledge from complex and diverse new data, thereby improving the training effect. Furthermore, when the multimodal model learns new knowledge, the category centers in the updated category center set can be considered as anchor points, preventing feature drift during continuous learning. This allows the multimodal model to minimize the forgetting of old categories while continuously receiving new knowledge, improving the performance and robustness of the multimodal model.

[0375] After obtaining the trained multimodal model, the trained multimodal model is configured in the backend server of the instant messaging platform so that the backend server can call the latest multimodal model according to the user needs in the instant messaging platform to perform different multimodal data processing tasks, such as image recognition tasks, natural language processing tasks, and various generation tasks based on image recognition and natural language processing.

[0376] In one embodiment, taking the application of a trained multimodal model in a content management system within an instant messaging platform as an example, the multimodal model performs image classification and recognition on multimodal data to process content based on the image classification results. For example... Figure 16 The diagram illustrates the process by which an instant messaging platform manages multimodal content, specifically including the following steps:

[0377] SA1: Obtain multimodal content from an instant messaging platform, which includes at least one image data and the corresponding text data.

[0378] The multimodal content of this instant messaging platform can be text and image data from articles posted by users in the platform's social channels. In other embodiments, the multimodal content may include image data, corresponding text data, and audio (and / or video) data.

[0379] SA2: Input multimodal content into the trained multimodal model to classify the multimodal content and obtain the predicted classification label of the multimodal content.

[0380] The multimodal model includes a text encoding network, an image encoding network, a multimodal encoder, and a recognition network (which can be a classifier). The process of inputting multimodal content into the trained multimodal model and processing the multimodal data includes:

[0381] SA21: Uses an image coding network to extract features from image data in multimodal content to obtain text features of the image data.

[0382] The specific structure of the image coding network is described above, as is the process of feature extraction from image data, and will not be repeated here.

[0383] SA22: Uses a text encoding network to extract features from text data in multimodal content to obtain text features of the text data.

[0384] The specific structure of the text encoding network is described above, as is the process of feature extraction from text data, and will not be repeated here.

[0385] SA23: A multimodal encoder is used to fuse text features and image features to obtain fused features of multimodal content.

[0386] The specific structure of the multimodal encoder is described above, as is the process of feature fusion of text features and image features, and will not be repeated here.

[0387] SA24: A recognition network is used to classify and predict the fused features to obtain predicted classification labels for multimodal content.

[0388] After extracting image features from the image data and text features from the text data, and fusing the two features to obtain fused features, the recognition network in the multimodal model performs image classification prediction (or text classification prediction) on the fused features to obtain the predicted classification label for the image data (or text data), which is also the predicted classification label for the multimodal content. The predicted classification label includes at least a primary classification label and secondary classification labels under the primary classification label. In other embodiments, the predicted classification label can be further refined according to requirements; for example, the predicted classification label may also include tertiary classification labels under the secondary classification labels.

[0389] For example, the image data in the multimodal content might be a picture of a short-tailed hawk, while the text data might be: "a short-tailed hawk with golden feathers." Feature extraction is performed on the image containing the short-tailed hawk to obtain a feature map containing the visual features of the hawk, i.e., image features. Feature extraction is also performed on the text data to obtain a feature combination containing entity words such as "golden," "feathers," and "short-tailed hawk," i.e., text features. The two types of features are then fused, for example, by performing a weighted summation of features using self-attention and cross-attention, resulting in a fused feature map and text features. The recognition network then performs classification prediction on the fused feature, outputting a classification label: animal-bird-hawk, which is also the predicted classification label for the multimodal content. Here, "animal" is the primary classification label, "bird" is the secondary classification label, and "hawk" is the tertiary classification label.

[0390] SA3: Based on the predicted classification labels of the multimodal content, perform content management on the multimodal content.

[0391] After obtaining the predicted classification labels for the multimodal content, content management can be performed on the multimodal content according to actual application needs to meet the different types of functional requirements in the instant messaging platform.

[0392] For example, the multimodal content might be an image generated by a social channel on an instant messaging platform. Taking the data classification and storage function of the instant messaging platform as an example, after obtaining the predicted classification tag for the multimodal content, the multimodal content is stored in the content database corresponding to its predicted classification tag for later retrieval. For instance, the backend server of the instant messaging platform can classify and store the image and text data of each multimodal content according to their predicted classification tags. This allows subsequent functions of the instant messaging platform, such as search systems, recommendation systems, or intelligent customer service, to search or retrieve various types of data based on the classification tags of the image (or text) data.

[0393] In one embodiment, the multimodal content is image data and corresponding text data of an article posted by a user in a social channel of an instant messaging platform. Taking the content moderation function of the content management system in the instant messaging platform as an example, after obtaining the predicted classification tag of the multimodal content, the backend server of the instant messaging platform performs content moderation on the multimodal content and the corresponding article based on its predicted classification tag.

[0394] Specifically, the backend server of the instant messaging platform can determine whether the predicted category tag of the multimodal content matches the preset filter tag. If the predicted category tag of the multimodal content does not match the preset filter tag, it means that the article containing the multimodal content does not meet the publishing conditions, the article review fails, and the article is not published in the social channel. If the predicted category tag of the multimodal content matches the preset filter tag, it means that the article containing the multimodal content does not meet the publishing conditions, the content review fails, and the article containing the multimodal content is not published in the social channel. Alternatively, if the predicted category tag of the multimodal content matches the preset filter tag, it means that the multimodal content does not meet the publishing conditions, the content review fails, and the multimodal content in the article is blocked when publishing the article.

[0395] In one embodiment, taking the content recommendation function of the content management system in an instant messaging platform as an example, after obtaining the predicted classification tags of the multimodal content, the backend server of the instant messaging platform recommends the multimodal content and its corresponding articles based on the predicted classification tags.

[0396] Specifically, the backend server of the instant messaging platform can determine whether the predicted category tag matches the preset priority recommendation tag. If the predicted category tag of the multimodal content does not match the priority recommendation tag, it means that the article containing the multimodal content does not meet the priority recommendation criteria. Therefore, the multimodal content and its corresponding article will not be recommended at this time. Instead, the multimodal content and its corresponding article will be stored in the content database according to their predicted category tags for later retrieval. If the predicted category tag of the multimodal content matches the preset priority recommendation tag, it means that the article containing the multimodal content meets the priority recommendation criteria. Based on the recommendation level corresponding to the matched valid recommendation tag, the multimodal content or its corresponding article will be recommended in the social channels. Higher recommendation levels prioritize the corresponding content.

[0397] In this embodiment, by introducing a multimodal superclass clustering loss function during the multimodal model training process, image and text multimodal features belonging to the same major category can consistently cluster around their corresponding category centers during continuous model learning. This enhances the multimodal model's ability to learn new knowledge from complex and diverse new data. Furthermore, when the multimodal model learns new knowledge, training with the category centers in the updated category center set as anchor points prevents feature drift during continuous learning and minimizes the forgetting of old categories, thereby improving the performance and robustness of the multimodal model. Using a multimodal model trained in this sampling method for content management based on content classification can improve the accuracy of content classification, thus enhancing the accuracy and efficiency of content management, increasing user satisfaction, and ultimately improving user stickiness on the instant messaging platform.

[0398] In one embodiment, taking the application of a trained multimodal model in a search system within an instant messaging platform as an example, such as... Figure 17 The diagram illustrates the implementation process of information search based on multimodal data in an instant messaging platform, specifically including the following steps:

[0399] SB1: Obtain search data input by the user, which includes at least one text data and the corresponding image data.

[0400] The search data may be text data and image data entered by the user on the search page of the instant messaging platform. In other embodiments, the search data may include at least one text data, the corresponding image data, and audio data (and / or video data).

[0401] SB2: Obtain at least one historical data item from the content database of the instant messaging platform.

[0402] The historical data stored in the content database can be at least one of text data, image data, video data, audio data, and multimodal data (such as text and image data).

[0403] SB3: Input the features of the search data and historical data into the trained multimodal model, and use the multimodal model to calculate the correlation between the search data and historical data to obtain the correlation value between the search data and historical data.

[0404] The multimodal model includes a text encoding network, an image encoding network, a multimodal encoder, and a recognition network (the recognition network is used to predict the relevance of the data). The process of inputting search data into the trained multimodal model and processing the multimodal search data includes:

[0405] SB31: Use a text encoding network to extract features from the text data in the search data to obtain the text features of the text data.

[0406] The specific structure of the text encoding network is described above, as is the process of feature extraction from text data, and will not be repeated here.

[0407] SB32: Uses an image coding network to extract features from the image data in the search data to obtain the text features of the image data.

[0408] The specific structure of the image coding network is described above, as is the process of feature extraction from image data, and will not be repeated here.

[0409] SB33: A multimodal encoder is used to fuse text features and image features to obtain fused features of the search data.

[0410] The specific structure of the multimodal encoder is described above, as is the process of feature fusion of text features and image features, and will not be repeated here.

[0411] SB34: A recognition network is used to calculate the correlation between the fused features and the features of historical data to obtain the correlation value between the search data and the historical data.

[0412] The correlation value can be a feature similarity score. That is, the similarity between the fused features and the features of historical data is calculated to obtain the correlation value between the search data and the historical data. The closer the correlation value is to 1, the more relevant the two sets of data are.

[0413] After extracting image features from the image data and text features from the text data, and fusing the two features to obtain the fused features, the recognition network in the multimodal model calculates the similarity between the fused features and the features of historical data to obtain the correlation value between the search data and the historical data.

[0414] SB4: Based on the relevance value between search data and historical data, retrieve historical data with a relevance value greater than a preset value from the content database and use it as recall data for the search data.

[0415] After obtaining the correlation value between the search data and historical data, based on the correlation value between the search data and historical data, historical data with a correlation value greater than a preset value are retrieved from the content database and used as the recall data for the search data.

[0416] For example, historical data might be an image of a gyrfalcon; search data might contain an image of a short-tailed hawk; and multimodal content might contain the text "a short-tailed hawk with golden feathers." Feature extraction is performed on the image containing the short-tailed hawk to obtain a feature map containing its visual features (image features). Feature extraction is also performed on the text data to obtain a feature combination containing entity words such as "golden," "feathers," and "short-tailed hawk" (text features). These two types of features are then fused, for example, using a weighted summation of self-attention and cross-attention features, resulting in a fused feature map. The recognition network then calculates the correlation between the fused feature map and the feature map containing the gyrfalcon feature from the historical data, obtaining the correlation value between the search data and the historical data. If the correlation value between the search data and the historical data is 0.7 (meaning the correlation value is greater than a preset value of 0.6), the historical data is included as recall data for the search. If the correlation value is 0.5 (meaning the correlation value is greater than a preset value of 0.6), the historical data is not included as recall data for the search.

[0417] SB5: Push the recall data of the search data to the terminal device.

[0418] After obtaining the recall data from the search data, the recall data is pushed to the user's terminal device for subsequent browsing and selection.

[0419] In this embodiment, by introducing a multimodal superclass clustering loss function during the multimodal model training process, image and text multimodal features belonging to the same major category can always cluster around the corresponding category centers during the model's continuous learning process. This improves the multimodal model's ability to learn new knowledge from complex and diverse new data. Furthermore, when the multimodal model learns new knowledge, training with the category centers in the updated category center set as anchor points can prevent feature drift during continuous learning and minimize the forgetting of old categories, thereby improving the performance and robustness of the multimodal model. Therefore, training the multimodal model using this sampling method for information search based on data relevance prediction can improve the accuracy of data relevance prediction, thereby improving the accuracy and efficiency of information search, increasing user satisfaction with search results, and ultimately enhancing user stickiness of the instant messaging platform.

[0420] In one embodiment, taking the application of this multimodal model in an intelligent customer service (i.e., intelligent assistant) system within an instant messaging platform as an example, such as... Figure 18 The diagram illustrates the process by which this multimodal model processes user-input multimodal data, specifically including the following steps:

[0421] SC1: Obtain text and image data entered by the user in the intelligent customer service interface.

[0422] In this process, after logging into the instant messaging platform through a terminal device, the user activates the platform's intelligent customer service function (which can be activated by clicking the intelligent customer service control) and controls the terminal device to display the interactive interface of the intelligent customer service. The user enters text and image data in the interactive interface, and the terminal device sends the entered text and image data to the instant messaging platform's backend server. The instant messaging platform's backend server receives the entered text and image data.

[0423] SC2: The text data and image data are used as a multimodal data pair and input into the trained multimodal model to perform user intent recognition, thereby obtaining the intent recognition result of the multimodal data pair.

[0424] The backend server of the instant messaging platform receives text and image data input by the user, treats the text and image data as a multimodal data pair, inputs it into the trained multimodal model for multimodal data processing, and identifies the user's intent to obtain the intent recognition result.

[0425] The multimodal model includes text coding networks, image coding networks, multimodal coding networks, and recognition networks. The specific structures of the text coding networks, image coding networks, and multimodal coding networks are described above. The process by which the multimodal model processes the input multimodal data pairs includes the following steps:

[0426] SC21: The text encoding network of the multimodal model extracts features from the text data in multimodal data pairs to obtain the text features of the text data;

[0427] SC22: The image coding network of the multimodal model extracts features from image data in multimodal data pairs to obtain text features of the image data;

[0428] SC23: The multimodal encoder of the multimodal model fuses the text features of text data and the text features of image data to obtain the fused features of the multimodal data pair;

[0429] SC24: The multimodal model recognition network performs intent recognition on the fused features to obtain the intent recognition result.

[0430] The processing procedures for each feature are described above and will not be repeated here.

[0431] After obtaining the fused features of the multimodal data pairs, the recognition network performs intent recognition on the fused features to obtain the intent recognition results of the multimodal data pairs.

[0432] SC3: Based on the intent recognition results, generate corresponding response data and push the response data to the terminal device for display in the intelligent customer service interface.

[0433] After receiving the intent recognition result, the backend server of the instant messaging platform performs data search or matching based on the intent recognition result to obtain the response data for the user, and sends the response data to the terminal device so that the terminal device can display the response data in the intelligent customer service interactive interface for the user to browse and view.

[0434] For example, when the intent recognition result of the multimodal data pair indicates an image search intent based on the multimodal data pair, the backend server can obtain at least one historical image data from the content database, and use a multimodal model to predict the relevance between the multimodal data pair and the historical image data to obtain the relevance value between the multimodal data pair and the historical image data; then determine the top N historical image data with the largest relevance value to generate response data; and send the response data to the terminal device so that the terminal device displays at least the historical image data with the largest relevance value in the interactive interface of the intelligent customer service.

[0435] For example, when the intent recognition result of the multimodal data pair indicates an intent to generate an image (or text) based on the multimodal data pair, the backend server can use a multimodal model to generate an image (or text) based on the multimodal data pair, obtaining a generated image (or generated text) related to the multimodal data pair; then, based on the generated image (or generated text), it generates response data including the generated image (or generated text); and sends the response data to the terminal device so that the terminal device displays the generated image (or generated text) in the interactive interface of the intelligent customer service for the user to browse.

[0436] In this embodiment, by introducing a multimodal superclass clustering loss function during the multimodal model training process, image and text multimodal features belonging to the same major category can always cluster around the corresponding category centers during the model's continuous learning process. This improves the multimodal model's ability to learn new knowledge from complex and diverse new data. Furthermore, when the multimodal model learns new knowledge, training with the category centers in the updated category center set as anchor points can prevent feature drift during continuous learning and minimize the forgetting of old categories, thereby improving the performance and robustness of the multimodal model. Therefore, training the multimodal model using this sampling method to perform intent-based interactive dialogue can improve the accuracy of user intent recognition, thereby improving the accuracy of response data and user satisfaction, and ultimately increasing user stickiness of the instant messaging platform.

[0437] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0438] In one embodiment, a model training apparatus is provided, which corresponds one-to-one with the model training methods described in the above embodiments. For example... Figure 19 As shown, the model training device includes an acquisition unit 191, a clustering unit 192, an extraction unit 193, an update unit 194, and a training unit 195. Detailed descriptions of each functional module are as follows:

[0439] The acquisition unit 191 is used to acquire a multimodal data set, which includes at least one text sample and an image sample corresponding to the text sample.

[0440] Clustering unit 192 is used to cluster text samples and image samples to obtain a data set of at least one sample category and the category center of the sample category. The data set includes at least one image-text sample pair.

[0441] Extraction unit 193 is used to perform feature fusion on the dataset using a preset multimodal model to obtain the fused features of the image-text sample pair, and to determine at least one multimodal loss based on the fused features;

[0442] The update unit 194 is used to update the historical set of category centers of the preset multimodal model according to the category centers of the sample categories, and to determine the clustering loss of the image and text sample pairs based on the updated set of category centers.

[0443] Training unit 195 is used to converge the preset multimodal model based on multimodal loss and clustering loss to obtain the trained multimodal model.

[0444] In this embodiment, during the multimodal model training process, the category centers of image and text samples are first obtained through clustering. Then, the historical category center set of the model is updated based on these centers to obtain the clustering loss. This allows the introduction of a multimodal superclass clustering loss function on top of the existing multimodal loss function. This ensures that multimodal features belonging to the same major category are consistently clustered within their corresponding category centers during the model's continuous learning process. This enhances the multimodal model's ability to learn new knowledge from complex and diverse new data, thereby improving the training effect. Furthermore, when the multimodal model learns new knowledge, the category centers in the updated category center set can be considered as anchor points, preventing feature drift during continuous learning. This allows the multimodal model to minimize the forgetting of old categories while continuously receiving new knowledge, improving the performance and robustness of the multimodal model.

[0445] In one embodiment, the update unit is specifically used to combine the category centers in the updated category center set to obtain the classification weights of the classification network in the preset multimodal model; based on the classification weights and the fused features, the classification network is used to predict the category of the image-text sample pair to obtain the predicted category; and the clustering loss of the image-text sample pair is determined according to the predicted category and the sample category.

[0446] In one embodiment, the updating unit is further configured to obtain category information corresponding to the category centers in the historical category center set; and based on the category information, add the category centers to the historical category center set to obtain the updated category center set.

[0447] In one embodiment, the historical category center set includes historical category centers of at least one historical category. The updating unit is further configured to: when category information indicates that the sample category does not contain a historical category, add the category center of the sample category to the historical category center set to obtain an updated category center set; when category information indicates that the sample category contains a historical category, filter out category centers other than historical categories from the sample category and add the filtered category centers to the historical category center set to obtain an updated category center set.

[0448] In one embodiment, the historical category center set includes historical category centers of at least one historical category. The updating unit is further configured to: when category information indicates that a sample category contains a historical category, filter out the target historical category contained in the sample category from the historical categories; filter out the category center corresponding to the target historical category from the category centers of the sample category to obtain the current category center; replace the historical category center of the target historical category with the current category center in the historical category center set to obtain a replaced category center set; and add the category centers of the sample category other than the current category center to the replaced category center set to obtain an updated category center set.

[0449] In one embodiment, the clustering unit is specifically used to cluster text samples and image samples to obtain an initial data set of at least one initial sample category; determine the category relationship between the initial sample categories based on the initial data set; and merge the initial sample categories according to the category relationship and the initial data set to obtain a data set of at least one sample category and a category center of the sample category.

[0450] In one embodiment, the clustering unit is further configured to: extract at least one text word from the text sample, and cluster the text sample based on the text word to obtain a set of text samples of at least one initial sample category; filter out the current image sample corresponding to the text sample in the text sample set from the image sample; and add the current image sample to the text sample set to obtain an initial data set of the initial sample category.

[0451] In one embodiment, the clustering unit is further configured to: merge the initial sample categories according to the category relationship to obtain at least one sample category; merge the initial data set based on the sample categories, and construct at least one image-text sample pair in the merged data to obtain a data set of sample categories; and calculate the category center of the sample categories based on the data set of sample categories.

[0452] In one embodiment, the extraction unit is specifically used to: determine the image-text matching loss of the image-text sample pair based on the fused features; determine the text masking loss of the image-text sample pair based on the fused features and a preset text mask; and use at least one of the image-text matching loss and the text masking loss as a multimodal loss.

[0453] In one embodiment, the extraction unit is further configured to: add a preset image mask to the image sample in the image-text sample pair to obtain a masked image sample; based on the fused features, use a multimodal coding network in a preset multimodal model to restore the masked image sample to obtain a restored image sample; determine the image masking loss of the image-text sample pair according to the restored image sample and the image sample; and use at least one of the image-text matching loss, text masking loss and image masking loss as the multimodal loss.

[0454] In one embodiment, the extraction unit is further specifically used to: predict the image-text matching information of the image-text sample pair using a multimodal coding network in a preset multimodal model based on the fused features, and obtain the predicted image-text matching information; identify the image-text matching relationship of the image-text sample pair in the multimodal dataset, and obtain the image-text matching label; and determine the image-text matching loss of the image-text sample pair based on the predicted image-text matching information and the image-text matching label.

[0455] In one embodiment, the extraction unit is further specifically used to add a preset text mask to the text sample in the image-text sample pair to obtain a masked text sample; based on the fused features, use a multimodal coding network in a preset multimodal model to restore the text of the masked text sample to obtain a restored text sample; and determine the text mask loss of the image-text sample pair based on the restored text sample and the text sample.

[0456] In one embodiment, the extraction unit is further specifically used to extract features from the dataset using a preset multimodal model to obtain text features of text samples and image features of image samples; and to fuse the text features of text samples in the image-text sample pair with the image features of the corresponding image samples to obtain the fused features of the image-text sample pair.

[0457] In one embodiment, the model training device further includes a retrieval unit, which is used to perform image-text retrieval on the dataset based on the text features of text samples and the image features of image samples after feature fusion using a preset multimodal model, to obtain an image-text retrieval loss. The training unit is used to converge the preset multimodal model according to the multimodal loss, clustering loss, and image-text retrieval loss to obtain a trained multimodal model.

[0458] In one embodiment, the retrieval unit is specifically configured to: calculate the similarity between the text features of a text sample and the image features of at least one image sample to obtain a first similarity set of text samples; calculate the similarity between the image features of an image sample and the text features of at least one text sample to obtain a second similarity set of image samples; determine a first retrieval loss of text samples based on the first similarity set, and determine a second retrieval loss of image samples based on the second similarity set; and fuse the first retrieval loss and the second retrieval loss to obtain an image-text retrieval loss.

[0459] In one embodiment, the training unit is specifically used to: fuse multimodal loss, clustering loss, and image / text retrieval loss to obtain the target loss corresponding to the multimodal dataset; obtain incremental learning parameters corresponding to at least one incremental learning method; and incrementally update the preset multimodal model according to the incremental learning parameters and the target loss to obtain the trained multimodal model.

[0460] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0461] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0462] In one embodiment, a computer device, which may be a server, is provided. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores data used and generated by the aforementioned model training method, such as preset multimodal models, multimodal datasets, sample category datasets, and various loss data. The network interface of the computer device is used for communication with terminal devices via a network connection. When the computer program is executed by the processor, it implements a model training method.

[0463] This application also provides an electronic device, such as... Figure 20 As shown, the electronic device 20 includes: at least one processor 201, a memory 202, and a computer program 203 stored in the memory 202 and executable on the at least one processor 201. When the processor 201 executes the computer program 203, it implements the steps in any of the above method embodiments, or when the processor 201 executes the computer program 203, it implements the functions of each module / unit in the above device embodiments.

[0464] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 202 and executed by the processor 201 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 203 in the electronic device.

[0465] Those skilled in the art will understand that Figure 17 The electronic device described is merely an example and does not constitute a limitation on the electronic device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0466] The aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0467] The memory can be an internal storage unit of the electronic device, such as a hard drive or RAM. The memory can also be an external storage device of the electronic device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units of the electronic device.

[0468] This application also provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0469] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0470] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0471] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0472] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0473] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0474] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0475] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A model training method, characterized in that, include: Obtain a multimodal data set, the multimodal data set including at least one text sample and an image sample corresponding to the text sample; The text samples and image samples are clustered to obtain a dataset of at least one sample category and a category center for the sample category, wherein the dataset includes at least one image-text sample pair; A preset multimodal model is used to perform feature fusion on the dataset to obtain the fused features of the image and text sample pairs, and at least one multimodal loss is determined based on the fused features. Based on the category centers of the sample categories, the historical category center set of the preset multimodal model is updated, and based on the updated category center set, the clustering loss of the image-text sample pair is determined. Based on the multimodal loss and clustering loss, the preset multimodal model is converged to obtain the trained multimodal model.

2. The model training method as described in claim 1, characterized in that, The step of determining the clustering loss for the image-text sample pairs based on the updated category center set includes: The category centers in the updated category center set are combined to obtain the classification weights of the classification network in the preset multimodal model; Based on the classification weights and the fused features, the classification network is used to predict the category of the image-text sample pair to obtain the predicted category; The clustering loss of the image-text sample pair is determined based on the predicted category and the sample category.

3. The model training method as described in claim 1, characterized in that, The step of updating the historical category center set of the preset multimodal model based on the category centers of the sample categories includes: Obtain the category information corresponding to the category centers in the historical category center set; Based on the category information, the category center is added to the historical category center set to obtain the updated category center set.

4. The model training method as described in claim 3, characterized in that, The historical category center set includes historical category centers for at least one historical category. The step of adding the category centers to the historical category center set based on the category information to obtain an updated category center set includes: When the category information indicates that the sample category does not contain the historical category, the category center of the sample category is added to the historical category center set to obtain the updated category center set; When the category information indicates that the sample category contains the historical category, the category centers other than the historical category are filtered out in the sample category, and the filtered category centers are added to the historical category center set to obtain the updated category center set.

5. The model training method as described in claim 4, characterized in that, The step of adding the category center to the historical category center set based on the category information to obtain the updated category center set also includes: When the category information indicates that the sample category contains the historical category, the target historical category contained in the sample category is filtered out from the historical category; The current category center is obtained by filtering out the category centers corresponding to the target historical category from the category centers of the sample category. In the historical category center set, the historical category center of the target historical category is replaced with the current category center to obtain the replaced category center set; Add the category centers of the sample category other than the current category center to the replacement category center set to obtain the updated category center set.

6. The model training method as described in claim 1, characterized in that, The clustering of the text samples and image samples to obtain a dataset of at least one sample category and the category centers of the sample categories includes: Cluster the text samples and image samples to obtain an initial data set with at least one initial sample category; Based on the initial dataset, determine the class relationships between the initial sample categories; Based on the category relationships and the initial data set, the initial sample categories are merged to obtain at least one data set of the sample categories and the category center of the sample categories.

7. The model training method as described in claim 6, characterized in that, The process of clustering the text samples and image samples to obtain an initial data set with at least one initial sample category includes: At least one text word is extracted from the text sample, and the text sample is clustered based on the text word to obtain a set of text samples of at least one initial sample category; Filter out the current image sample corresponding to the text sample in the text sample set from the image samples; The current image sample is added to the text sample set to obtain the initial data set of the initial sample category.

8. The model training method as described in claim 6, characterized in that, The step of merging the initial sample categories based on the category relationship and the initial data set to obtain at least one data set of the sample categories and the category center of the sample categories includes: Based on the category relationship, the initial sample categories are merged to obtain at least one of the sample categories; Based on the sample category, the initial dataset is merged, and at least one image-text sample pair is constructed from the merged data to obtain the dataset of the sample category; Based on the dataset of the sample categories, calculate the category center of the sample categories.

9. The model training method as described in claim 1, characterized in that, The determination of at least one multimodal loss based on the fused features includes: Based on the fused features, determine the image-text matching loss of the image-text sample pair; Based on the fused features and the preset text mask, the text mask loss of the image-text sample pair is determined; At least one of the image-text matching loss and the text masking loss is used as the multimodal loss.

10. The model training method as described in claim 9, characterized in that, The step of determining the image-text matching loss of the image-text sample pair based on the fused features includes: Based on the fused features, the multimodal coding network in the preset multimodal model is used to predict the image-text matching information of the image-text sample pair, thereby obtaining the predicted image-text matching information; The image-text matching relationship of the image-text sample pairs is identified in the multimodal dataset to obtain image-text matching tags; Based on the predicted image-text matching information and the image-text matching labels, the image-text matching loss of the image-text sample pair is determined.

11. The model training method as described in claim 9, characterized in that, The step of determining the text mask loss for the image-text sample pair based on the fused features and the preset text mask includes: Add a preset text mask to the text sample in the image-text sample pair to obtain the masked text sample; Based on the fused features, the multimodal coding network in the preset multimodal model is used to restore the masked text sample to obtain the restored text sample. Based on the restored text sample and the text sample, determine the text mask loss of the image-text sample pair.

12. The model training method as described in claim 9, characterized in that, Also includes: Add a preset image mask to the image sample in the image-text sample pair to obtain the masked image sample; Based on the fused features, the multimodal coding network in the preset multimodal model is used to restore the masked image sample to obtain the restored image sample. Based on the restored image sample and the image sample, determine the image mask loss of the image-text sample pair; At least one of the image-text matching loss, text masking loss, and image masking loss is used as the multimodal loss.

13. The model training method as described in claim 1, characterized in that, The step of using a preset multimodal model to perform feature fusion on the dataset to obtain the fused features of the image-text sample pairs includes: The preset multimodal model is used to extract features from the dataset to obtain the text features of the text samples and the image features of the image samples. The text features of the text sample in the image-text sample pair are fused with the image features of the corresponding image sample to obtain the fused features of the image-text sample pair.

14. The model training method as described in claim 1, characterized in that, After performing feature fusion on the dataset using a preset multimodal model, the method further includes: Based on the text features of the text samples and the image features of the image samples, image-text retrieval is performed on the dataset to obtain the image-text retrieval loss; The step of converging the preset multimodal model based on the multimodal loss and clustering loss to obtain the trained multimodal model includes: Based on the multimodal loss, clustering loss, and image / text retrieval loss, the preset multimodal model is converged to obtain the trained multimodal model.

15. The model training method as described in claim 14, characterized in that, The step of performing image-text retrieval on the dataset based on the text features of the text samples and the image features of the image samples to obtain the image-text retrieval loss includes: Calculate the similarity between the text features of the text sample and the image features of at least one of the image samples to obtain a first similarity set of the text samples; Calculate the similarity between the image features of the image sample and the text features of at least one of the text samples to obtain a second similarity set of the image samples; Based on the first similarity set, a first retrieval loss for the text sample is determined, and based on the second similarity set, a second retrieval loss for the image sample is determined; The first retrieval loss and the second retrieval loss are fused together to obtain the image and text retrieval loss.

16. The model training method as described in claim 14, characterized in that, The step of converging the preset multimodal model based on the multimodal loss, clustering loss, and image / text retrieval loss to obtain the trained multimodal model includes: The multimodal loss, clustering loss, and image / text retrieval loss are fused together to obtain the target loss corresponding to the multimodal dataset; Obtain the incremental learning parameters corresponding to at least one incremental learning method; The preset multimodal model is incrementally updated based on the incremental learning parameters and the target loss to obtain the trained multimodal model.

17. A model training device, characterized in that, include: An acquisition unit is used to acquire a multimodal data set, the multimodal data set including at least one text sample and an image sample corresponding to the text sample; A clustering unit is used to cluster the text samples and image samples to obtain a data set of at least one sample category and the category center of the sample category, the data set including at least one image-text sample pair; An extraction unit is used to perform feature fusion on the dataset using a preset multimodal model to obtain the fused features of the image-text sample pair, and to determine at least one multimodal loss based on the fused features; The update unit is used to update the historical category center set of the preset multimodal model according to the category center of the sample category, and determine the clustering loss of the image-text sample pair based on the updated category center set. The training unit is used to converge the preset multimodal model based on the multimodal loss and clustering loss to obtain the trained multimodal model.

18. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor running the computer program in the memory to perform the steps of the model training method according to any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the model training method according to any one of claims 1 to 16.

20. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the model training method according to any one of claims 1 to 16.

Images