Modal alignment model training, modal alignment method, device and storage medium

Abstract

The application relates to a modal alignment model training method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: inputting first training modal information and second training modal information into an initial modal alignment model to perform modal alignment representation, obtaining a training modal alignment representation vector; calculating the modal similarity between a first modal alignment representation vector in the training modal alignment representation vector and a second modal alignment representation vector in the training modal alignment representation vector; calculating the probability distribution distance of the first modal alignment representation vector and the second modal alignment representation vector based on the modal similarity; reversely updating the initial modal alignment model based on the probability distribution distance and performing cyclic iteration to obtain a first target modal alignment model, the first target modal alignment model is used for extracting semantic representations of different modal information, and the semantic representations of the same instances in the different modal information have a corresponding relationship. The method can improve the accuracy of modal alignment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, in particular to a modality alignment model training method and device, a modality alignment method and device, a computer device, a storage medium and a computer program product. BACKGROUND

[0002] With the development of artificial intelligence technology, modality alignment technology appears. Modality alignment refers to finding a corresponding relationship between different modal information sub-branches / elements from the same instance, such as aligning "shoes" in a text description to "shoes" in a picture. Modality refers to the organization form or source of information, and common modalities include vision, hearing, text, etc. Different modal information has different organization forms or sources. Currently, when performing modality alignment, a binary classification machine learning model is usually used to perform modality alignment, that is, a machine learning model is trained using training data labeled with whether to align, and the machine learning model is used to perform modality alignment.

[0003] However, since the data labeling quality cannot be guaranteed for modality alignment, the machine learning model trained in this way has the problem of low accuracy when performing modality alignment. SUMMARY

[0004] Therefore, it is necessary to provide a modality alignment model training method and device, a modality alignment method and device, a computer device, a computer readable storage medium and a computer program product, which can improve the accuracy of modality alignment.

[0005] In one aspect, the present application provides a modality alignment model training method. The method comprises:

[0006] obtaining first training modality information and second training modality information;

[0007] inputting the first training modality information and the second training modality information into an initial modality alignment model to perform modality alignment representation, and obtaining a training modality alignment representation vector, the training modality alignment representation vector comprising a first modality alignment representation vector corresponding to the first training modality information and a second modality alignment representation vector corresponding to the second training modality information, and the first modality alignment representation vector and the second modality alignment representation vector having an initial corresponding relationship between the same instance representations;

[0008] calculating the similarity between the first modality alignment representation vector and the second modality alignment representation vector to obtain a modality similarity;

[0009] calculating a probability distribution distance between the first modality alignment representation vector and the second modality alignment representation vector based on the modality similarity, and obtaining vector loss information based on the probability distribution distance;

[0010] The initial modality alignment model is updated reversely based on the vector loss information to obtain an updated modality alignment model, the updated modality alignment model is taken as the initial modality alignment model, and the step of obtaining the first training modality information and the second training modality information is executed until a modality alignment model training completion condition is reached, thereby obtaining a first target modality alignment model, and the first target modality alignment model is used to extract semantic representations of different modality information, and semantic representations of the same instance in the semantic representations of the different modality information have a corresponding relationship.

[0011] In another aspect, the present application also provides a modality alignment model training device. The device comprises:

[0012] An information obtaining module is configured to obtain first training modality information and second training modality information;

[0013] An initial alignment module is configured to input the first training modality information and the second training modality information into an initial modality alignment model for modality alignment representation, thereby obtaining a training modality alignment representation vector, the training modality alignment representation vector comprises a first modality alignment representation vector corresponding to the first training modality information and a second modality alignment representation vector corresponding to the second training modality information, and the same instance representation in the first modality alignment representation vector and the second modality alignment representation vector has an initial corresponding relationship;

[0014] A similarity calculation module is configured to calculate a similarity degree between the first modality alignment representation vector and the second modality alignment representation vector, thereby obtaining a modality similarity degree;

[0015] A loss calculation module is configured to calculate a probability distribution distance of the first modality alignment representation vector and the second modality alignment representation vector based on the modality similarity degree, and obtain vector loss information based on the probability distribution distance;

[0016] An iteration module is configured to update the initial modality alignment model reversely based on the vector loss information, thereby obtaining an updated modality alignment model, taking the updated modality alignment model as the initial modality alignment model, and returning to execute the step of obtaining the first training modality information and the second training modality information until a modality alignment model training completion condition is reached, thereby obtaining a first target modality alignment model, and the first target modality alignment model is used to extract semantic representations of different modality information, and semantic representations of the same instance in the semantic representations of the different modality information have a corresponding relationship.

[0017] In another aspect, the present application also provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0018] Obtaining first training modality information and second training modality information;

[0019] inputting the first training modality information and the second training modality information into an initial modality alignment model for modality alignment representation, to obtain a training modality alignment representation vector, the training modality alignment representation vector including a first modality alignment representation vector corresponding to the first training modality information and a second modality alignment representation vector corresponding to the second training modality information, the same instance representations in the first modality alignment representation vector and the second modality alignment representation vector having an initial corresponding relationship;

[0020] calculating a similarity degree between the first modality alignment representation vector and the second modality alignment representation vector, to obtain a modality similarity degree;

[0021] calculating a probability distribution distance of the first modality alignment representation vector and the second modality alignment representation vector based on the modality similarity degree, and obtaining vector loss information based on the probability distribution distance;

[0022] updating the initial modality alignment model based on the vector loss information, to obtain an updated modality alignment model, taking the updated modality alignment model as the initial modality alignment model, and returning to the step of obtaining the first training modality information and the second training modality information for execution until a modality alignment model training completion condition is reached, to obtain a first target modality alignment model, the first target modality alignment model being used for extracting semantic representations of different modality information, the semantic representations of the same instances in the semantic representations of the different modality information having a corresponding relationship.

[0023] On the other hand, the present application also provides a computer readable storage medium. The computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the following steps:

[0024] obtaining first training modality information and second training modality information;

[0025] inputting the first training modality information and the second training modality information into an initial modality alignment model for modality alignment representation, to obtain a training modality alignment representation vector, the training modality alignment representation vector including a first modality alignment representation vector corresponding to the first training modality information and a second modality alignment representation vector corresponding to the second training modality information, the same instance representations in the first modality alignment representation vector and the second modality alignment representation vector having an initial corresponding relationship;

[0026] calculating a similarity degree between the first modality alignment representation vector and the second modality alignment representation vector, to obtain a modality similarity degree;

[0027] calculating a probability distribution distance of the first modality alignment representation vector and the second modality alignment representation vector based on the modality similarity degree, and obtaining vector loss information based on the probability distribution distance;

[0028] updating the initial modality alignment model based on the vector loss information to obtain an updated modality alignment model, taking the updated modality alignment model as the initial modality alignment model, and returning to the step of obtaining the first training modality information and the second training modality information to execute until a modality alignment model training completion condition is reached to obtain a first target modality alignment model, the first target modality alignment model being used to extract semantic representations of different modality information, and semantic representations of the same instance in the semantic representations of the different modality information having a corresponding relationship.

[0029] In another aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the following steps:

[0030] obtaining first training modality information and second training modality information;

[0031] inputting the first training modality information and the second training modality information into an initial modality alignment model for modality alignment representation to obtain a training modality alignment representation vector, the training modality alignment representation vector comprising a first modality alignment representation vector corresponding to the first training modality information and a second modality alignment representation vector corresponding to the second training modality information, the first modality alignment representation vector and the second modality alignment representation vector having an initial corresponding relationship with respect to the same instance representation;

[0032] calculating a similarity degree between the first modality alignment representation vector and the second modality alignment representation vector to obtain a modality similarity degree;

[0033] calculating a probability distribution distance of the first modality alignment representation vector and the second modality alignment representation vector based on the modality similarity degree, and obtaining vector loss information based on the probability distribution distance;

[0034] updating the initial modality alignment model based on the vector loss information to obtain an updated modality alignment model, taking the updated modality alignment model as the initial modality alignment model, and returning to the step of obtaining the first training modality information and the second training modality information to execute until a modality alignment model training completion condition is reached to obtain a first target modality alignment model, the first target modality alignment model being used to extract semantic representations of different modality information, and semantic representations of the same instance in the semantic representations of the different modality information having a corresponding relationship.

[0035] In one aspect, the present application provides a modality alignment model training method. The method comprises:

[0036] obtaining first training modality information and second training modality information;

[0037] inputting the first training modality information and the second training modality information into an initial modality alignment model for modality alignment representation to obtain a training modality alignment representation vector, the training modality alignment representation vector comprising a first modality alignment representation vector corresponding to the first training modality information and a second modality alignment representation vector corresponding to the second training modality information, the first modality alignment representation vector and the second modality alignment representation vector having an initial corresponding relationship with respect to the same instance representation;

[0038] The first training modal information and the second training modal information are input into an initial modal alignment model to perform modal alignment representation, to obtain a training modal alignment representation vector, the training modal alignment representation vector includes a first modal alignment representation vector corresponding to the first training modal information and a second modal alignment representation vector corresponding to the second training modal information, a similarity degree between the first modal alignment representation vector and the second modal alignment representation vector is calculated, to obtain a modal similarity degree; a probability distribution distance of the first modal alignment representation vector and the second modal alignment representation vector is calculated based on the modal similarity degree, a vector loss information is obtained based on the probability distribution distance, and the initial modal alignment model is iteratively updated based on the vector loss information, until a modal alignment model training completion condition is reached, to obtain the first target modal alignment model.

[0039] The first target modal alignment model is used for modal alignment representation of the first to-be-aligned modal information and the second to-be-aligned modal information, to obtain a target modal alignment representation vector.

[0040] In another aspect, the application also provides a modal alignment model training device. The device comprises:

[0041] A to-be-aligned information acquisition module is configured to acquire first to-be-aligned modal information and second to-be-aligned modal information;

[0042] An input module is configured to input the first to-be-aligned modal information and the second to-be-aligned modal information into a first target modal alignment model; wherein the first training modal information and the second training modal information are input into an initial modal alignment model to perform modal alignment representation, to obtain a training modal alignment representation vector, the training modal alignment representation vector includes a first modal alignment representation vector corresponding to the first training modal information and a second modal alignment representation vector corresponding to the second training modal information, a similarity degree between the first modal alignment representation vector and the second modal alignment representation vector is calculated, to obtain a modal similarity degree; a probability distribution distance of the first modal alignment representation vector and the second modal alignment representation vector is calculated based on the modal similarity degree, a vector loss information is obtained based on the probability distribution distance, and the initial modal alignment model is iteratively updated based on the vector loss information, until a modal alignment model training completion condition is reached, to obtain the first target modal alignment model;

[0043] A modal alignment module is configured to perform modal alignment representation of the first to-be-aligned modal information and the second to-be-aligned modal information by the first target modal alignment model, to obtain a target modal alignment representation vector.

[0044] In another aspect, the present application also provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0045] In another aspect, the present application also provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0046] obtaining first to-be-aligned modal information and second to-be-aligned modal information;

[0047] inputting the first to-be-aligned modal information and the second to-be-aligned modal information into a first target modal alignment model;

[0048] wherein, by inputting the first training modal information and the second training modal information into the initial modal alignment model to perform modal alignment representation, a training modal alignment representation vector is obtained, the training modal alignment representation vector includes a first modal alignment representation vector corresponding to the first training modal information and a second modal alignment representation vector corresponding to the second training modal information, a similarity degree between the first modal alignment representation vector and the second modal alignment representation vector is calculated, and a modal similarity degree is obtained; a probability distribution distance of the first modal alignment representation vector and the second modal alignment representation vector is calculated based on the modal similarity degree, a vector loss information is obtained based on the probability distribution distance, the initial modal alignment model is iteratively updated based on the vector loss information, and until a modal alignment model training completion condition is reached, the first target modal alignment model is obtained;

[0049] by the first target modal alignment model, the first to-be-aligned modal information and the second to-be-aligned modal information are modal alignment represented to obtain a target modal alignment representation vector.

[0050] In another aspect, the present application also provides a computer program product. The computer program product comprises a computer program, and the computer program is executed by a processor to implement the following steps:

[0051] obtaining first to-be-aligned modal information and second to-be-aligned modal information;

[0052] inputting the first to-be-aligned modal information and the second to-be-aligned modal information into a first target modal alignment model;

[0053] The first training modal information and the second training modal information are input into an initial modal alignment model to perform modal alignment representation, to obtain a training modal alignment representation vector, the training modal alignment representation vector includes a first modal alignment representation vector corresponding to the first training modal information and a second modal alignment representation vector corresponding to the second training modal information, a similarity degree between the first modal alignment representation vector and the second modal alignment representation vector is calculated, to obtain a modal similarity degree; a probability distribution distance of the first modal alignment representation vector and the second modal alignment representation vector is calculated based on the modal similarity degree, vector loss information is obtained based on the probability distribution distance, and the initial modal alignment model is iteratively updated based on the vector loss information until a modal alignment model training completion condition is reached, to obtain a first target modal alignment model.

[0054] The first target modal alignment model is used for performing modal alignment representation on the first to-be-aligned modal information and the second to-be-aligned modal information, to obtain a target modal alignment representation vector.

[0055] The modal alignment model training method, the modal alignment method, the device, the computer equipment, the storage medium and the computer program product, the first training modal information and the second training modal information are input into an initial modal alignment model to perform modal alignment representation, to obtain a training modal alignment representation vector, the training modal alignment representation vector includes a first modal alignment representation vector corresponding to the first training modal information and a second modal alignment representation vector corresponding to the second training modal information. Then a similarity degree between the first modal alignment representation vector and the second modal alignment representation vector is calculated, to obtain a modal similarity degree. Then a probability distribution distance of the first modal alignment representation vector and the second modal alignment representation vector is calculated based on the modal similarity degree, and vector loss information is obtained based on the probability distribution distance. The vector loss information is obtained by calculating the probability distribution distance, which can make the obtained vector loss information more accurate. Then the initial modal alignment model is trained based on the vector loss information until a modal alignment model training completion condition is reached, to obtain a first target modal alignment model, so that the modal alignment model obtained by training improves the accuracy of modal alignment. Then the first target modal alignment model obtained by training is used for performing modal alignment representation on the first to-be-aligned modal information and the second to-be-aligned modal information, to obtain a target modal alignment representation vector, so that the accuracy of the obtained target modal alignment representation vector is improved. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 An application environment diagram of the modal alignment model training method in one embodiment;

[0057] Figure 2 A flowchart of the modal alignment model training method in one embodiment;

[0058] Figure 3A flowchart of a process for obtaining a second target modal alignment model in an embodiment;

[0059] Figure 4 A network architecture diagram of a modal alignment classification recognition model in an embodiment;

[0060] Figure 5 A flowchart of a process for obtaining a training modal alignment representation vector in an embodiment;

[0061] Figure 6 A network structure diagram of text feature vector extraction in an embodiment;

[0062] Figure 7 A network structure diagram of image feature vector extraction in an embodiment;

[0063] Figure 8 A flowchart of a process for obtaining a training modal alignment representation vector in an embodiment;

[0064] Figure 9 A process diagram of probability distribution conversion in an embodiment;

[0065] Figure 10 A flowchart of a modal alignment method in an embodiment;

[0066] Figure 11 A flowchart of a process for obtaining a target multi-modal classification recognition model in an embodiment;

[0067] Figure 12 A flowchart of a modal alignment model training method in an embodiment;

[0068] Figure 13 A modal alignment effect diagram in an embodiment;

[0069] Figure 14 A block diagram of a modal alignment model training apparatus in an embodiment;

[0070] Figure 15 A block diagram of a modal alignment apparatus in an embodiment;

[0071] Figure 16 An internal structure diagram of a computer device in an embodiment;

[0072] Figure 17 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0073] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0074] Computer vision (CV) Computer vision is a science that studies how to make machines "see". More specifically, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further perform image processing to make computer processing more suitable for human eye observation or image transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, and tries to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, intelligent transportation, etc. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0075] The key technologies of speech technology include automatic speech recognition technology, speech synthesis technology and voiceprint recognition technology. Letting computers listen, see, speak and feel is the development direction of future human-computer interaction, and voice has become one of the most promising human-computer interaction methods in the future.

[0076] Natural language processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science and mathematics. Therefore, the research in this field will involve natural language, i.e. the language used in daily life, so it has a close relationship with the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph and other technologies.

[0077] The scheme provided by the embodiments of the present application relates to image semantic understanding, speech technology and text processing and other technologies of artificial intelligence, which are specifically explained as follows:

[0078] The modality alignment model training method provided by the embodiments of the present application can be applied to, for example Figure 1The application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 to process. The data storage system can be integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send a model training instruction to the server 104, and the server 104 can obtain first training modal information and second training modal information from the data storage system according to the model training instruction; the server 104 inputs the first training modal information and the second training modal information into the initial modal alignment model for modal alignment representation, to obtain a training modal alignment representation vector, the training modal alignment representation vector includes a first modal alignment representation vector corresponding to the first training modal information and a second modal alignment representation vector corresponding to the second training modal information, and the first modal alignment representation vector and the second modal alignment representation vector have an initial corresponding relationship with the same instance representation; the server 104 calculates the similarity between the first modal alignment representation vector and the second modal alignment representation vector to obtain a modal similarity; based on the modal similarity, the probability distribution distance of the first modal alignment representation vector and the second modal alignment representation vector is calculated, and the vector loss information is obtained based on the probability distribution distance; the server 104 updates the initial modal alignment model based on the vector loss information to obtain an updated modal alignment model, takes the updated modal alignment model as the initial modal alignment model, and returns to the step of obtaining the first training modal information and the second training modal information for execution, until the alignment model training completion condition is reached, to obtain a first target modal alignment model, the first target modal alignment model is used to extract semantic representations of different modal information, and the semantic representations of the same instances in the semantic representations of different modal information have a corresponding relationship. Among them, the terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices, the Internet of Things devices can be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0079] In one embodiment, as shown in Figure 2 , a modal alignment model training method is provided. Taking the server in Figure 1 as an example for illustration, it can be understood that the method can also be applied to the server, and can also be applied to a system including a terminal and a server, and is realized through the interaction of the terminal and the server. In this embodiment, the modal alignment model training method includes the following steps:

[0080] Step 202, obtaining first training modal information and second training modal information.

[0081] The first training modal information refers to modal information used during training, which includes but is not limited to text information, image information, voice information, video information, etc. The second training modal information refers to modal information used during training. The first training modal information and the second training modal information are information of different modalities used during training.

[0082] Specifically, the server can obtain training samples from a database, the training samples including the first training modal information and the second training modal information. The server can also obtain the first training modal information and the second training modal information uploaded by the terminal. The server can also obtain the first training modal information and the second training modal information from a service provider providing data. The server can obtain the first training modal information and the second training modal information from a service provider.

[0083] In step 204, the first training modal information and the second training modal information are input into an initial modal alignment model for modal alignment representation, to obtain a training modal alignment representation vector. The training modal alignment representation vector includes a first modal alignment representation vector corresponding to the first training modal information and a second modal alignment representation vector corresponding to the second training modal information. The first modal alignment representation vector and the second modal alignment representation vector have an initial corresponding relationship with the same instance representation.

[0084] The initial modal alignment model refers to a model parameter initialized modal alignment model. The model parameter initialization can be random initialization, zero initialization, and Gaussian distribution initialization, etc. The modal alignment model can extract a modal alignment semantic representation vector corresponding to the modal information, i.e., the same instance representation in the semantic representation vectors extracted from different modal information has a corresponding relationship. The training modal alignment representation vector refers to a model output modal alignment semantic representation vector obtained during training. The first modal alignment representation vector refers to a modal alignment semantic representation vector corresponding to the first training modal information. The second modal alignment representation vector refers to a modal alignment semantic representation vector corresponding to the second training modal information. The semantic representation vector refers to a vector representation in the semantic space converted from the modal information. The same instance refers to the same instance in different modal information, for example, the description of "clothes" in the text and the "clothes" region in the image are the same instance. The initial corresponding relationship refers to the corresponding relationship extracted using the initialized model parameters.

[0085] Specifically, the server can use a deep neural network to establish an initial modal alignment model, and then train the initial modal alignment model. That is, the first training modal information and the second training modal information are input into the initial modal alignment model to perform modal alignment representation using initialized modal alignment parameters, to obtain an output training modal alignment representation vector. The initialized modal alignment parameters are used to perform modal alignment when performing semantic vector representation. The training modal alignment representation vector includes a first modal alignment representation vector corresponding to the first training modal information and a second modal alignment representation vector corresponding to the second training modal information, and the same instance representation in the first modal alignment representation vector and the second modal alignment representation vector has an initial corresponding relationship.

[0086] In step 206, the similarity between the first modal alignment representation vector and the second modal alignment representation vector is calculated to obtain a modal similarity.

[0087] The modal similarity is used to represent the similarity between the first modal alignment representation vector and the second modal alignment representation vector, and the higher the modal similarity, the more similar the first training modal information corresponding to the first modal alignment representation vector is to the second training modal information corresponding to the second modal alignment representation vector.

[0088] Specifically, the server calculates the similarity between the first modal alignment representation vector and the second modal alignment representation vector using a similarity algorithm to obtain the modal similarity, where the similarity algorithm includes but is not limited to a cosine similarity algorithm, a distance similarity algorithm, etc.

[0089] In step 208, the probability distribution distance between the first modal alignment representation vector and the second modal alignment representation vector is calculated based on the modal similarity, and vector loss information is obtained based on the probability distribution distance.

[0090] The probability distribution distance refers to the minimum moving distance required for the first probability distribution corresponding to the first modal alignment representation vector to be converted to the second probability distribution corresponding to the second modal alignment representation vector. That is, the probability distribution distance refers to the sum of the moving distances of all points in one probability distribution to the nearest points in another probability distribution for minimization. The vector loss information is used to represent the modal alignment error between the first modal alignment representation vector and the second modal alignment representation vector. The smaller the vector loss information, the smaller the modal alignment error, and the higher the accuracy of the modal alignment model when performing modal alignment.

[0091] Specifically, the server can use a probability distribution distance measurement algorithm to calculate the probability distribution distance between the first modal alignment representation vector and the second modal alignment representation vector based on the modal similarity, and then directly use the probability distribution distance as the loss of model training, i.e., to obtain the vector loss information. The probability distribution distance measurement algorithm is an algorithm for measuring the distance between two probability distributions.

[0092] In step 210, the initial modal alignment model is updated based on the vector loss information to obtain an updated modal alignment model. The updated modal alignment model is used as the initial modal alignment model, and the step of obtaining the first training modal information and the second training modal information is executed until the alignment model training completion condition is reached to obtain a first target modal alignment model. The first target modal alignment model is used to extract semantic representations of different modal information, and the semantic representations of the same instances in the different modal information have a corresponding relationship.

[0093] The updated modal alignment model refers to the modal alignment model after updating the model parameters. The first target modal alignment model refers to the trained modal alignment model. The first target modal alignment model is used to extract semantic representations of different modal information, and the semantic representations of the same instances in the different modal information have a corresponding relationship.

[0094] Specifically, the server uses the vector loss information to update the initial model parameters in the initial modal alignment model to obtain an updated modal alignment model. The gradient descent algorithm can be used to update the model parameters, or heuristic algorithms such as simulated annealing, second-order optimization algorithms such as adaptive learning rate, and the like can be used for back propagation. At this time, it is determined whether the alignment model training completion condition is reached. When the alignment model training completion condition is not reached, the updated modal alignment model is used as the initial modal alignment model, and the step of obtaining the first training modal information and the second training modal information is executed until the alignment model training completion condition is reached. When the alignment model training completion condition is reached, the modal alignment model when the alignment model training completion condition is reached is used as the first target modal alignment model. The alignment model training completion condition refers to the condition for training the first target modal alignment model, including but not limited to the number of iterations reaching a maximum upper limit, the loss information reaching a preset threshold, the model parameters no longer changing, and the like. The first target modal alignment model is used to extract semantic representations of different modal information, and the semantic representations of the same instances in the different modal information have a corresponding relationship.

[0095] The model alignment model training method aligns the first training modality information and the second training modality information by inputting the first training modality information and the second training modality information into the initial modality alignment model to perform modality alignment representation, to obtain a training modality alignment representation vector. The training modality alignment representation vector includes a first modality alignment representation vector corresponding to the first training modality information and a second modality alignment representation vector corresponding to the second training modality information. Then, a similarity between the first modality alignment representation vector and the second modality alignment representation vector is calculated to obtain a modality similarity. Then, a probability distribution distance between the first modality alignment representation vector and the second modality alignment representation vector is calculated based on the modality similarity, and a vector loss information is obtained based on the probability distribution distance. The vector loss information is obtained by calculating the probability distribution distance, which can make the obtained vector loss information more accurate. Then, the initial modality alignment model is trained based on the vector loss information until a model alignment training completion condition is reached, to obtain a first target modality alignment model, so that the modality alignment model obtained by training improves the accuracy of modality alignment.

[0096] In one embodiment, as shown in Figure 3 The model alignment model training method further includes:

[0097] In step 302, a modality alignment label corresponding to the first training modality information and the second training modality information is obtained.

[0098] The modality alignment label is a label indicating whether the first training modality information and the second training modality information are aligned.

[0099] Specifically, the server can obtain the modality alignment label corresponding to the first training modality information and the second training modality information from the database. The server can also obtain the modality alignment label corresponding to the first training modality information and the second training modality information uploaded by the terminal.

[0100] In step 304, the first training modality information and the second training modality information are input into the initial modality alignment classification and recognition model to perform modality alignment representation, to obtain a training modality alignment representation vector. The modality alignment classification and recognition is performed based on the training modality alignment representation vector, to obtain a modality alignment classification and recognition result.

[0101] The initial modality alignment classification and recognition model is a model parameter initialized modality alignment classification and recognition model. The modality alignment classification and recognition model is a binary classification model used to identify whether the same instances between the first training modality information and the second training modality information are aligned. The modality alignment classification and recognition result is a classification and recognition result of whether the first training modality information and the second training modality information are aligned, including an aligned result of the same instances between the first training modality information and the second training modality information and a non-aligned result of the same instances between the first training modality information and the second training modality information.

[0102] Specifically, the server inputs the first training modal information and the second training modal information into the initial modal alignment classification recognition model, performs modal alignment representation using the initialized modal alignment parameters, obtains a training modal alignment representation vector, and performs modal alignment classification recognition based on the training modal alignment representation vector using the initial classification recognition parameters, to obtain a modal alignment classification recognition result. The initialized modal alignment parameters are used to perform modal alignment when performing semantic vector representation. The initial classification recognition parameters are used to classify whether the same instances in the training modal alignment representation vector are aligned.

[0103] Step 306, perform classification loss calculation based on the modal alignment classification recognition result and the modal alignment label, to obtain classification loss information.

[0104] The classification loss information is used to represent the error between the modal alignment classification recognition result and the modal alignment label. The smaller the classification loss information is, the smaller the error is, and the more accurate the modal alignment classification recognition model obtained by training is in performing modal alignment.

[0105] Specifically, the server can use a cross-entropy loss function to calculate the error between the modal alignment classification recognition result and the modal alignment label, to obtain the classification loss information.

[0106] Step 308, perform model loss information based on the classification loss information and the vector loss information, to obtain model loss information.

[0107] The model loss information is used to represent the error of the modal alignment classification recognition model in performing modal alignment classification recognition during training.

[0108] Specifically, the server calculates the average loss information of the classification loss information and the vector loss information, and takes the average loss information as the model loss information.

[0109] In one embodiment, the server can weight the classification loss information and the vector loss information to obtain weighted classification loss information and weighted vector loss information, wherein the weighting weight can be pre-set. Then, the sum of the loss information of the weighted classification loss information and the weighted vector loss information is calculated to obtain the model loss information.

[0110] Step 310, update the initial modal alignment classification recognition model based on the model loss information to obtain an updated modal alignment classification recognition model, take the updated modal alignment classification recognition model as the initial modal alignment classification recognition model, and return to the step of obtaining the first training modal information and the second training modal information to execute until a classification model training completion condition is reached, to obtain a target modal alignment classification recognition model.

[0111] The classification model training completion condition refers to a condition for training the target modality alignment classification recognition model, including that the classification loss information reaches a preset threshold, the number of iterations reaches a maximum number of iterations, or the model parameters no longer change, etc. The target modality alignment classification recognition model refers to the trained modality alignment classification recognition model.

[0112] Specifically, the server uses a back propagation algorithm to update the initial model parameters in the initial modality alignment classification recognition model through model loss information to obtain an updated modality alignment classification recognition model. The back propagation algorithm can be a gradient descent algorithm, a simulated annealing heuristic algorithm, and a second-order optimization algorithm such as AdamW. At this time, it can be determined whether the classification model training completion condition is reached. When the classification model training completion condition is not reached, the updated modality alignment classification recognition model is used as the initial modality alignment classification recognition model, and the step of obtaining the first training modality information and the second training modality information is returned to be executed until the classification model training completion condition is reached. The modality alignment classification recognition model when the classification model training completion condition is reached is used as the target modality alignment classification recognition model.

[0113] In step 312, a second target modality alignment model is obtained based on the target modality alignment classification recognition model.

[0114] The second target modality alignment model refers to a target modality alignment model trained using the classification loss information and the vector loss information.

[0115] Specifically, the server can use the trained modality alignment parameters in the target modality alignment classification recognition model and the network structure corresponding to the modality alignment parameters as the second target modality alignment model.

[0116] In one embodiment, the initial modality alignment classification recognition model includes an initial modality alignment representation network and an initial classification network.

[0117] In step 304, the first training modality information and the second training modality information are input into the initial modality alignment classification recognition model for modality alignment representation, to obtain a training modality alignment representation vector, and based on the training modality alignment representation vector, modality alignment classification recognition is performed to obtain a modality alignment classification recognition result, including the steps of:

[0118] The first training modality information and the second training modality information are modality alignment represented by the initial modality alignment representation network in the initial modality alignment classification recognition model to obtain a training modality alignment representation vector. The initial classification network in the initial modality alignment classification recognition model is used for modality alignment classification recognition to obtain a modality alignment classification recognition result.

[0119] The initial modal alignment representation network refers to a parameter-initialized semantic representation neural network used to extract semantic representation vectors of model information after modal alignment. The initial classification network refers to a parameter-initialized classification neural network used to perform binary classification identification of whether the modal alignment is performed on the training modal alignment representation vectors.

[0120] Specifically, when the initial modal alignment classification identification model in the server obtains the input first training modal information and second training modal information, the initial modal alignment representation network is used to perform modal alignment representation to obtain the training modal alignment representation vector output by the initial modal alignment representation network, and then the training modal alignment representation vector is input to the initial classification network to perform modal alignment classification identification to obtain the output modal alignment classification identification result.

[0121] Step 312, obtaining the second target modal alignment model based on the target modal alignment classification identification model, includes the following steps:

[0122] The target modal alignment representation network in the target modal alignment classification identification model is used as the second target modal alignment model.

[0123] The target modal alignment representation network refers to a trained modal alignment representation network.

[0124] Specifically, the server can directly use the target modal alignment representation network in the target modal alignment classification identification model as the second target modal alignment model.

[0125] In one specific embodiment, as shown in Figure 4 A network architecture diagram of the modal alignment classification identification model is provided, and specifically, the modal alignment classification identification model includes a modal alignment network, a fully connected layer, and a classification layer, and the modal alignment network is established using the network architecture of the encoding network of the transformer network. When the input first training modal information and second training modal information are obtained, the modal alignment network is used to perform modal alignment representation, that is, the attention feature extraction is performed through the multi-head attention network in the modal alignment network, then the attention feature is fused with the input modal information, and is standardized and then forwarded through the neural network, then the forward propagation result is obtained, then the forward propagation result is fused with the last standardization result and is standardized, and then the output training modal alignment representation vector is obtained, then the training modal alignment representation vector is input to the fully connected layer and the classification layer to perform modal alignment classification identification, and then the output modal alignment classification identification result is obtained.

[0126] In the above embodiment, by obtaining the modal alignment label corresponding to the first training modal information and the second training modal information, then obtaining the modal alignment classification recognition result through the initial modal alignment classification recognition model, then performing classification loss calculation based on the modal alignment classification recognition result and the modal alignment label to obtain the classification loss information, and then using the classification loss information and the vector loss information to obtain the model loss information, the obtained model loss information can be more accurate, and finally using the model loss information to train the initial modal alignment classification recognition model to obtain the target modal alignment classification recognition model, and then obtaining the second target modal alignment model from the target modal alignment classification recognition model, thereby improving the accuracy of the obtained second target modal alignment model.

[0127] In one embodiment, as shown in Figure 5 step 204, inputting the first training modal information and the second training modal information into the initial modal alignment model to perform modal alignment representation to obtain a training modal alignment representation vector, including:

[0128] Step 502, respectively extracting features of the first training modal information and the second training modal information to obtain a first modal feature vector and a second modal feature vector.

[0129] Wherein, the first modal feature vector refers to the feature vector of the first training modal information extracted. The second modal feature vector refers to the feature vector of the second training modal information extracted. The first modal feature vector and the second modal feature are modal unaligned feature vectors.

[0130] Specifically, the server respectively performs feature extraction on the first training modality information and the second training modality information to obtain a first modality feature vector and a second modality feature vector. The first modality feature vector and the second modality feature vector can be obtained in parallel through concurrent feature extraction, or the first modality feature vector can be obtained through feature extraction on the first training modality information first, and then the second modality feature vector can be obtained through feature extraction on the second training modality information. Alternatively, the second modality feature vector can be obtained through feature extraction on the second training modality information first, and then the first modality feature vector can be obtained through feature extraction on the first training modality information. Different modalities of information can use different feature extraction methods. For example, when the first training modality information is text information, a text feature extraction algorithm can be used to perform text feature extraction to obtain a text feature vector; when the first training modality information is image information, an image feature extraction algorithm can be used to perform image feature extraction to obtain an image feature vector; when the first training modality information is speech information, the speech can be converted into text, and a text feature extraction algorithm can be used to perform feature extraction to obtain a speech feature vector, and so on. When the second training modality information is text information, a text feature extraction algorithm can be used to perform text feature extraction to obtain a text feature vector; when the second training modality information is image information, an image feature extraction algorithm can be used to perform image feature extraction to obtain an image feature vector; when the second training modality information is speech information, the speech can be converted into text, and a text feature extraction algorithm can be used to perform feature extraction to obtain a speech feature vector, and so on. The text feature extraction algorithm can be a bert (Bidirectional Encoder Representation from Transformers, pre-trained language representation model) model to extract text features, an LSTM (Long Short-Term Memory) model, a CNN (Convolutional Neural Network) model, or the like to extract text modality features, and so on. The image feature extraction algorithm can be a vision transformer model to extract picture features, or a resnet model, a noisy student model, or the like to extract picture features.

[0131] In one specific embodiment, as Figure 6The diagram illustrates the network structure for text feature vector extraction. Specifically, the input text "Cats are cute, I like them very much" is obtained and fed into the BERT model for classification. The BERT model vectorizes the text semantics and then performs classification through a classification layer. At this point, all representation vectors from the hidden layer preceding the BERT classifier (classification layer) are obtained. The CLS token vectors are removed from all representation vectors, resulting in representation vectors corresponding to all characters. These representation vectors are then used as the extracted text feature vectors.

[0132] In a specific embodiment, such as Figure 7 The diagram illustrates the network structure for image feature vector extraction. Specifically, after acquiring image modal information, the image is input into the Vision Transformer model. The Vision Transformer model divides the image into blocks, flattens them into sequences, and inputs them into an image vectorization layer for vectorization. Then, these sequences are input into an encoding layer for encoding, and finally, they are input into a fully connected layer for classification to obtain the classification category. Next, the output of the hidden layer preceding the Vision Transformer Classification layer (visual translation classification layer) is used to extract all representation vectors. The CLS token vectors are then removed from all representation vectors to obtain the representation vectors corresponding to all image blocks. These representation vectors corresponding to all image blocks are used as the image feature vectors extracted from the image modal information.

[0133] Step 502: Fuse the first modality feature vector and the second modality feature vector to obtain the fused feature vector.

[0134] Specifically, the server can fuse the first modality feature vector and the second modality feature vector. This can be done by directly concatenating the first and second modality feature vectors to obtain the fused feature vector. During concatenation, the first modality feature vector can be concatenated first, followed by the second modality feature vector, or vice versa. Then, vector operations can be performed on the first and second modality feature vectors to obtain the fused features, such as vector addition or vector multiplication.

[0135] Step 502: Input the fused feature vector into the initial modality alignment model for modality alignment representation to obtain the trained modality alignment representation vector.

[0136] Specifically, the server then inputs the fused feature vector into the initial modality alignment model for modality alignment representation, thereby obtaining the trained modality alignment representation vector.

[0137] In one specific embodiment, the server can input the fused feature vector into an initial modal alignment model of a network structure as shown in Figure 3 to perform modal alignment representation, to obtain a training modal alignment representation vector.

[0138] In the above embodiment, by first extracting features of different training modal information respectively to obtain corresponding feature vectors, and then performing modal alignment representation after fusing the feature vectors corresponding to different training modal information to obtain a training modal alignment representation vector, the accuracy of the obtained training modal alignment representation vector is improved.

[0139] In one embodiment, the first training modal information includes text information, and the second training modal information includes picture information.

[0140] The feature extraction of the first training modal information and the second training modal information respectively obtains a first modal feature vector and a second modal feature vector, including:

[0141] The text information is input into a text feature extraction model, and the text global representation vector and the text character representation vector are obtained by the text feature extraction model, and the text character representation vector is taken as the first modal feature vector.

[0142] The picture information is input into a picture feature extraction model, and the picture global representation vector and the picture content representation vector are obtained by the picture feature extraction model, and the picture content representation vector is taken as the second modal feature vector.

[0143] Wherein, the text information refers to the modal information in the form of text, and the picture information refers to the modal information in the form of picture. The text global representation vector refers to a vector for representing the global of the text information. The text character representation vector refers to a representation vector for representing the characters in the text. The picture global representation vector refers to a vector for representing the global of the image information. The picture content representation vector refers to a representation vector for representing the content in the image.

[0144] Specifically, the server inputs the text information into a text feature extraction model, and obtains the text global representation vector and the text character representation vector by the text feature extraction model, and then takes the text character representation vector as the first modal feature vector. The server inputs the picture information into a picture feature extraction model, and extracts the picture global representation vector and the picture content representation vector by the picture feature extraction model, and takes the picture content representation vector as the second modal feature vector. In one embodiment, the vector dimensions of the first modal feature vector and the second modal feature vector are the same.

[0145] In the above embodiment, by using the text character representation vector and the picture content representation vector as the modal feature vector, the accuracy of the modal feature vector in modal alignment can be improved.

[0146] In one embodiment, as shown in Figure 8 Step 204, input the first training modal information and the second training modal information into the initial modal alignment model for modal alignment representation, to obtain a training modal alignment representation vector, including:

[0147] Step 802, input the first training modal information and the second training modal information into the initial modal alignment model.

[0148] Step 804, the initial modal alignment model is used to extract features from the first training modal information and the second training modal information respectively, to obtain a first modal feature vector and a second modal feature vector.

[0149] Specifically, the server can directly input the first training modal information and the second training modal information into the initial modal alignment model, and then the initial modal alignment model can extract features from the first training modal information and the second training modal information respectively to obtain the first modal feature vector and the second modal feature vector which need to be aligned. That is, feature extraction can be performed through the initial modal feature extraction network in the initial modal alignment model, for example, a transformer network can be used for modal feature extraction and fusion.

[0150] Step 806, the initial modal alignment model is used to fuse the first modal feature vector and the second modal feature vector to obtain a fusion feature vector, and modal alignment representation is performed based on the fusion feature vector to obtain a training modal alignment representation vector.

[0151] Specifically, the server fuses the first modal feature vector and the second modal feature vector through the initial modal alignment model to obtain a fusion feature vector, where the fusion can be directly splicing the first modal feature vector and the second modal feature vector, or performing vector operation on the first modal feature vector and the second modal feature vector. Then, the fusion feature vector is used for modal alignment representation to obtain a training modal alignment representation vector.

[0152] In the above embodiment, after feature extraction of the modal information by the initial modal alignment model, fusion is performed to obtain a fusion feature vector, and then modal alignment representation is directly performed to obtain a training modal alignment representation vector, so as to improve the efficiency of obtaining the training modal alignment representation vector.

[0153] In one embodiment, step 206, that is, calculating the similarity between the first modal alignment representation vector and the second modal alignment representation vector to obtain a modal similarity, includes the steps of:

[0154] The cosine distance between the first modality alignment representation vector and the second modality alignment representation vector is calculated to obtain the modality similarity degree.

[0155] Specifically, the server can use a cosine distance algorithm to calculate the similarity between the first modality alignment representation vector and the second modality alignment representation vector to obtain the modality similarity degree. The modality similarity degree can measure the cost of converting from one modality to another modality.

[0156] In one embodiment, the first modality alignment representation vector and the second modality alignment representation vector can be normalized to obtain a normalized first modality alignment representation vector and a normalized second modality alignment representation vector, and then the Euclidean distance between the normalized first modality alignment representation vector and the normalized second modality alignment representation vector is calculated to obtain the modality similarity degree. For example, the modality similarity degree can be calculated using formula (1) as shown below.

[0157]

[0158] wherein w i denotes the first modality alignment representation vector, V j denotes the second modality alignment representation vector, and c denotes the modality similarity degree.

[0159] In the above embodiment, by using the cosine distance to calculate the similarity between the first modality alignment representation vector and the second modality alignment representation vector, the modality similarity degree is obtained, which improves the accuracy of the modality similarity degree.

[0160] In one embodiment, step 208, i.e., calculating the probability distribution distance between the first modality alignment representation vector and the second modality alignment representation vector based on the modality similarity degree, obtaining the vector loss information based on the probability distribution distance, includes the steps of:

[0161] Obtaining target probability distribution conversion parameter information; calculating the product of the target probability distribution conversion parameter information and the modality similarity degree to obtain the probability distribution distance between the first modality alignment representation vector and the second modality alignment representation vector; and taking the probability distribution distance as the vector loss information.

[0162] The target probability distribution conversion parameter information refers to the minimum transmission quality required when converting the first modality information into the second modality information, i.e., the optimal transmission distance. For example, it can be a conversion parameter matrix between modalities. The target probability distribution conversion parameter information is used to represent the minimum value of the average distance required when moving data from one distribution to another distribution, i.e., it refers to the minimum consumption under the optimal movement path.

[0163] Specifically, the server obtains initialized probability distribution conversion parameter information, which can be a unit matrix. Then the initialized probability distribution conversion parameter information is iteratively optimized and calculated to obtain target probability distribution conversion parameter information, and then the target probability distribution conversion parameter information is multiplied by the modal similarity degree to obtain the probability distribution distance between the first modal alignment representation vector and the second modal alignment representation vector, and then the probability distribution distance is directly taken as the vector loss information.

[0164] In one specific embodiment, the probability distribution distance is calculated using a probability distribution distance measurement algorithm, which includes but is not limited to KL (Kullback-Leibler divergence, which can measure the distance between two random distributions) divergence, JS (Jensen-Shannon divergence, which measures the similarity of two probability distributions) divergence, and Wasserstein (Bulldozer distance) distance algorithm, etc. As shown in FIG. 2, a process diagram of optimal probability distribution conversion is shown. Wherein, one probability distribution is converted into another probability distribution using target probability distribution conversion parameter information, Figure 9

[0165] In one specific embodiment, the loss information can be calculated using the formula (2) shown below as a loss function.

[0166]

[0167] Wherein, L wpa is the target loss function, through which the minimum loss information is obtained by iterative optimization. μ represents the first probability distribution, and v represents the second probability distribution. D ot is the Wasserstein distance between the first probability distribution and the second probability distribution. T is the target probability distribution conversion parameter information, represented by a transport matrix. a and b are parameters for iterative calculation of T. a and b can be calculated by the initial probability distribution conversion parameter information, the value of the first probability distribution and the value of the second probability distribution.

[0168] In one embodiment, obtaining the target probability distribution conversion parameter information includes the steps of:

[0169] Obtaining initial probability distribution conversion parameter information, first probability distribution information corresponding to the first modal alignment representation vector, and second probability distribution information corresponding to the second modal alignment representation vector.

[0170] ​The initial probability distribution conversion parameter information refers to the initialized probability distribution conversion parameter information, which can be a unit matrix. The first probability distribution information refers to the value of the discrete distribution to which the first modality alignment representation vector belongs as a sample, and the second probability distribution information refers to the value of the discrete distribution to which the second modality alignment representation vector belongs as a sample. The first probability distribution and the second probability distribution respectively satisfy corresponding constraint conditions. The constraint condition of the first probability distribution can be that the sample sum of the first probability distribution converted to the second probability distribution is the same as the original sample sum of the first probability distribution. The constraint condition of the first probability distribution can be that the sample sum of the second probability distribution obtained after conversion is the same as the original sample sum of the first probability distribution.

[0171] Specifically, the server obtains initial probability distribution conversion parameter information, first probability distribution information corresponding to the first modality alignment representation vector, and second probability distribution information corresponding to the second modality alignment representation vector.

[0172] Further, the initial probability distribution conversion parameter information is iteratively calculated based on the initial probability distribution conversion parameter information, the modality similarity degree, the first probability distribution information, and the second probability distribution information. When a preset iterative calculation completion condition is reached, target probability distribution conversion parameter information is obtained.

[0173] Specifically, the server performs an exponential operation with a natural constant as the base using the modality similarity degree to obtain an exponential operation result, then performs a product operation using the initial probability distribution conversion parameter information and the exponential operation result to obtain a product operation result, then obtains a preset unit parameter matrix, calculates the product of the preset unit matrix and the product operation result, and then calculates the ratio of the first probability distribution information and the product to obtain a first matrix parameter. Then, the product of the first matrix parameter and the transpose of the product operation result is calculated, and the ratio of the second probability distribution information and the product is calculated to obtain a second matrix parameter. The product operation result, the first matrix parameter, and the second matrix parameter are used to perform matrix multiplication to obtain a target matrix, and then a vector formed by the diagonal elements in the target matrix is obtained as the probability distribution conversion parameter information obtained by the first iteration. Alternatively, diagonal elements in the first matrix parameter and the second matrix parameter can be obtained as a first vector and a second vector, and then the product of the first vector, the second vector, and the product operation result is obtained to obtain the probability distribution conversion parameter information of the first iteration. Then, the probability distribution conversion parameter information is used as the initial probability distribution conversion parameter information and is iterated repeatedly. When a preset iterative calculation completion condition is reached, target probability distribution conversion parameter information is obtained. The preset iterative calculation completion condition refers to a preset upper limit of the number of iterations.

[0174] In a specific embodiment, the IPOT (Inexact Proximal point method for Optimal Transport, an approximate solution algorithm for optimal transport) algorithm can be used to calculate the target probability distribution conversion parameter information, and Sinkhorn (an iterative solution algorithm for optimal transport) and other numerical analysis algorithms based on regular constraints can be used to calculate the target probability distribution conversion parameter information. In a specific embodiment, a third-party library can be used to calculate the target probability distribution conversion parameter information.

[0175] In a specific embodiment, iterative calculation can be performed using formula (3) as shown below to finally obtain the target probability distribution conversion parameter information.

[0176]

[0177] wherein T t+1 represents the target probability distribution conversion parameter information, which is obtained after the t+1 iteration. T (t) is the initial probability distribution conversion parameter information. t is a positive integer. β is a preset value, which can be 0.5. C is the modal similarity degree. T is the probability distribution conversion parameter information to be optimized. B is the Bregman divergence.

[0178] In the above embodiment, the initial probability distribution conversion parameter information is iteratively calculated by using the initial probability distribution conversion parameter information, the modal similarity degree, the first probability distribution information and the second probability distribution information. When the preset iteration calculation completion condition is reached, the target probability distribution conversion parameter information is obtained, and the accuracy of the obtained target probability distribution conversion parameter information is improved.

[0179] In an embodiment, as shown in Figure 10 , a modal alignment method is provided. Taking a server in Figure 1 as an example, it can be understood that the method can also be applied to the server, and can also be applied to a system including a terminal and a server, and is realized through the interaction of the terminal and the server. In this embodiment, the modal alignment model training method includes the following steps:

[0180] Step 902, obtaining first and second to-be-aligned modal information.

[0181] Specifically, the first to-be-aligned modal information refers to modal information that needs to be aligned with the second to-be-aligned modal information. The first to-be-aligned modal information can be text information, picture information, voice information, video information, etc. The second to-be-aligned modal information refers to modal information that needs to be aligned with the first to-be-aligned modal information. The first to-be-aligned modal information can also be text information, picture information, voice information, video information, etc. The first to-be-aligned modal information and the second to-be-aligned modal information are information of different modalities. The server can obtain the first to-be-aligned modal information and the second to-be-aligned modal information from a database, can obtain the first to-be-aligned modal information and the second to-be-aligned modal information uploaded by a terminal, or can obtain the first to-be-aligned modal information and the second to-be-aligned modal information from a business party.

[0182] In step 904, the first to-be-aligned modal information and the second to-be-aligned modal information are input into the first target modal alignment model. The first training modal information and the second training modal information are input into the initial modal alignment model to perform modal alignment representation, to obtain a training modal alignment representation vector. The training modal alignment representation vector includes a first modal alignment representation vector corresponding to the first training modal information and a second modal alignment representation vector corresponding to the second training modal information. The similarity between the first modal alignment representation vector and the second modal alignment representation vector is calculated to obtain a modal similarity. The probability distribution distance of the first modal alignment representation vector and the second modal alignment representation vector is calculated based on the modal similarity. The vector loss information is obtained based on the probability distribution distance. The initial modal alignment model is iteratively updated based on the vector loss information. When the alignment model training completion condition is reached, the first target modal alignment model is obtained.

[0183] Specifically, the server can call the pre-trained first target modal alignment model to perform modal alignment on the first to-be-aligned modal information and the second to-be-aligned modal information, that is, the server inputs the first to-be-aligned modal information and the second to-be-aligned modal information into the first target modal alignment model. The first training modal information and the second training modal information are input into the initial modal alignment model to perform modal alignment representation, to obtain a training modal alignment representation vector. The training modal alignment representation vector includes a first modal alignment representation vector corresponding to the first training modal information and a second modal alignment representation vector corresponding to the second training modal information. The similarity between the first modal alignment representation vector and the second modal alignment representation vector is calculated to obtain a modal similarity. The probability distribution distance of the first modal alignment representation vector and the second modal alignment representation vector is calculated based on the modal similarity. The vector loss information is obtained based on the probability distribution distance. The initial modal alignment model is iteratively updated based on the vector loss information. When the alignment model training completion condition is reached, the first target modal alignment model is obtained.

[0184] In an embodiment, the server can input the first to-be-aligned modal information and the second to-be-aligned modal information into the modal alignment model trained using any embodiment of the above modal alignment model training method for modal alignment representation. For example, the server can input the first to-be-aligned modal information and the second to-be-aligned modal information into the second target modal alignment model for modal alignment representation.

[0185] At step 906, the first to-be-aligned modal information and the second to-be-aligned modal information are subjected to modal alignment representation by the first target modal alignment model to obtain a target modal alignment representation vector.

[0186] Specifically, the server subjects the first to-be-aligned modal information and the second to-be-aligned modal information to modal alignment representation by the modal alignment representation parameters in the first target modal alignment model to obtain an output target modal alignment representation vector. The modal alignment representation parameters refer to the trained model parameters in the first target modal alignment model. The target modal alignment representation vector can include a modal alignment representation vector corresponding to the first to-be-aligned modal information and a modal alignment representation vector corresponding to the second to-be-aligned modal information. The target modal alignment representation vector can then be used for downstream task processing, such as a task of classifying and identifying modal information or a task of making recommendations based on modal information.

[0187] In an embodiment, the server can first extract features from the first to-be-aligned modal information and the second to-be-aligned modal information to obtain a first to-be-aligned feature vector and a second to-be-aligned feature vector, then concatenate the first to-be-aligned feature vector and the second to-be-aligned feature vector to obtain a concatenated vector, and then input the concatenated vector into the first target modal alignment model for modal alignment representation to obtain a target modal alignment representation vector.

[0188] The above modal alignment method obtains a target modal alignment representation vector by subjecting the first to-be-aligned modal information and the second to-be-aligned modal information to modal alignment representation by the first target modal alignment model. The first target modal alignment model is obtained by calculating the similarity between the first modal alignment representation vector and the second modal alignment representation vector, obtaining the modal similarity, calculating the probability distribution distance between the first modal alignment representation vector and the second modal alignment representation vector based on the modal similarity, obtaining the vector loss information based on the probability distribution distance, and iteratively training based on the vector loss information. Therefore, the obtained target modal alignment representation vector is more accurate, and the accuracy of modal alignment is improved.

[0189] In an embodiment, as shown in FIG. 10, the modal alignment method further includes: Figure 11

[0190] ​At step 1102, an initial multi-modal information classification and recognition model is established based on the first target modality alignment model.

[0191] The initial multi-modal information classification and recognition model refers to an initial multi-modal information classification and recognition model, which is used for classification and recognition using multiple modal information to recognize the categories in the modal information.

[0192] Specifically, the server can use the first target modality alignment model as a pre-training model, and then add a fully connected classification network based on the network structure of the first target modality alignment model to obtain the initial multi-modal information classification and recognition model. Then, the initial multi-modal information classification and recognition model is fine-tuned using training data for classification and recognition.

[0193] At step 1102, a multi-modal training sample and a corresponding classification and recognition label are obtained.

[0194] The multi-modal training sample refers to a training sample including different modal information, which is a training sample used for multi-modal classification and recognition. The multi-modal training sample includes two modal information. When more than two modal information needs to be classified and recognized, it can be split into two modal information combinations for classification and recognition, and finally the classification and recognition results are combined to obtain the classification and recognition results of more than two modal information. The classification and recognition label refers to the category label corresponding to the modal information in the multi-modal training sample, which is a label used for training of the classification and recognition task.

[0195] Specifically, the server can obtain the multi-modal training sample and the corresponding classification and recognition label from the database, or obtain the multi-modal training sample and the corresponding classification and recognition label from a service provider providing data services, or obtain the multi-modal training sample and the corresponding classification and recognition label uploaded by the terminal.

[0196] At step 1102, the multi-modal training sample is input into the initial multi-modal information classification and recognition model, the multi-modal training sample is represented by the initial multi-modal information classification and recognition model, a training modal alignment representation vector is obtained, multi-modal classification and recognition is performed based on the training modal alignment representation vector, and an initial classification and recognition result is obtained.

[0197] Specifically, the server inputs the multi-modal training sample into the initial multi-modal information classification and recognition model, performs modal alignment representation on the multi-modal training sample through the modal alignment representation parameter in the initial multi-modal information classification and recognition model, obtains a training modal alignment representation vector, performs multi-modal classification and recognition based on the training modal alignment representation vector through the initial classification and recognition parameter, and obtains an initial classification and recognition result. The initial classification and recognition parameter refers to an initialized classification and recognition parameter. The modal alignment representation parameter refers to a model parameter in the first target modal alignment model.

[0198] In step 1102, multi-modal classification and recognition loss calculation is performed based on the initial classification and recognition result and the classification and recognition label, and multi-modal classification and recognition loss information is obtained.

[0199] Specifically, the server uses a classification loss function to calculate the error between the initial classification and recognition result and the classification and recognition label, and obtains the multi-modal classification and recognition loss information. The classification loss function can use a cross-entropy loss function.

[0200] In step 1102, the initial multi-modal information classification and recognition model is updated in reverse based on the multi-modal classification and recognition loss information and is iterated cyclically, and a target multi-modal classification and recognition model is obtained.

[0201] Specifically, the server uses a gradient descent algorithm to update the initial multi-modal information classification and recognition model in reverse based on the multi-modal classification and recognition loss information, obtains an updated multi-modal information classification and recognition model, then takes the updated multi-modal information classification and recognition model as the initial multi-modal information classification and recognition model, and returns to the step of obtaining the multi-modal training sample and the corresponding classification and recognition label for execution, until a preset training completion condition is reached, and a target multi-modal classification and recognition model is obtained.

[0202] In the above embodiment, the initial multi-modal information classification and recognition model is established by using the first target modal alignment model, and then the initial multi-modal information classification and recognition model is trained to obtain the target multi-modal classification and recognition model, which can improve the accuracy of classification and recognition.

[0203] In one embodiment, after step 1102, that is, after obtaining the target multi-modal recognition task model, the following steps are further included:

[0204] Each of the to-be-classified and recognized modal information is obtained; each of the to-be-classified and recognized modal information is input into the target multi-modal recognition task model; each of the to-be-classified and recognized modal information is subjected to modal alignment representation through the target multi-modal classification and recognition model, and a target modal alignment representation vector is obtained; and each of the to-be-classified and recognized modal information is subjected to multi-modal classification and recognition through the target multi-modal classification and recognition model, and a classification and recognition result corresponding to each of the to-be-classified and recognized modal information is obtained.

[0205] The to-be-classified and identified modal information refers to modal information that needs to be classified and identified. Each to-be-classified and identified modal information includes information of at least two different modalities.

[0206] Specifically, the server obtains each to-be-classified and identified modal information, which includes information of two different modalities, that is, two to-be-classified and identified modal information is obtained. At this time, the two to-be-classified and identified modal information is directly input into the target multi-modal identification task model, the modal alignment representation of each to-be-classified and identified modal information is performed through the target multi-modal classification and identification model, the target modal alignment representation vector is obtained, and the multi-modal classification and identification of the target modal alignment representation vector is performed through the target multi-modal classification and identification model, and the classification and identification result corresponding to the two to-be-classified and identified modal information is obtained.

[0207] When there are more than two different modalities of information in each to-be-classified and identified modal information, the to-be-classified and identified modal information is combined two by two to obtain the combined to-be-classified and identified modal information. Then the combined to-be-classified and identified modal information is input into the target multi-modal identification task model for classification and identification, and the classification and identification result output by each combined to-be-classified and identified modal information is obtained. Finally, the classification and identification result corresponding to the more than two different modalities of information is obtained by fusing the classification and identification result output by each combined to-be-classified and identified modal information.

[0208] In the above embodiment, the target multi-modal identification task model is used to classify and identify the to-be-classified and identified modal information, and the classification and identification result corresponding to the to-be-classified and identified modal information is obtained, thereby improving the accuracy of classification and identification.

[0209] In one specific embodiment, as shown in FIG. 1, a flowchart of a modal alignment model training method is shown, which includes the following steps: Figure 12

[0210] Step 1202, obtaining first training modal information and second training modal information and modal alignment label. The first training modal information and the second training modal information are respectively subjected to feature extraction to obtain a first modal feature vector and a second modal feature vector; and the first modal feature vector and the second modal feature vector are fused to obtain a fused feature vector.

[0211] Step 1204, inputting the fused feature vector into the initial modal alignment classification and identification model for modal alignment representation to obtain a training modal alignment representation vector, and performing modal alignment classification and identification based on the training modal alignment representation vector to obtain a modal alignment classification and identification result. The training modal alignment representation vector includes a first modal alignment representation vector corresponding to the first training modal information and a second modal alignment representation vector corresponding to the second training modal information.​

[0212] In step 1206, the similarity between the first modality alignment representation vector and the second modality alignment representation vector is calculated to obtain a modality similarity, and initial probability distribution conversion parameter information, first probability distribution information corresponding to the first modality alignment representation vector, and second probability distribution information corresponding to the second modality alignment representation vector are obtained.

[0213] In step 1208, the initial probability distribution conversion parameter information is iteratively calculated based on the initial probability distribution conversion parameter information, the modality similarity, the first probability distribution information, and the second probability distribution information, and when a preset iteration calculation completion condition is reached, target probability distribution conversion parameter information is obtained.

[0214] In step 1210, the product of the target probability distribution conversion parameter information and the modality similarity is calculated to obtain the probability distribution distance between the first modality alignment representation vector and the second modality alignment representation vector, and the probability distribution distance is taken as vector loss information. Classification loss information is calculated based on the modality alignment classification recognition result and the modality alignment label to obtain classification loss information, and model loss information is calculated based on the classification loss information and the vector loss information to obtain model loss information.

[0215] In step 1212, the initial modality alignment classification recognition model is updated based on the model loss information to obtain an updated modality alignment classification recognition model, the updated modality alignment classification recognition model is taken as the initial modality alignment classification recognition model, and the step of obtaining the first training modality information and the second training modality information is executed until a classification model training completion condition is reached to obtain a target modality alignment classification recognition model.

[0216] In step 1214, a second target modality alignment model is obtained based on the target modality alignment classification recognition model.

[0217] In one specific embodiment, the modality alignment method is applied in an e-commerce platform, a merchant uploads product information to be listed, the product information includes a product with a title of "new fashion waistline belt hooded shirt" and an original product sketch. Then the category of the product information is to be identified, at this time, the platform server inputs the title text and the original product sketch into the target modality alignment classification recognition model for modality alignment and classification recognition to obtain the identified clothing category. Among them, the target modality alignment classification recognition model improves the corresponding relationship between the clothing vectors in the text representation vector and the image representation vector. Then the text representation vector and the image representation vector are used for classification recognition to obtain the identified clothing category, which improves the accuracy of classification recognition. For example Figure 13As shown, it is a modal alignment effect schematic diagram, wherein the left picture is a commodity picture, the middle picture is the alignment effect before modal alignment, and the right picture is the alignment effect after the alignment of the present scheme. It can be seen that the alignment effect of the clothes commodity in the image and the clothes in the text description is better in the present application, that is, the accuracy of the modal alignment of the present application is higher, and the alignment result is more accurate and clear, thereby making the accuracy of subsequent task processing higher.

[0218] It should be understood that, although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, these steps are not necessarily executed in sequence according to the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowcharts involved in the embodiments described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0219] Based on the same inventive concept, the present application also provides a modal alignment model training device for implementing the above-mentioned modal alignment model training method and a modal alignment device for implementing the modal alignment method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more modal alignment model training device or modal alignment device embodiments provided below can refer to the limitations of the modal alignment model training method or the modal alignment method described above, which will not be repeated here.

[0220] In one embodiment, as Figure 14 shown, a modal alignment model training device 1400 is provided, comprising: an information acquisition module 1402, an initial alignment module 1404, a similarity calculation module 1406, a loss calculation module 1408 and an iteration module 1410, wherein:

[0221] The information acquisition module 1402 is configured to acquire first training modal information and second training modal information;

[0222] The initial alignment module 1404 is configured to input the first training modal information and the second training modal information into an initial modal alignment model for modal alignment representation, to obtain a training modal alignment representation vector, the training modal alignment representation vector comprising a first modal alignment representation vector corresponding to the first training modal information and a second modal alignment representation vector corresponding to the second training modal information, the first modal alignment representation vector and the second modal alignment representation vector having an initial corresponding relationship with the same instance representation.

[0223] The similarity calculation module 1406 is configured to calculate a similarity between the first modality alignment representation vector and the second modality alignment representation vector, to obtain a modality similarity.

[0224] The loss calculation module 1408 is configured to calculate a probability distribution distance between the first modality alignment representation vector and the second modality alignment representation vector based on the modality similarity, and obtain vector loss information based on the probability distribution distance.

[0225] The iteration module 1410 is configured to update the initial modality alignment model reversely based on the vector loss information, to obtain an updated modality alignment model, take the updated modality alignment model as the initial modality alignment model, and return to the step of obtaining the first training modality information and the second training modality information for execution until a modality alignment model training completion condition is reached, to obtain a first target modality alignment model. The first target modality alignment model is used to extract semantic representations of different modality information, and semantic representations of the same instance in the semantic representations of different modality information have a corresponding relationship.

[0226] In an embodiment, the modality alignment model training apparatus 1400 further includes:

[0227] The label acquisition module is configured to acquire modality alignment labels corresponding to the first training modality information and the second training modality information.

[0228] The classification module is configured to input the first training modality information and the second training modality information into the initial modality alignment classification and recognition model to perform modality alignment representation, to obtain a training modality alignment representation vector, and perform modality alignment classification and recognition based on the training modality alignment representation vector, to obtain a modality alignment classification and recognition result.

[0229] The classification loss calculation module is configured to perform classification loss calculation based on the modality alignment classification and recognition result and the modality alignment label, to obtain classification loss information.

[0230] The model loss obtaining module is configured to perform model loss information based on the classification loss information and the vector loss information, to obtain model loss information.

[0231] The classification model iteration module is configured to update the initial modality alignment classification and recognition model reversely based on the model loss information, to obtain an updated modality alignment classification and recognition model, take the updated modality alignment classification and recognition model as the initial modality alignment classification and recognition model, and return to the step of obtaining the first training modality information and the second training modality information for execution until a classification model training completion condition is reached, to obtain a target modality alignment classification and recognition model.

[0232] The second model obtaining module is configured to obtain a second target modality alignment model based on the target modality alignment classification and recognition model.

[0233] In an embodiment, the initial modal alignment classification identification model comprises an initial modal alignment representation network and an initial classification network.

[0234] The classification module is further configured to perform modal alignment representation on the first training modal information and the second training modal information by the initial modal alignment representation network in the initial modal alignment classification identification model, to obtain a training modal alignment representation vector; and perform modal alignment classification identification by the initial classification network in the initial modal alignment classification identification model, to obtain a modal alignment classification identification result.

[0235] The second model obtaining module is further configured to take the target modal alignment representation network in the target modal alignment classification identification model as a second target modal alignment model.

[0236] In an embodiment, the initial alignment module 1404 is further configured to perform feature extraction on the first training modal information and the second training modal information respectively, to obtain a first modal feature vector and a second modal feature vector; fuse the first modal feature vector and the second modal feature vector, to obtain a fused feature vector; and input the fused feature vector into the initial modal alignment model for modal alignment representation, to obtain a training modal alignment representation vector.

[0237] In an embodiment, the first training modal information comprises text information, and the second training modal information comprises picture information.

[0238] The initial alignment module 1404 is further configured to input the text information into a text feature extraction model, to obtain a text global representation vector and a text character representation vector by the text feature extraction model, and take the text character representation vector as the first modal feature vector; and input the picture information into a picture feature extraction model, to obtain a picture global representation vector and a picture content representation vector by the picture feature extraction model, and take the picture content representation vector as the second modal feature vector.

[0239] In an embodiment, the initial alignment module 1404 is further configured to input the first training modal information and the second training modal information into the initial modal alignment model; perform feature extraction on the first training modal information and the second training modal information respectively by the initial modal alignment model, to obtain a first modal feature vector and a second modal feature vector; fuse the first modal feature vector and the second modal feature vector by the initial modal alignment model, to obtain a fused feature vector, and perform modal alignment representation based on the fused feature vector, to obtain a training modal alignment representation vector.

[0240] In an embodiment, the similarity calculation module 1406 is further configured to calculate a cosine distance between the first modal alignment representation vector and the second modal alignment representation vector, to obtain a modal similarity degree.

[0241] In one embodiment, the loss calculation module 1408 is further configured to obtain target probability distribution conversion parameter information; calculate a product of the target probability distribution conversion parameter information and the modality similarity degree, to obtain a probability distribution distance between the first modality alignment representation vector and the second modality alignment representation vector; and take the probability distribution distance as the vector loss information.

[0242] In one embodiment, the loss calculation module 1408 is further configured to obtain initial probability distribution conversion parameter information, first probability distribution information corresponding to the first modality alignment representation vector, and second probability distribution information corresponding to the second modality alignment representation vector; perform iterative calculation on the initial probability distribution conversion parameter information based on the initial probability distribution conversion parameter information, the modality similarity degree, the first probability distribution information, and the second probability distribution information; and when a preset iterative calculation completion condition is reached, obtain the target probability distribution conversion parameter information.

[0243] In one embodiment, as shown in FIG. 13, a modality alignment device 1300 is provided, which includes a to-be-aligned information obtaining module 1302, an input module 1304, and a modality alignment module 1306, wherein: Figure 15

[0244] The to-be-aligned information obtaining module 1302 is configured to obtain first to-be-aligned modality information and second to-be-aligned modality information.

[0245] The input module 1304 is configured to input the first to-be-aligned modality information and the second to-be-aligned modality information into a first target modality alignment model; wherein by inputting first training modality information and second training modality information into an initial modality alignment model to perform modality alignment representation, a training modality alignment representation vector is obtained, the training modality alignment representation vector includes a first modality alignment representation vector corresponding to the first training modality information and a second modality alignment representation vector corresponding to the second training modality information, a similarity degree between the first modality alignment representation vector and the second modality alignment representation vector is calculated, to obtain a modality similarity degree; a probability distribution distance between the first modality alignment representation vector and the second modality alignment representation vector is calculated based on the modality similarity degree, vector loss information is obtained based on the probability distribution distance, and the initial modality alignment model is iteratively updated based on the vector loss information, until an alignment model training completion condition is reached, to obtain the first target modality alignment model.

[0246] The modality alignment module 1306 is configured to perform modality alignment representation on the first to-be-aligned modality information and the second to-be-aligned modality information through the first target modality alignment model, to obtain a target modality alignment representation vector.

[0247] In one embodiment, the modality alignment device 1300 further includes: ​

[0248] The classification recognition model training module is configured to establish an initial multi-modal information classification recognition model based on the first target modal alignment model, acquire multi-modal training samples and corresponding classification recognition labels, input the multi-modal training samples into the initial multi-modal information classification recognition model, perform modal alignment representation on the multi-modal training samples by using the initial multi-modal information classification recognition model to obtain training modal alignment representation vectors, perform multi-modal classification recognition based on the training modal alignment representation vectors to obtain initial classification recognition results, perform multi-modal classification recognition loss calculation based on the initial classification recognition results and the classification recognition labels to obtain multi-modal classification recognition loss information, and update the initial multi-modal information classification recognition model based on the multi-modal classification recognition loss information and perform cyclic iteration to obtain a target multi-modal classification recognition model.

[0249] In an embodiment, the modal alignment device 1500 further includes:

[0250] The classification recognition module is configured to acquire each piece of to-be-classified modal information, input each piece of to-be-classified modal information into the target multi-modal classification recognition model, perform modal alignment representation on each piece of to-be-classified modal information by using the target multi-modal classification recognition model to obtain target modal alignment representation vectors, and perform multi-modal classification recognition on the target modal alignment representation vectors by using the target multi-modal classification recognition model to obtain classification recognition results corresponding to each piece of to-be-classified modal information.

[0251] Each module in the modal alignment model training device or the modal alignment device described above can be realized in whole or in part by software, hardware, or a combination thereof. Each module described above can be embedded in or independent of a processor in a computer device in a hardware form, or can be stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0252] In an embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in FIG. 8. Figure 16As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control ability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store various modal data used during training. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a modal alignment model training method or a modal alignment method.

[0253] In one embodiment, a computer device is provided, which can be a terminal, and its internal structure diagram can be as shown in the figure. Figure 17 As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control ability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store various modal data used during training. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a modal alignment model training method or a modal alignment method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or can be a key, trackball or touchpad arranged on the shell of the computer device, or can be an external keyboard, touchpad or mouse, etc.

[0254] Those skilled in the art can understand that, Figure 16Or the structure shown in 17, only the block diagram of part of the structure related to the scheme of the present application, does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0255] In one embodiment, a computer device is also provided, including a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0256] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in the above method embodiments.

[0257] In one embodiment, a computer program product is provided, which includes a computer program, and the computer program is executed by a processor to implement the steps in the above method embodiments.

[0258] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant national and regional laws, regulations and standards.

[0259] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0260] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0261] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for training a modal alignment model, the method comprising: The method comprises: obtaining first training modal information and second training modal information, the first training modal information and the second training modal information being information of different modalities used during training, and the modal information comprising any one of text information, image information, voice information and video information; inputting the first training modal information and the second training modal information into an initial modal alignment model to perform modal alignment representation, to obtain a training modal alignment representation vector, the training modal alignment representation vector comprising a first modal alignment representation vector corresponding to the first training modal information and a second modal alignment representation vector corresponding to the second training modal information, and the first modal alignment representation vector and the second modal alignment representation vector having an initial corresponding relationship for the same instance representation; calculating the similarity between the first modal alignment representation vector and the second modal alignment representation vector to obtain a modal similarity; calculating the probability distribution distance of the first modal alignment representation vector and the second modal alignment representation vector based on the modal similarity, obtaining vector loss information based on the probability distribution distance, comprising: obtaining target probability distribution conversion parameter information; calculating the product of the target probability distribution conversion parameter information and the modal similarity to obtain the probability distribution distance of the first modal alignment representation vector and the second modal alignment representation vector; and taking the probability distribution distance as the vector loss information; updating the initial modal alignment model based on the vector loss information to obtain an updated modal alignment model, taking the updated modal alignment model as the initial modal alignment model, and returning to the step of obtaining the first training modal information and the second training modal information for execution until a modal alignment model training completion condition is reached, to obtain a first target modal alignment model, the first target modal alignment model being used to extract semantic representations of different modal information, and the semantic representations of the same instances in the semantic representations of the different modal information having a corresponding relationship.

2. The method of claim 1, wherein, The method further comprises: obtaining modal alignment labels corresponding to the first training modal information and the second training modal information; inputting the first training modal information and the second training modal information into an initial modal alignment classification and identification model to perform modal alignment representation, to obtain the training modal alignment representation vector, and performing modal alignment classification and identification based on the training modal alignment representation vector to obtain a modal alignment classification and identification result; performing classification loss calculation based on the modal alignment classification and identification result and the modal alignment labels to obtain classification loss information; performing model loss information based on the classification loss information and the vector loss information to obtain model loss information; updating the initial modal alignment classification and identification model based on the model loss information to obtain an updated modal alignment classification and identification model, taking the updated modal alignment classification and identification model as the initial modal alignment classification and identification model, and returning to the step of obtaining the first training modal information and the second training modal information for execution until a classification model training completion condition is reached, to obtain a target modal alignment classification and identification model; The target modal alignment classification identification model is used to obtain a second target modal alignment model.

3. The method of claim 2, wherein, The initial modal alignment classification identification model comprises an initial modal alignment representation network and an initial classification network. The initial modal alignment classification identification model is used to obtain a training modal alignment representation vector and a modal alignment classification identification result, including: The initial modal alignment representation network in the initial modal alignment classification identification model is used to perform modal alignment representation on the first training modal information and the second training modal information to obtain the training modal alignment representation vector. The initial classification network in the initial modal alignment classification identification model is used to perform modal alignment classification identification to obtain the modal alignment classification identification result. The target modal alignment classification identification model is used to obtain a second target modal alignment model, including: The target modal alignment representation network in the target modal alignment classification identification model is used as the second target modal alignment model.

4. The method of claim 1, wherein, The initial modal alignment classification identification model is used to obtain a training modal alignment representation vector, including: The first training modal information and the second training modal information are respectively subjected to feature extraction to obtain a first modal feature vector and a second modal feature vector. The first modal feature vector and the second modal feature vector are fused to obtain a fused feature vector. The fused feature vector is input into the initial modal alignment model to perform modal alignment representation to obtain the training modal alignment representation vector.

5. The method of claim 4, wherein, The first training modal information comprises text information, and the second training modal information comprises picture information. The first training modal information and the second training modal information are respectively subjected to feature extraction to obtain a first modal feature vector and a second modal feature vector, including: The text information is input into a text feature extraction model, and a text global representation vector and a text character representation vector are obtained through the text feature extraction model, wherein the text character representation vector is used as the first modal feature vector. The picture information is input into a picture feature extraction model, and a picture global representation vector and a picture content representation vector are obtained through the picture feature extraction model, wherein the picture content representation vector is used as the second modal feature vector.

6. The method of claim 1, wherein, The initial modal alignment classification identification model is used to obtain a training modal alignment representation vector, including: The first training modal information and the second training modal information are input into the initial modal alignment model. The initial modal alignment model is used to respectively perform feature extraction on the first training modal information and the second training modal information to obtain a first modal feature vector and a second modal feature vector. The first modality feature vector and the second modality feature vector are fused through the initial modality alignment model to obtain a fused feature vector, and modality alignment representation is performed based on the fused feature vector to obtain the training modality alignment representation vector.

7. The method of claim 1, wherein, The similarity between the first modality alignment representation vector and the second modality alignment representation vector is calculated to obtain a modality similarity, including: The cosine distance between the first modality alignment representation vector and the second modality alignment representation vector is calculated to obtain the modality similarity.

8. The method of claim 1, wherein, The target probability distribution conversion parameter information is obtained, including: The initial probability distribution conversion parameter information, the first probability distribution information corresponding to the first modality alignment representation vector, and the second probability distribution information corresponding to the second modality alignment representation vector are obtained; The initial probability distribution conversion parameter information is iteratively calculated based on the initial probability distribution conversion parameter information, the modality similarity, the first probability distribution information, and the second probability distribution information, and when a preset iterative calculation completion condition is reached, the target probability distribution conversion parameter information is obtained.

9. A method of modal alignment, characterized by, The method includes: Obtaining first and second to-be-aligned modality information; The first and second to-be-aligned modality information is input into a first target modality alignment model; Wherein, by inputting first and second training modality information into an initial modality alignment model to perform modality alignment representation, a training modality alignment representation vector is obtained, the training modality alignment representation vector includes a first modality alignment representation vector corresponding to the first training modality information and a second modality alignment representation vector corresponding to the second training modality information, the similarity between the first modality alignment representation vector and the second modality alignment representation vector is calculated to obtain a modality similarity; the probability distribution distance between the first modality alignment representation vector and the second modality alignment representation vector is calculated based on the modality similarity, the vector loss information is obtained based on the probability distribution distance, and the initial modality alignment model is iteratively updated based on the vector loss information until the alignment model training completion condition is reached, obtaining the first target modality alignment model, the first training modality information and the second training modality information are information of different modalities used during training, modality information includes any one of text information, image information, voice information, and video information, including: obtaining target probability distribution conversion parameter information; calculating the product of the target probability distribution conversion parameter information and the modality similarity to obtain the probability distribution distance between the first modality alignment representation vector and the second modality alignment representation vector; the probability distribution distance is taken as the vector loss information; The first and second to-be-aligned modality information is modality alignment represented through the first target modality alignment model to obtain a target modality alignment representation vector.

10. The method of claim 9, wherein, The method further includes: An initial multi-modality information classification identification model is established based on the first target modality alignment model; A multi-modality training sample and a corresponding classification identification label are obtained; inputting the multi-modal training sample into the initial multi-modal information classification and recognition model, performing modal alignment representation on the multi-modal training sample through the initial multi-modal information classification and recognition model to obtain a training modal alignment representation vector, performing multi-modal classification and recognition based on the training modal alignment representation vector to obtain an initial classification and recognition result; performing multi-modal classification and recognition loss calculation based on the initial classification and recognition result and the classification and recognition label to obtain multi-modal classification and recognition loss information; updating the initial multi-modal information classification and recognition model reversely based on the multi-modal classification and recognition loss information and performing cyclic iteration to obtain a target multi-modal classification and recognition model.

11. The method of claim 10, wherein, After the target multi-modal classification and recognition model is obtained, the method further includes: obtaining each to-be-classified and recognized modal information; inputting the each to-be-classified and recognized modal information into the target multi-modal classification and recognition model; performing modal alignment representation on the each to-be-classified and recognized modal information through the target multi-modal classification and recognition model to obtain a target modal alignment representation vector; performing multi-modal classification and recognition on the target modal alignment representation vector through the target multi-modal classification and recognition model to obtain a classification and recognition result corresponding to the each to-be-classified and recognized modal information. 12.A mode alignment model training apparatus, characterized in that, The device includes: an information obtaining module, configured to obtain first training modal information and second training modal information, the first training modal information and the second training modal information being information of different modalities used during training, and the modal information including any one of text information, image information, voice information and video information; an initial alignment module, configured to input the first training modal information and the second training modal information into an initial modal alignment model to perform modal alignment representation, and obtain a training modal alignment representation vector, the training modal alignment representation vector including a first modal alignment representation vector corresponding to the first training modal information and a second modal alignment representation vector corresponding to the second training modal information, and the first modal alignment representation vector and the second modal alignment representation vector having an initial corresponding relationship in representation of the same instance; a similarity calculation module, configured to calculate a similarity degree between the first modal alignment representation vector and the second modal alignment representation vector to obtain a modal similarity degree; a loss calculation module, configured to calculate a probability distribution distance between the first modal alignment representation vector and the second modal alignment representation vector based on the modal similarity degree, and obtain vector loss information based on the probability distribution distance, including: obtaining target probability distribution conversion parameter information; calculating a product of the target probability distribution conversion parameter information and the modal similarity degree to obtain the probability distribution distance between the first modal alignment representation vector and the second modal alignment representation vector; and taking the probability distribution distance as the vector loss information; The iteration module is configured to update the initial modal alignment model based on the vector loss information to obtain an updated modal alignment model, take the updated modal alignment model as the initial modal alignment model, and return to the step of obtaining the first training modal information and the second training modal information until a condition of completing training of the alignment model is reached to obtain a first target modal alignment model, which is used to extract semantic representations of different modal information, and semantic representations of the same instance in the semantic representations of the different modal information have a corresponding relationship.

13. The apparatus of claim 12, wherein, The device further comprises: The label obtaining module is configured to obtain modal alignment labels corresponding to the first training modal information and the second training modal information. The classification module is configured to input the first training modal information and the second training modal information into an initial modal alignment classification and recognition model to perform modal alignment representation to obtain a training modal alignment representation vector, and perform modal alignment classification and recognition based on the training modal alignment representation vector to obtain a modal alignment classification and recognition result. The classification loss calculation module is configured to calculate a classification loss based on the modal alignment classification and recognition result and the modal alignment label to obtain classification loss information. The model loss obtaining module is configured to obtain model loss information based on the classification loss information and the vector loss information. The classification model iteration module is configured to update the initial modal alignment classification and recognition model based on the model loss information to obtain an updated modal alignment classification and recognition model, take the updated modal alignment classification and recognition model as the initial modal alignment classification and recognition model, and return to the step of obtaining the first training modal information and the second training modal information until a condition of completing training of the classification model is reached to obtain a target modal alignment classification and recognition model. The second model obtaining module is configured to obtain a second target modal alignment model based on the target modal alignment classification and recognition model.

14. The apparatus of claim 13, wherein, The initial modal alignment classification and recognition model comprises an initial modal alignment representation network and an initial classification network. The classification module is further configured to perform modal alignment representation on the first training modal information and the second training modal information through the initial modal alignment representation network in the initial modal alignment classification and recognition model to obtain the training modal alignment representation vector. The initial classification network in the initial modal alignment classification and recognition model is used to perform modal alignment classification and recognition to obtain the modal alignment classification and recognition result. The second model obtaining module is further configured to take a target modal alignment representation network in the target modal alignment classification and recognition model as the second target modal alignment model.

15. The apparatus of claim 12, wherein, The initial alignment module is further configured to perform feature extraction on the first training modal information and the second training modal information respectively to obtain a first modal feature vector and a second modal feature vector, fuse the first modal feature vector and the second modal feature vector to obtain a fused feature vector, and input the fused feature vector into the initial modal alignment model to perform modal alignment representation to obtain a training modal alignment representation vector.

16. The apparatus of claim 15, wherein, The first training modality information comprises text information, and the second training modality information comprises picture information. The initial alignment module is further configured to input the text information into a text feature extraction model, obtain a text global representation vector and a text character representation vector through the text feature extraction model, and take the text character representation vector as the first modality feature vector; input the picture information into a picture feature extraction model, obtain a picture global representation vector and a picture content representation vector through the picture feature extraction model, and take the picture content representation vector as the second modality feature vector.

17. The apparatus of claim 12, wherein, The initial alignment module is further configured to input the first training modality information and the second training modality information into an initial modality alignment model; perform feature extraction on the first training modality information and the second training modality information through the initial modality alignment model, respectively, to obtain a first modality feature vector and a second modality feature vector; perform fusion on the first modality feature vector and the second modality feature vector through the initial modality alignment model to obtain a fusion feature vector, perform modality alignment representation based on the fusion feature vector, and obtain the training modality alignment representation vector.

18. The apparatus of claim 12, wherein, The similarity calculation module is further configured to calculate a cosine distance between the first modality alignment representation vector and the second modality alignment representation vector to obtain the modality similarity degree.

19. The apparatus of claim 12, wherein, The loss calculation module is further configured to obtain initial probability distribution conversion parameter information, first probability distribution information corresponding to the first modality alignment representation vector, and second probability distribution information corresponding to the second modality alignment representation vector; perform iterative calculation on the initial probability distribution conversion parameter information based on the initial probability distribution conversion parameter information, the modality similarity degree, the first probability distribution information, and the second probability distribution information; and when a preset iterative calculation completion condition is reached, obtain the target probability distribution conversion parameter information.

20. A modal alignment apparatus, characterized by, The device comprises: a to-be-aligned information acquisition module configured to acquire first to-be-aligned modality information and second to-be-aligned modality information; The input module is configured to input the first to-be-aligned modal information and the second to-be-aligned modal information into the first target modal alignment model; wherein, by inputting first training modal information and second training modal information into an initial modal alignment model for modal alignment representation, a training modal alignment representation vector is obtained, the training modal alignment representation vector includes a first modal alignment representation vector corresponding to the first training modal information and a second modal alignment representation vector corresponding to the second training modal information, a similarity degree between the first modal alignment representation vector and the second modal alignment representation vector is calculated, and a modal similarity degree is obtained; a probability distribution distance of the first modal alignment representation vector and the second modal alignment representation vector is calculated based on the modal similarity degree, a vector loss information is obtained based on the probability distribution distance, the initial modal alignment model is iteratively updated based on the vector loss information, and until an alignment model training completion condition is reached, the first target modal alignment model is obtained, the first training modal information and the second training modal information are information of different modalities used during training, and the modal information includes any one of text information, image information, voice information, and video information, and includes: obtaining target probability distribution conversion parameter information; calculating a product of the target probability distribution conversion parameter information and the modal similarity degree to obtain a probability distribution distance of the first modal alignment representation vector and the second modal alignment representation vector; and taking the probability distribution distance as the vector loss information. The modal alignment module is configured to perform modal alignment representation on the first to-be-aligned modal information and the second to-be-aligned modal information by using the first target modal alignment model to obtain a target modal alignment representation vector.

21. The apparatus of claim 20, wherein, The device further includes: The classification and identification model training module is configured to establish an initial multi-modal information classification and identification model based on the first target modal alignment model; obtain a multi-modal training sample and a corresponding classification and identification label; input the multi-modal training sample into the initial multi-modal information classification and identification model, perform modal alignment representation on the multi-modal training sample by using the initial multi-modal information classification and identification model to obtain a training modal alignment representation vector, perform multi-modal classification and identification based on the training modal alignment representation vector to obtain an initial classification and identification result, perform multi-modal classification and identification loss calculation based on the initial classification and identification result and the classification and identification label to obtain multi-modal classification and identification loss information, and iteratively update the initial multi-modal information classification and identification model based on the multi-modal classification and identification loss information to obtain a target multi-modal classification and identification model.

22. The apparatus of claim 20, wherein, The device further includes: The classification recognition module is configured to: acquire each to-be-classified and recognized modality information; input the each to-be-classified and recognized modality information into the target multi-modality classification recognition model; perform modality alignment representation on the each to-be-classified and recognized modality information through the target multi-modality classification recognition model, to obtain a target modality alignment representation vector; and perform multi-modality classification recognition on the target modality alignment representation vector through the target multi-modality classification recognition model, to obtain a classification recognition result corresponding to the each to-be-classified and recognized modality information. 23.A computer device, comprising a memory and a processor, wherein the memory stores a computer program. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 11.

24. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 11.

25. A computer program product comprising a computer program, characterised in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 11. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 11.

Images