Artificial intelligence model training method and device for multi-modal feature space alignment

Abstract

The invention provides an artificial intelligence model training method and device for multi-modal feature space alignment. The method comprises the following steps: calculating modal alignment loss of paired data; calculating distribution alignment loss of non-paired data; calculating cross-modal comparison loss of the training data; calculating non-orthogonal penalty constraint loss; constructing a total loss function of the model based on modal alignment loss, distribution alignment loss, cross-modal comparison loss and non-orthogonal penalty constraint loss; updating parameters of the model based on the total loss function; and under the condition that the model meets the preset training index, completing the training of the model, otherwise, repeating the above steps until the model meets the preset training index. The model obtained by training through the method can directly align the centers and rotation differences of a plurality of modalities on a feature level, and realizes stable fusion and alignment of different modal features in a unified space.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model training technology, and in particular to an artificial intelligence model training method and apparatus for multimodal feature space alignment. Background Technology

[0002] In the field of multimodal machine learning, integrating information from different modalities such as images, text, and audio to improve model performance has become an important research direction. However, data from different modalities differ fundamentally in physical characteristics, sampling methods, feature encoding mechanisms, and intrinsic distribution patterns, resulting in significant gaps (modal differences) in the feature representations of each modality in the semantic space.

[0003] The modal discrepancy problem manifests itself in several ways: First, the distribution centers, scale ranges, and orientations of features in different modalities within the embedding space are inconsistent, making it difficult for the model to perform effective similarity measurement and feature comparison in a common semantic space. Second, even with advanced methods such as shared parameter networks or contrastive learning, the lack of alignment and rotational relationships between different modal feature spaces makes it difficult to guarantee the global geometric structure consistency of cross-modal features. Furthermore, in downstream tasks such as cross-modal retrieval, matching, and fusion, this modal discrepancy directly leads to chaotic feature clustering results and reduces the semantic similarity between samples from different modalities, thus severely impacting task performance.

[0004] To alleviate the aforementioned problems, existing technologies often employ methods such as adversarial training or complex nonlinear projection networks to forcibly reduce the differences between feature distributions of different modalities. However, these methods typically suffer from inherent drawbacks such as unstable training processes, high optimization difficulty, and weak interpretability. Furthermore, they often struggle to simultaneously maintain the geometric structure of the feature space and achieve distribution alignment, thus limiting their practical applications. Therefore, a novel artificial intelligence model training method for multimodal feature space alignment is urgently needed to overcome the aforementioned problems in existing technologies. Summary of the Invention

[0005] This invention provides an artificial intelligence model training method and apparatus for multimodal feature space alignment, which addresses the shortcomings of existing technologies such as difficulty in performing effective similarity measurement and feature comparison in a common semantic space, difficulty in ensuring the global geometric structure consistency of cross-modal features, and reduction in cross-modal similarity due to modal differences. It achieves stable and interpretable alignment of cross-modal features in terms of center, scale, and direction, thereby significantly improving the performance of downstream tasks.

[0006] This invention provides a method for training an artificial intelligence model for multimodal feature space alignment, comprising: The features of the training data are obtained, and the training data is determined to be paired or unpaired based on the correspondence between the features. When the training data consists of paired data, calculate the modal alignment loss of the paired data. When the training data is unpaired, calculate the distribution alignment loss of the unpaired data. ; Calculate the cross-modal contrastive loss of the training data. ; Calculate the non-orthogonal penalty constraint loss used to prevent the model from learning distortion or reflection transformations. ; Based on the modal alignment loss Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss Construct the total loss function L of the model; The parameters of the model are updated based on the total loss function L; If the model meets the preset training metrics, the training of the model is completed; otherwise, the above steps are repeated until the model meets the preset training metrics.

[0007] According to the present invention, an artificial intelligence model training method for multimodal feature space alignment is provided, wherein the modal alignment loss of the pairwise data is... It is calculated using the following formula:

[0008] Wherein, K is the total number of modalities in the training data; For modality The number of samples, 1≤ ≤K; For the first Feature matrix of each modality , k=1,2,…,K; D is the feature dimension, R is the set of real numbers; T k (x) represents the affine transformation; Z is the global template center for all modes.

[0009] According to the present invention, an artificial intelligence model training method for multimodal feature space alignment is characterized in that, when the training data is paired data, the modal alignment loss of the paired data is calculated. Previously, it also included: In the case where the paired data does not have category labels, a clustering algorithm is used to add category labels to the paired data. Based on the category labels, the category-conditional alignment loss of the pairwise data is calculated using the following formula. As modality alignment loss :

[0010] Where C is the total number of categories in the training data; K is the total number of modalities in the training data; For modality The number of samples, 1≤ ≤K; For the first The feature matrix of each modality; T k (x) represents the affine transformation; Z c The template center for category c.

[0011] According to the present invention, an artificial intelligence model training method for multimodal feature space alignment is provided, wherein the affine transformation... The definition is as follows:

[0012] in, , indicating mode Scale factor; , indicating mode rotation matrix; , indicating mode The translation vector; The global template center Z for all modalities is defined as follows:

[0013] in, The transformed mode The center is defined as:

[0014] For the mode before transformation The center is defined as: .

[0015] According to the present invention, an artificial intelligence model training method for multimodal feature space alignment is provided, which updates the parameters of the model based on the total loss function L, including: Based on the total loss function L, update , and ;

[0016] Update the parameters of the backbone network of the model.

[0017] According to the present invention, an artificial intelligence model training method for multimodal feature space alignment is provided, based on the modal alignment loss. Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss Construct the total loss function L of the model, including: For the modal alignment loss Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss We perform a weighted summation to construct the total loss function L of the model: .

[0018] According to the present invention, an artificial intelligence model training method for multimodal feature space alignment is provided, wherein training of the model is completed when the model meets preset training metrics, comprising: Training of the model is completed when the decrease in the total loss function L of the model is less than a preset threshold. Alternatively, training of the model can be completed when the model has reached a preset number of training rounds.

[0019] The present invention also provides an artificial intelligence model training device for multimodal feature space alignment, comprising: The data acquisition and determination module is used to acquire features of the training data and determine whether the training data is paired or unpaired based on the correspondence between the features. The pairwise data alignment module is used to calculate the mode alignment loss of the pairwise data when the training data is pairwise data. ; The unpaired data alignment module is used to calculate the distribution alignment loss of the unpaired data when the training data is unpaired data. ; The contrastive learning module is used to calculate the cross-modal contrastive loss of the training data. ; The geometric constraint module is used to calculate the non-orthogonal penalty constraint loss to prevent the model from learning distortions or reflection transformations. ; Loss fusion module, used for fusion based on the modality alignment loss Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss Construct the total loss function L of the model; The parameter optimization module is used to update the parameters of the model based on the total loss function L; The iterative training module is used to complete the training of the model if the model meets the preset training metrics; otherwise, the above steps are repeated until the model meets the preset training metrics.

[0020] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the artificial intelligence model training method for multimodal feature space alignment as described above.

[0021] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the artificial intelligence model training method for multimodal feature space alignment as described above.

[0022] The present invention provides an artificial intelligence model training method and apparatus for multimodal feature space alignment. This method acquires features from the training data and determines whether the training data is paired or unpaired based on the correspondence between features. If the training data is paired, it calculates the modal alignment loss for the paired data. When the training data is unpaired, calculate the distribution alignment loss for the unpaired data. ; Calculate the cross-modal contrastive loss of the training data ; Calculate the non-orthogonal penalty constraint loss used to prevent the model from learning distortions or reflection transformations. Based on modal alignment loss Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss The process involves constructing the total loss function L for the model; updating the model parameters based on the total loss function L; completing the training of the model if it meets the preset training metrics, otherwise repeating the above steps until the model meets the preset training metrics. The model trained using the above method can directly align the center and rotation differences of multiple modalities at the feature level, achieving stable fusion and alignment of features from different modalities in a unified space. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the artificial intelligence model training method for multimodal feature space alignment provided by the present invention. Figure 2 This is a schematic diagram of the training process of the total loss function in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the artificial intelligence model training device for multimodal feature space alignment provided by the present invention; Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] The following is combined with Figure 1 and Figure 2 This invention describes an artificial intelligence model training method for multimodal feature space alignment.

[0027] like Figure 1 As shown, the artificial intelligence model training method for multimodal feature space alignment provided by the present invention includes the following steps:

[0028] S1. Obtain the features of the training data, and determine whether the training data is paired or unpaired based on the correspondence between the features.

[0029] In an optional embodiment of the invention, raw feature representations of different modalities (such as images, text, and audio) can be obtained from pre-trained or multiple independent multimodal feature extractors (such as visual CNNs and text encoders). Each modality... The features are represented as matrices Their feature dimension is D, but their distribution center, scale and direction are significantly different (i.e., there is "modal difference").

[0030] The acquired training data is categorized into paired and unpaired data. Paired data consists of feature samples with a clear pairing relationship (e.g., an image and its corresponding text description), while unpaired data consists of feature samples without a direct correspondence. By clearly defining the nature of the training data, the method provided by this invention can handle both supervised (paired / labeled data) and unsupervised (unpaired data) scenarios. For example, in an image-text retrieval task, a photo of a dog and its description "a golden retriever running in the grass" constitute a paired sample. However, a picture of a cat without descriptive text crawled from the internet, and an unrelated audio clip saying "The weather is nice today," belong to unpaired data.

[0031] S2. When the training data consists of paired data, calculate the modal alignment loss for the paired data. When the training data is unpaired, calculate the distribution alignment loss for the unpaired data. .

[0032] For data with pairing relationships (corresponding to supervised training conditions), calculate the modality alignment loss. This loss directly minimizes the distance between each modally transformed feature and the global template center Z, achieving accurate point-level center alignment. For example, suppose that after transformation, the center point of the image feature is in [1, 2], and the center point of the text feature is in [4, 5]. The translation vectors are adjusted using gradient descent to bring both centers closer to the global center Z (assumed to be [2.5, 3.5]). Specifically, this involves the modal alignment loss of paired data. It is calculated using the following formula:

[0033] Where K is the total number of modalities in the training data; For modality The number of samples, 1≤ ≤K; For the first Feature matrix of each modality k=1,2,…,K; D is the feature dimension, and R is the set of real numbers; For affine transformation; Z is the global template center for all modes.

[0034] The mode alignment loss For paired data without category labels, simple modality alignment may ignore the semantic relationships between samples. Introducing category information (whether pseudo-labels obtained from clustering or real labels) can achieve fine-grained, category-conditional alignment, ensuring that samples of different modalities within the same category (e.g., "dog") are clustered together in a unified space, while samples of different categories (e.g., "dog" and "cat") are separated. Therefore, as... Figure 2 As shown, in an optional embodiment of the invention, for such data, the modal alignment loss of paired data is calculated. Previously, it also included:

[0035] When paired data does not have category labels, a clustering algorithm is used to add category labels to the paired data.

[0036] Based on the category labels, the category-conditional alignment loss of paired data is calculated using the following formula. As modality alignment loss :

[0037]

[0038] Where C is the total number of categories in the training data; K is the total number of modalities in the training data; For modality The number of samples, 1≤ ≤K; For the first Feature matrix of each modality; For affine transformation; Z c The template center is for category c. Category conditional alignment loss. This ensures that samples from different modalities within the same semantic category cluster more tightly after transformation, achieving fine-grained semantic alignment. For example, during training, we not only want the feature centers of all "image modalities" and all "text modalities" to align, but also want all image features related to "dogs" and all text features related to "dogs" to cluster tightly within a unified space. The loss function is designed to achieve this goal. In an optional embodiment of the invention, clustering algorithms can be used to construct category labels for the data by combining the data's own features or additional features (from other models), including but not limited to using the DBSCAN algorithm, K-means algorithm, classification models, or other clustering methods.

[0039] The affine transformation T in the above formula k The definition of (x) is as follows:

[0040]

[0041] This invention establishes an independent linear transformation module for each mode. ,in, , indicating mode The scaling factor is used for distribution scaling matching; , indicating mode The rotation matrix is ​​used to preserve the geometry through orthogonal constraints; , indicating mode The translation vector is used for center alignment.

[0042] The global template center Z for all modalities is defined as follows:

[0043] in, The transformed mode The center is defined as:

[0044] For the mode before transformation The center is defined as: .

[0045] For data without pairing relationships (corresponding to the condition of unsupervised training), calculate the distribution alignment loss. In an optional embodiment of the present invention, the distribution alignment loss can be calculated using MMD (Maximum Mean Discrepancy, a metric that measures the difference in means between two distributions in a feature space) or CORAL (Correlation Alignment, an algorithm that aligns feature distributions by matching covariance matrices). To unify the distribution structure of different modalities, MMD achieves distribution matching by comparing the means in the reproducing kernel Hilbert space, while CORAL matches distributions by aligning second-order statistics (covariance matrices). They bring different modalities closer together in terms of overall distribution shape, rather than just aligning centers.

[0046] S3. Calculate the cross-modal contrastive loss of the training data. .

[0047] In this step, regardless of whether the data are paired or not, as long as positive and negative sample pairs can be constructed (paired data can be constructed naturally, and unpaired data can be obtained through data mining), the contrastive loss is calculated. This loss does not directly concern itself with the global center or distribution, but rather optimizes the local structure of features, bringing positive sample pairs closer together and pushing negative sample pairs apart, directly improving the performance of downstream tasks (such as retrieval). In an optional embodiment of the invention, common algorithms such as infoNCE can be used to calculate the cross-modal contrastive loss. .

[0048] S4. Calculate the non-orthogonal penalty constraint loss used to prevent the model from learning distortions or reflection transformations. .

[0049] Non-orthogonal penalty constraint loss Used to constrain the rotation matrix W 𝑘 Approximate orthogonality is maintained to preserve the angular structure between modes, which is crucial for ensuring "global structural consistency" and "not disrupting the angular structure." This loss prevents the model from learning distortion or reflection transformations, thus ensuring the interpretability of rotations.

[0050] It should be understood that the above steps S2, S3 and S4 have no specific execution order and can be performed simultaneously or sequentially as needed. This invention does not impose any restrictions on this.

[0051] S5, Based on Modal Alignment Loss Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss , construct the total loss function L of the model.

[0052] like Figure 2 As shown, the total loss function L of the model consists of the modal alignment loss. Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss It consists of four parts. Specifically, it addresses the modal alignment loss. Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss We perform a weighted summation to construct the model's total loss function L:

[0053]

[0054] in, , , and Hyperparameters are used to balance the importance of different objectives. For example, when there are many paired data points, the hyperparameter can be increased. (Compare loss weights); When concerned about the model distorting geometry, you can increase the weights. (Orthogonal constraint weights). It should be understood that in one training iteration, and They are mutually exclusive, depending on whether the input data batches are paired or unpaired, with one of the items having a weight coefficient of 0.

[0055] S6. Update the model parameters based on the total loss function L.

[0056] Specifically, based on the total loss function L, update , and This also includes updating the parameters of the model's backbone network. In an optional embodiment of the invention, a gradient descent algorithm can be used to perform backpropagation based on the total loss L, while simultaneously updating the mode-specific alignment transformation parameters. The parameters of the backbone network of the model enable the backbone network to learn to extract original features that are easier to align.

[0057] S7. If the model meets the preset training metrics, complete the training of the model; otherwise, repeat steps S1 to S7 until the model meets the preset training metrics.

[0058] Specifically, model training is completed when the decrease in the model's total loss function L is less than a preset threshold; or, when the model has reached a preset number of training epochs. For example, a threshold can be set according to the actual situation. When the decrease in the model's total loss L is less than this preset threshold (i.e., the total loss L no longer decreases significantly), or when the model has reached a preset number of training epochs N, the model is considered to have converged, i.e., training is complete.

[0059] In summary, the artificial intelligence model training method for multimodal feature space alignment provided by this invention obtains the features of the training data, determines whether the training data is paired or unpaired based on the correspondence between the features, and calculates the modal alignment loss of the paired data when the training data is paired. When the training data is unpaired, calculate the distribution alignment loss for the unpaired data. ; Calculate the cross-modal contrastive loss of the training data ; Calculate the non-orthogonal penalty constraint loss used to prevent the model from learning distortions or reflection transformations. Based on modal alignment loss Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss The process involves constructing the total loss function L for the model; updating the model parameters based on the total loss function L; completing the training of the model if it meets the preset training metrics, otherwise repeating the above steps until the model meets the preset training metrics. The model trained using this method can directly align the center and rotation differences of multiple modalities at the feature level, reducing the multimodal distribution gap and improving the accuracy of cross-modal retrieval and matching tasks. It does not destroy the angular structure of the features and can be flexibly applied under supervised (class-conditional) or unsupervised (MMD / CORAL) conditions. The model exhibits stable convergence, strong interpretability, and can achieve stable fusion and alignment of features from different modalities in a unified space.

[0060] Based on the same inventive concept, the present invention also provides an artificial intelligence model training device for multimodal feature space alignment. The artificial intelligence model training device for multimodal feature space alignment provided by the present invention will be described below. The artificial intelligence model training device for multimodal feature space alignment described below can be referred to in correspondence with the artificial intelligence model training method for multimodal feature space alignment described above.

[0061] like Figure 3 As shown, the artificial intelligence model training device for multimodal feature space alignment provided by the present invention includes a data acquisition and judgment module 31, a paired data alignment module 32, an unpaired data alignment module 33, a contrastive learning module 34, a geometric constraint module 35, a loss fusion module 36, a parameter optimization module 37, and an iterative training module 38.

[0062] The data acquisition and determination module 31 is used to acquire features of the training data and determine whether the training data is paired data or unpaired data based on the correspondence between the features.

[0063] The pairwise data alignment module 32 is used to calculate the mode alignment loss of the pairwise data when the training data is pairwise data. .

[0064] The unpaired data alignment module 33 is used to calculate the distribution alignment loss of the unpaired data when the training data is unpaired data. .

[0065] Contrast learning module 34 is used to calculate the cross-modal contrastive loss of the training data. .

[0066] Geometric constraint module 35 is used to calculate the non-orthogonal penalty constraint loss to prevent the model from learning distortions or reflection transformations. .

[0067] Loss fusion module 36, used for fusion based on the modality alignment loss Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss Construct the total loss function L of the model.

[0068] The parameter optimization module 37 is used to update the parameters of the model based on the total loss function L.

[0069] The iterative training module 38 is used to complete the training of the model if the model meets the preset training indicators; otherwise, the above steps are repeated until the model meets the preset training indicators.

[0070] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute the artificial intelligence model training method for multimodal feature space alignment provided by the above methods, which includes:

[0071] The features of the training data are obtained, and the training data is determined to be paired or unpaired based on the correspondence between the features.

[0072] When the training data consists of paired data, calculate the modal alignment loss of the paired data. When the training data is unpaired, calculate the distribution alignment loss of the unpaired data. ;

[0073] Calculate the cross-modal contrastive loss of the training data. ;

[0074] Calculate the non-orthogonal penalty constraint loss used to prevent the model from learning distortion or reflection transformations. ;

[0075] Based on the modal alignment loss Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss Construct the total loss function L of the model;

[0076] The parameters of the model are updated based on the total loss function L;

[0077] If the model meets the preset training metrics, the training of the model is completed; otherwise, the above steps are repeated until the model meets the preset training metrics.

[0078] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0079] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the artificial intelligence model training method for multimodal feature space alignment provided by the above methods, the method comprising:

[0080] The features of the training data are obtained, and the training data is determined to be paired or unpaired based on the correspondence between the features.

[0081] When the training data consists of paired data, calculate the modal alignment loss of the paired data. When the training data is unpaired, calculate the distribution alignment loss of the unpaired data. ;

[0082] Calculate the cross-modal contrastive loss of the training data. ;

[0083] Calculate the non-orthogonal penalty constraint loss used to prevent the model from learning distortion or reflection transformations. ;

[0084] Based on the modal alignment loss Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss Construct the total loss function L of the model;

[0085] The parameters of the model are updated based on the total loss function L;

[0086] If the model meets the preset training metrics, the training of the model is completed; otherwise, the above steps are repeated until the model meets the preset training metrics.

[0087] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the artificial intelligence model training method for multimodal feature space alignment provided by the methods described above, the method comprising:

[0088] The features of the training data are obtained, and the training data is determined to be paired or unpaired based on the correspondence between the features.

[0089] When the training data consists of paired data, calculate the modal alignment loss of the paired data. When the training data is unpaired, calculate the distribution alignment loss of the unpaired data. ;

[0090] Calculate the cross-modal contrastive loss of the training data. ;

[0091] Calculate the non-orthogonal penalty constraint loss used to prevent the model from learning distortion or reflection transformations. ;

[0092] Based on the modal alignment loss Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss Construct the total loss function L of the model;

[0093] The parameters of the model are updated based on the total loss function L;

[0094] If the model meets the preset training metrics, the training of the model is completed; otherwise, the above steps are repeated until the model meets the preset training metrics.

[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training an artificial intelligence model for multimodal feature space alignment, characterized in that, include: The features of the training data are obtained, and the training data is determined to be paired or unpaired based on the correspondence between the features. When the training data consists of paired data, calculate the modal alignment loss of the paired data. ; When the training data is unpaired, calculate the distribution alignment loss of the unpaired data. ; Calculate the cross-modal contrastive loss of the training data. ; Calculate the non-orthogonal penalty constraint loss used to prevent the model from learning distortion or reflection transformations. ; Based on the modal alignment loss Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss Construct the total loss function L of the model; The parameters of the model are updated based on the total loss function L; If the model meets the preset training metrics, the training of the model is completed; otherwise, the above steps are repeated until the model meets the preset training metrics.

2. The artificial intelligence model training method for multimodal feature space alignment according to claim 1, characterized in that, The modal alignment loss of the pairwise data It is calculated using the following formula: Wherein, K is the total number of modalities in the training data; For modality The number of samples, 1≤ ≤K; For the first Feature matrix of each modality , k=1,2,…,K; D is the feature dimension, R is the set of real numbers; T k (x) represents the affine transformation; Z is the global template center for all modes.

3. The artificial intelligence model training method for multimodal feature space alignment according to claim 2, characterized in that, When the training data is paired data, the modal alignment loss of the paired data is calculated. Previously, it also included: In the case where the paired data does not have category labels, a clustering algorithm is used to add category labels to the paired data. Based on the category labels, the category-conditional alignment loss of the pairwise data is calculated using the following formula. As modal alignment loss : ， Where C is the total number of categories in the training data; K is the total number of modalities in the training data; For modality The number of samples, 1≤ ≤K; For the first The feature matrix of each modality; T k (x) represents the affine transformation; Z c The template center for category c.

4. The artificial intelligence model training method for multimodal feature space alignment according to claim 3, characterized in that, The affine transformation The definition is as follows: ， in, , indicating mode Scale factor; , indicating mode rotation matrix; , indicating mode Translation vector; The global template center Z for all modalities is defined as follows: ， in, The transformed mode The center is defined as: ， For the mode before transformation The center is defined as: 。 5. The artificial intelligence model training method for multimodal feature space alignment according to claim 4, characterized in that, Based on the total loss function L, the parameters of the model are updated, including: Based on the total loss function L, update , and ; Update the parameters of the backbone network of the model.

6. The method for training an artificial intelligence model for multimodal feature space alignment according to claim 1, characterized in that, Based on the modal alignment loss Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss Construct the total loss function L of the model, including: For the modal alignment loss Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss We perform a weighted summation to construct the total loss function L of the model: .

7. The method for training an artificial intelligence model for multimodal feature space alignment according to any one of claims 1 to 6, characterized in that, Training the model is completed when the model meets preset training metrics, including: Training of the model is completed when the decrease in the total loss function L of the model is less than a preset threshold. Alternatively, training of the model can be completed when the model has reached a preset number of training rounds.

8. An artificial intelligence model training device for multimodal feature space alignment, characterized in that, include: The data acquisition and determination module is used to acquire features of the training data and determine whether the training data is paired or unpaired based on the correspondence between the features. The pairwise data alignment module is used to calculate the mode alignment loss of the pairwise data when the training data is pairwise data. ; The unpaired data alignment module is used to calculate the distribution alignment loss of the unpaired data when the training data is unpaired data. ; The contrastive learning module is used to calculate the cross-modal contrastive loss of the training data. ; The geometric constraint module is used to calculate the non-orthogonal penalty constraint loss to prevent the model from learning distortions or reflection transformations. ; Loss fusion module, used for fusion based on the modality alignment loss Distribution alignment loss Cross-modal contrast loss and non-orthogonal penalty constraint loss Construct the total loss function L of the model; The parameter optimization module is used to update the parameters of the model based on the total loss function L; The iterative training module is used to complete the training of the model if the model meets the preset training metrics; otherwise, the above steps are repeated until the model meets the preset training metrics.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the artificial intelligence model training method for multimodal feature space alignment as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the artificial intelligence model training method for multimodal feature space alignment as described in any one of claims 1 to 7.

Images