Low-quality multi-modal animal image division method based on unbalanced feature distribution multidirectional alignment

Abstract

The invention discloses a low-quality multi-modal animal image division method based on unbalanced feature distribution multi-directional alignment, and belongs to the field of data division in low-quality multi-modal animal image data, and the method comprises the steps: firstly obtaining potential embedded features of a low-quality multi-modal animal image data sample through a coding network of an auto-encoder; and the low-quality multi-modal animal image data sample is processed by a Transform mechanism to obtain discriminative embedded representation, and at the same time, a reconstruction loss function is used to constrain the low-quality multi-modal animal image data sample and a reconstruction sample feature. And inputting the potential embedded features of different modes into a feature distribution multidirectional alignment function to align the feature distributions of the different modes so as to obtain consistency information among the different modes. And then fusing the potential embedding features of the different modes through a feature fusion layer to obtain common embedding representations of all the modes, and aligning the common embedding representations with discriminative embedding representations by comparing a loss function to obtain complementarity information of the different modes. And finally, using a small-batch stochastic gradient descent algorithm to optimize the overall loss function of the low-quality multi-modal animal image division method based on unbalanced feature distribution multi-directional alignment to convergence. And after the model converges, performing category division on the public embedded representation by using a k-means algorithm, and calculating the division accuracy of the low-quality multi-modal animal image data according to the division result.

Description

Technical Field

[0001] The present invention belongs to the field of data partitioning in low-quality multimodal animal image data, and in particular relates to a low-quality multimodal animal image partitioning method based on multi-directional alignment of unbalanced feature distribution. Background Art

[0002] In the field of multimedia information processing, animal image data is not only widely available but can also be collected from multiple perspectives or channels to describe the same data. Animal image data with multiple modes or collected from multiple sources is referred to as multimodal animal image data. The key to clustering this type of data lies in how to mine the consistency and complementarity between modalities within multimodal data, thereby exploring the distribution characteristics of multimodal data and better clustering it. However, due to information loss or channel failures at various stages of acquisition, transmission, and storage, multimodal animal image data can be incomplete, resulting in low-quality multimodal data. Therefore, clustering incomplete multimodal animal image data has become an integral part of low-quality multimodal learning. Furthermore, the lack of modalities not only severely impacts the accuracy of clustering but also disrupts the consistency of the distribution of multimodal animal image data in the latent space, making data fusion difficult. Based on this, this paper proposes a low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distributions. This method uses a multi-directional feature distribution alignment function and a contrastive loss function to align the latent embedded feature distributions between modalities, fully capturing the consistency and complementarity between the modalities. Furthermore, by introducing a Transformer mechanism, the network's latent feature extraction capabilities are enhanced, thereby achieving accurate clustering of low-quality multimodal animal image data. Summary of the Invention

[0003] To address the above challenges, the present invention provides a low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distribution, the method comprising the steps of:

[0004] First, the low-quality animal image data samples in each modality are input into the encoding network to obtain the latent embedding features of each modality. Then, the latent embedding features are used to obtain discriminative embedding representations through the Transformer mechanism. Finally, the discriminative embedding representations are input into the decoding network to obtain the reconstructed sample features.

[0005] The feature distribution multi-directional alignment function is used in the embedding space to align the potential embedding feature distributions of multiple modalities to obtain consistency information between different modalities.

[0006] The latent embedding features of each modality are fused through a feature fusion layer to obtain a common embedding representation for all modalities, which is then aligned with the discriminative embedding representation through a contrastive loss function to obtain complementary information of different modalities.

[0007] The overall loss function of a low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distributions is optimized to convergence using a mini-batch stochastic gradient descent algorithm.

[0008] After the model converges, the k-means algorithm is used to classify the common embedding representation. Based on the classification results, the classification accuracy of low-quality multimodal animal image data is calculated.

[0009] Furthermore, the method inputs low-quality animal image data samples in each modality into the encoding network to obtain the latent embedding features of each modality, then obtains a discriminative embedding representation of the latent embedding features through the Transformer mechanism, and finally inputs the discriminative embedding representation into the decoding network to obtain the reconstructed sample features.

[0010] The low-quality animal image data samples in each modality are input into the encoding network to obtain the potential embedding features of each modality. The proposed encoding network expression is as follows:

[0011] H v =E v (X v ψ v )

[0012] Among them, E v (·) The encoding network of the vth modality, ψ v represents the learnable parameters of the encoding network of the vth modality, X v Indicates the existence of animal image data samples of the vth modality, H v represents the latent embedding features of the v-th modality.

[0013] The latent embedding features of low-quality multimodal animal image data samples are transformed into discriminative embedding representations through the Transformer mechanism. The proposed Transformer mechanism expression is:

[0014]

[0015] Z v =LayerNorm(H v +Dropout(Attention(H v ))

[0016] Among them, Q represents the Query value, K represents the Key value, V represents the Value value, and WQ 、W K 、W V represents three different linear layer weight matrices, Attention(·) represents the self-attention mechanism layer, Softmax(·) represents the Softmax activation function, LayerNorm(·) represents the regularization layer, and Z v represents a discriminative embedding representation.

[0017] The discriminative embedding representation is input into the decoding network to obtain the reconstructed sample features. The proposed decoding network expression is:

[0018]

[0019] in, represents the decoding network of the vth view, φ v represents the learnable parameters of the decoding network for the vth view, Represents the reconstructed sample features of the v-th view.

[0020] The reconstruction loss function is used to constrain the low-quality multimodal animal image data samples and the reconstructed sample features. The proposed reconstruction loss function is expressed as:

[0021]

[0022] Among them, X v represents the existence of animal image data samples of the vth modality, represents the reconstructed sample features of the vth mode, represents the i-th animal image data sample in the v-th modality, represents the i-th reconstructed sample feature in the v-th mode, V represents the number of modes, N v represents the number of samples in the vth mode. ‖·‖2 represents the L2 norm of the matrix, represents the reconstruction loss function of the v-th modality, represents the reconstruction loss function for all modalities.

[0023] Furthermore, the feature distribution multi-directional alignment function is used in the embedding space to align the potential embedding feature distributions of multiple modalities to obtain consistency information between different modalities. The proposed feature distribution multi-directional alignment function is expressed as:

[0024]

[0025] Among them, H v represents the potential embedding features of the vth modality, H t represents the potential embedding features of the t-th modality, P v 、Pt Respectively represent H v and H t The potential feature distribution, D v (·) represents the discriminator network of the vth modality, represents the learnable parameters of the discriminator network for the vth modality, ψ t represents the learnable parameters of the encoding network of the t-th mode, V represents the number of modes, Represents the feature distribution multi-directional alignment loss function.

[0026] Furthermore, the latent embedding features of each modality are fused through a feature fusion layer to obtain a common embedding representation of all modalities, and it is aligned with the discriminative embedding representation through a contrastive loss function to obtain complementary information of different modalities.

[0027] The potential embedded features of different modalities are fused through the feature fusion layer. The proposed feature fusion layer expression is:

[0028]

[0029] in, represents the potential embedding features of the i-th sample in the v-th modality, γ v , γ j They represent the learnable parameters of the vth and jth modalities respectively, V represents the number of modalities, N represents the number of samples, and H represents the public embedding representation.

[0030] The common embedding representation is aligned with the discriminative embedding representation through the contrastive loss function. The proposed contrastive loss function is expressed as:

[0031]

[0032] Among them, V represents the number of modes, N represents the number of samples, represents the contrast loss function of the v-th modality, represents the contrastive loss function for all modalities, represents the similarity between the i-th sample in the common embedding representation and the j-th sample in the v-th modality, which is defined as follows:

[0033]

[0034] Among them, h i represents the i-th sample in the common embedding representation, The jth sample in the vth mode, N is the number of samples.

[0035] Furthermore, the mini-batch stochastic gradient descent algorithm is used to optimize the overall loss function of the low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distribution to convergence.

[0036] The reconstruction loss function, the feature distribution multi-directional alignment loss function, and the contrast loss function are summed to obtain the overall loss function of the low-quality multimodal animal image segmentation method based on unbalanced feature distribution multi-directional alignment. The proposed expression formula is:

[0037]

[0038] in, represents the reconstruction loss function, represents the feature distribution multi-directional alignment loss function, represents the contrastive loss function.

[0039] Furthermore, after the model converges, the k-means algorithm is used to classify the common embedding representation, and the classification accuracy of the low-quality multimodal animal image data is calculated based on the classification results.

[0040] The present invention provides a low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distribution, which has the following advantages:

[0041] (1) The method adopts a deep multimodal segmentation framework, which fully utilizes the consistency and complementary information between the modalities of low-quality animal image data, can effectively mine the potential segmentation structure of the data, and thus achieve accurate category division.

[0042] (2) The method uses a multi-directional alignment function of feature distribution to align the potential embedded feature distributions between modalities of low-quality animal image data to fully obtain the consistency information between modalities.

[0043] (3) The method uses a contrastive loss function to align the common embedding representation with the discriminative embedding representation, so that the common embedding representation fully integrates the complementary information of different modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the drawings without paying any creative work.

[0045] Figure 1 This is a flow chart of a low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distribution provided by the present invention.

[0046] Figure 2 This is a schematic diagram of some sample data of the Animal-50 dataset image. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present invention.

[0048] Exemplary Methods

[0049] like Figure 1 As shown, after the processing flow starts, step S110 is first executed.

[0050] Step S110: Input the existing sample into the encoding network to obtain the corresponding potential embedding features, then pass it through the Transformer mechanism to obtain a discriminative embedding representation, and finally obtain the reconstructed sample features through the decoding network.

[0051] The encoding network is used to obtain potential embedding features for low-quality multimodal animal image data samples. As an example, the encoding network defined in step S110 is expressed as follows:

[0052] H v =E v (X v ψ v ) (1)

[0053] Among them, E v (·) The encoding network of the vth modality, ψ v represents the learnable parameters of the encoding network of the vth modality, X v Indicates the existence of animal image data samples of the vth modality, H v represents the latent embedding features of the v-th modality.

[0054] The latent embedding features of low-quality multimodal animal image data samples are used through the Transformer mechanism to obtain a discriminative embedding representation. As an example, the Transformer mechanism expression defined in step S110 is as follows:

[0055]

[0056] Z v =LayerNorm(H v +Dropout(Attention(H v ))

[0057] Among them, Q represents the Query value, K represents the Key value, V represents the Value value, and W Q 、W K 、W V represents three different linear layer weight matrices, Attention(·) represents the self-attention mechanism layer, Softmax(·) represents the Softmax activation function, LayerNorm(·) represents the regularization layer, and Z v represents a discriminative embedding representation.

[0058] The discriminative embedding representation is input into the decoding network to obtain the reconstructed sample features. As an example, the expression of the decoding network defined in step S110 is as follows:

[0059]

[0060] in, Represents the reconstructed sample features of the vth modality, G v (·) represents the decoding network of the vth modality, φ v represents the learnable parameters of the decoding network for the vth modality.

[0061] The reconstruction loss function is used to constrain the low-quality multimodal animal image data samples and the reconstructed sample features. As an example, the reconstruction loss function expression is defined as:

[0062]

[0063] Among them, X v represents the existence of animal image data samples of the vth modality, represents the reconstructed sample features of the vth mode, represents the i-th animal image data sample in the v-th modality, represents the i-th reconstructed sample feature in the v-th mode, V represents the number of modes, N v represents the number of samples in the vth mode. ‖·‖2 represents the L2 norm of the matrix, represents the reconstruction loss function of the v-th modality, represents the reconstruction loss function for all modalities.

[0064] Step S120: Use the feature distribution multi-directional alignment function to align the potential embedding feature distribution to obtain consistency information between different modalities. As an example, the distribution alignment function defined in step S120 is expressed as follows:

[0065]

[0066] Among them, H vrepresents the potential embedding features of the vth modality, H t represents the potential embedding features of the t-th modality, P v 、P t Respectively represent H v and H t The potential feature distribution, D v (·) represents the discriminator network of the vth modality, represents the learnable parameters of the discriminator network for the vth modality, ψ t represents the learnable parameters of the encoding network of the t-th mode, V represents the number of modes, Represents the feature distribution multi-directional alignment loss function.

[0067] Step S130: The latent embedding features are fused through a feature fusion layer to obtain a common embedding representation, and are aligned with the discriminative embedding representation through a contrast loss function to obtain complementary information between modalities.

[0068] The latent embedding features of different modalities are fused through the feature fusion layer. As an example, the feature fusion layer expression defined in step S130 is as follows:

[0069]

[0070] in, represents the potential embedding features of the i-th sample in the v-th modality, γ v , γ j They represent the learnable parameters of the vth and jth modalities respectively, V represents the number of modalities, N represents the number of samples, and H represents the public embedding representation.

[0071] The common embedding representation is aligned with the discriminative embedding representation through the contrastive loss function. As an example, the contrastive loss function defined in step S130 is as follows:

[0072]

[0073] Among them, V represents the number of modes, N represents the number of samples, represents the contrast loss function of the v-th modality, represents the contrastive loss function for all modalities, represents the similarity between the i-th sample in the common embedding representation and the j-th sample in the v-th modality, which is defined as follows:

[0074]

[0075] Among them, h i represents the i-th sample in the common embedding representation, The jth sample in the vth mode, N is the number of samples.

[0076] Step S140: Using a mini-batch stochastic gradient descent algorithm, the overall loss function of the low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distribution is optimized to convergence.

[0077] The reconstruction loss function, the feature distribution multi-directional alignment loss function, and the contrast loss function are summed to obtain the overall loss function of the low-quality multimodal animal image segmentation method based on feature distribution alignment. As an example, the overall loss function defined in step S140 is expressed as follows:

[0078]

[0079] in, represents the reconstruction loss function, represents the feature distribution multi-directional alignment loss function, represents the contrastive loss function.

[0080] Step S150: After the model converges, the k-means algorithm is used to classify the common embedding representation, and the classification accuracy of the low-quality multimodal animal image data is calculated based on the classification results.

[0081] Through this embodiment, the latent embedding features of low-quality multimodal animal image data samples are first obtained through the encoding network of the autoencoder, and a discriminative embedding representation is obtained through the Transformer mechanism. At the same time, the reconstruction loss function is used to constrain the low-quality multimodal animal image data samples and the reconstructed sample features. The latent embedding features of different modalities are then input into the feature distribution multi-directional alignment function to align the feature distributions of different modalities to obtain consistency information between different modalities. The latent embedding features of different modalities are then fused through the feature fusion layer to obtain a common embedding representation of all modalities, and it is aligned with the discriminative embedding representation through the contrast loss function to obtain complementary information of different modalities. Finally, the mini-batch stochastic gradient descent algorithm is used to optimize the overall loss function of the low-quality multimodal animal image segmentation method based on the multi-directional alignment of unbalanced feature distributions until convergence. After the model converges, the k-means algorithm is used to classify the common embedding representation, and the segmentation accuracy of the low-quality multimodal animal image data is calculated based on the segmentation results.

[0082] Detailed description of the invention Results

[0083] This implementation uses the publicly available Animal-50 multimodal animal image dataset and simulates low-quality multimodal animal image data with varying sample missing ratios. The details of the Animal-50 dataset are as follows:

[0084] The Animal-50 dataset contains 10,158 images of animals from 50 different animal categories. These images were collected in a variety of natural and artificial environments, at varying times of day and weather, and in various behavioral states (such as running, still, and eating). The animals are shown in a variety of poses, including frontal, side, and back views. The images were captured against diverse backgrounds, ranging from natural grasslands and forests to simple, solid-color backgrounds. Individual animals also exhibit variations in appearance details, such as fur length and color.

[0085] To verify the superiority of our method (Ours), we compared our method with several common low-quality multimodal animal image data classification methods, including AIMC, ICMVC, GIMVC, DVIMVC, etc., and compared the clustering accuracy (ACC) of these methods on the above-mentioned face image data. The specific clustering results are shown in Table 1.

[0086] Table 1. Classification accuracy of Animal-50 multimodal animal image data (%)

[0087]

[0088] The comparison of clustering accuracy of different methods in the above table clearly shows that for low-quality multimodal animal image data with different missing ratios, the method of the present invention (Ours) achieves the best performance index and has a relatively high clustering accuracy. The method of the present invention obtains the potential embedding representation of the sample by using an autoencoder, and uses the Transformer mechanism in this latent space to obtain a discriminative embedding representation. The distribution of animal image data in different modalities is aligned through the feature distribution multi-directional alignment function and the contrast loss function to obtain consistency information and complementary information between the modalities, thereby achieving the purpose of accurately classifying low-quality multimodal animal image data.

[0089] This embodiment proposes a low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distribution, which is used to classify low-quality multimodal animal image data commonly found in life. This method obtains the potential embedding features of different modalities through an autoencoder network, and aligns the distribution of animal image data between different modalities through a multi-directional alignment function of feature distribution and a contrast loss function to obtain consistency information and complementary information between modalities, thereby achieving the purpose of accurately classifying low-quality multimodal animal image data. Experimental results on five missing sample ratios of low-quality multimodal animal image data simulated using the Animal-50 dataset show that this embodiment has higher classification accuracy and better superiority than other methods.

[0090] It should be understood that the above-described specific embodiments of the present invention are merely illustrative or illustrative of the principles of the present invention and do not constitute limitations of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included within the scope of protection of the present invention. In addition, the appended claims are intended to cover all variations and modifications that fall within the scope and metes and bounds of the appended claims, or equivalents thereof.

Claims

1. A low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distribution, characterized in that: The method comprises the steps of: For low-quality multimodal animal image data samples, an encoding network is used to obtain the potential embedding features of different modalities, and then a discriminative embedding representation is obtained through the Transformer mechanism. Finally, the reconstructed sample features are obtained through the decoding network. In the embedding space, the feature distribution multi-directional alignment function is used to align the potential embedding feature distributions of multiple modalities to obtain consistency information between different modalities; The latent embedding features of different modalities are fused through the feature fusion layer to obtain a common embedding representation for all modalities, and then aligned with the discriminative embedding representation through the contrastive loss function to obtain complementary information of different modalities; The overall loss function of a low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distributions is optimized to convergence using a mini-batch stochastic gradient descent algorithm. After the model converges, the k-means algorithm is used to classify the common embedding representation. Based on the classification results, the classification accuracy of low-quality multimodal animal image data is calculated.

2. The low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distribution according to claim 1, characterized in that: For low-quality multimodal animal image data samples, an encoding network is used to obtain the potential embedding features of different modalities, and then a discriminative embedding representation is obtained through the Transformer mechanism. Finally, the reconstructed sample features are obtained through the decoding network. The encoding network is used to obtain the potential embedding features for low-quality multimodal animal image data samples. The expression of the encoding network is as follows: H v =E v (X v ;ψ v ) Among them, E v (·) The encoding network of the vth modality, ψ v represents the learnable parameters of the encoding network of the vth modality, X v Indicates the existence of animal image data samples of the vth modality, H v represents the potential embedding features of the v-th modality; The potential embedding features of low-quality multimodal animal image data samples are used through the Transformer mechanism to obtain a discriminative embedding representation, which is expressed as follows: Z v =LayerNorm(H v +Dropout(Attention(H v )) Among them, Q represents the Query value, K represents the Key value, V represents the Value value, and W Q 、W K 、W V represents three different linear layer weight matrices, Attention(·) represents the self-attention mechanism layer, Softmax(·) represents the Softmax activation function, LayerNorm(·) represents the regularization layer, and Z v Represents discriminative embedding representation; The discriminative embedding representation is input into the decoding network to obtain the reconstructed sample features, which is expressed as: in, Represents the reconstructed sample features of the vth modality, G v (·) represents the decoding network of the vth modality, φ v represents the learnable parameters of the decoding network of the vth modality; The reconstruction loss function is used to constrain the low-quality multimodal animal image data samples and the reconstructed sample features. The expression of the reconstruction loss function is as follows: Among them, X v represents the existence of animal image data samples of the vth modality, represents the reconstructed sample features of the vth mode, represents the i-th animal image data sample in the v-th modality, represents the i-th reconstructed sample feature in the v-th mode, V represents the number of modes, N v represents the number of samples in the vth mode. ‖·‖2 represents the L2 norm of the matrix, represents the reconstruction loss function of the v-th modality, represents the reconstruction loss function for all modalities.

3. The low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distribution according to claim 1, characterized in that: In the embedding space, the feature distribution multi-directional alignment function is used to align the potential embedding feature distributions of multiple modalities to obtain the consistency information between different modalities. The expression formula is: Among them, H v represents the potential embedding features of the vth modality, H t represents the potential embedding features of the t-th modality, P v 、P t Respectively represent H v and H t The potential feature distribution, D v (·) represents the discriminator network of the vth modality, represents the learnable parameters of the discriminator network for the vth modality, ψ t represents the learnable parameters of the encoding network of the t-th mode, V represents the number of modes, Represents the feature distribution multi-directional alignment loss function.

4. The low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distribution according to claim 1, characterized in that: The latent embedding features of different modalities are fused through the feature fusion layer to obtain a common embedding representation for all modalities, and then aligned with the discriminative embedding representation through the contrastive loss function to obtain complementary information of different modalities; The latent embedding features of different modalities are fused through the feature fusion layer. The definition of the feature fusion layer is as follows: in, represents the potential embedding features of the i-th sample in the v-th modality, γ v , γ j Represent the learnable parameters of the vth and jth modalities respectively, V represents the number of modalities, N represents the number of samples, and H represents the public embedding representation; The common embedding representation is aligned with the discriminative embedding representation through the contrastive loss function, which is expressed as: Among them, V represents the number of modes, N represents the number of samples, represents the contrast loss function of the v-th modality, represents the contrastive loss function for all modalities, represents the similarity between the i-th sample in the common embedding representation and the j-th sample in the v-th modality, which is defined as follows: Among them, h i represents the i-th sample in the common embedding representation, The jth sample in the vth mode, N is the number of samples.

5. The low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distribution according to claim 1, characterized in that: The overall loss function of a low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distributions is optimized to convergence using a mini-batch stochastic gradient descent algorithm. The reconstruction loss function, the feature distribution multidirectional alignment loss function, and the contrast loss function are summed to obtain the overall loss function of the low-quality multimodal animal image segmentation method based on feature distribution alignment, which is expressed as follows: in, represents the reconstruction loss function, represents the feature distribution multi-directional alignment loss function, represents the contrastive loss function.

6. The low-quality multimodal animal image segmentation method based on multi-directional alignment of unbalanced feature distribution according to claim 1, characterized in that: After the model converges, the k-means algorithm is used to classify the common embedding representation. Based on the classification results, the classification accuracy of low-quality multimodal animal image data is calculated.

Images