Image feature extraction method based on self-supervised triple neural network

Abstract

The present application relates to an image feature extraction method based on a self-supervised triple neural network. The method comprises: constructing an image feature extraction model based on a self-supervised triple neural network; the image feature extraction model comprises a student network encoder, a homogeneous teacher network encoder and a heterogeneous teacher network encoder and according to the three encoders, a first teacher-student network distillation structure, a second teacher-student network distillation structure and a teacher-teacher network distillation structure are formed; according to a pre-designed feature differential regression loss and a pre-designed feature diversity retention loss, the first teacher-student network distillation structure, the teacher-teacher network distillation structure and the second teacher-student network distillation structure are respectively distilled and trained, and the trained image feature extraction model is used to extract features from the domain image to be extracted. The use of this method can improve the accuracy of image feature extraction.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image feature extraction method based on a self-supervised triple neural network. Background Art

[0002] In the field of computer vision, image recognition and image segmentation are key links in many applications and are widely used in many fields such as medicine and transportation. However, the accuracy of image recognition and image segmentation of many images is often unsatisfactory. In the medical field, such as the analysis of medical images, due to the complexity and diversity of images, traditional image recognition and segmentation technologies are difficult to accurately identify diseased tissues, resulting in a high misdiagnosis rate. In the field of transportation, the images taken by road monitoring cameras will also affect the recognition and segmentation accuracy of targets such as vehicles and pedestrians under complex weather or lighting conditions. The root cause of these problems lies in the insufficient accuracy of image feature extraction. Image features are the basis of image recognition and segmentation. Accurate extraction of image features can provide strong support for subsequent analysis and decision-making.

[0003] Existing image feature extraction methods have many limitations. Deep learning has achieved remarkable results in image tasks in many fields, such as image classification, image forgery, semantic segmentation, etc. However, supervised learning methods rely on manually designed labels. In the fields of medicine and military, labels are expensive and difficult to obtain, which makes it difficult to obtain relevant data and impossible to train effectively. Self-supervised learning, as an unsupervised learning method, can effectively solve these problems. Early self-supervised learning methods such as K-Means, based on data similarity, performed similarity analysis in feature space to achieve image classification, but performed poorly when processing data with small feature differences or different modalities. For example, in complex image scenes such as medical imaging and reconnaissance, the feature differences of images are small, and the traditional K-Means method is difficult to accurately extract features, resulting in low image recognition and segmentation accuracy. At the same time, some methods start from the image content and use rotation, masking or jigsaw puzzles to predict image attributes, but have difficulties in semantic recognition of complex image scenes. For example, in traffic monitoring images, the semantics of images under different weather and lighting conditions are complex, and existing methods are difficult to accurately recognize. Although generative adversarial networks (GANs) can play a role in complex scenarios and achieve learning goals by pitting the generator against the discriminator, this method requires a lot of computing resources and is limited in practical applications. In recent years, self-supervised learning methods based on contrastive networks have shown strong representation learning performance in image processing. Contrastive learning encourages the model to learn input feature representations of different data enhancements, is more adaptable to data, and has reached the level of supervised learning on large-scale datasets such as ImageNet.

[0004] However, most of these methods are designed for image data. When faced with multimodal tasks, such as combining infrared images with visible light images, ultrasound images with magnetic resonance images, etc., it is difficult to effectively handle the feature differences of different modal data, resulting in difficulty in improving image recognition and segmentation accuracy. Therefore, improving the accuracy of image feature extraction has become the key to solving the problem of image recognition and segmentation accuracy. By improving the image feature extraction method, the key features of the image can be better captured, information loss can be reduced, the accuracy of image recognition and segmentation can be improved, and the needs of image recognition and segmentation accuracy in various fields can be met. Summary of the invention

[0005] Based on this, it is necessary to provide an image feature extraction method based on a self-supervised triple neural network that can improve the accuracy of image feature extraction in response to the above technical problems.

[0006] A method for extracting image features based on a self-supervised triple neural network, the method comprising:

[0007] Acquire the domain image to be extracted; construct an image feature extraction model based on a self-supervised triple neural network; the image feature extraction model includes a student network encoder, a homogeneous teacher network encoder with the same structure as the student network encoder, and a heterogeneous teacher network encoder; the student network encoder and the homogeneous teacher network encoder constitute the first teacher-student network distillation structure; the student network encoder and the heterogeneous teacher network encoder constitute the second teacher-student network distillation structure; the homogeneous teacher network encoder and the heterogeneous teacher network encoder constitute the teacher-teacher network distillation structure;

[0008] According to the pre-designed feature differential regression loss and the pre-designed feature diversity preservation loss, the first teacher-student network distillation structure, the teacher-teacher network distillation structure, and the second teacher-student network distillation structure are respectively distilled and trained to obtain a trained image feature extraction model;

[0009] The trained image feature extraction model is used to extract features from the domain image to be extracted.

[0010] The above-mentioned image feature extraction method based on self-supervised triple neural network, this application aims at the shortcomings of traditional Siamese network in processing multimodal data, introduces heterogeneous teacher network branches to form a triple heterogeneous network architecture, this structure can use homogeneous and heterogeneous networks to perform distillation training on the same target network (student network), so that the student network can learn feature representation from different angles, and different network branches can capture their own unique feature information, and then pass it to the student network through distillation training, thereby improving the student network's ability to extract complex multimodal image features, thereby improving the accuracy of image recognition and segmentation. In the teacher-student distillation model structure composed of homogeneous teacher network-student network, the gradient disappearance phenomenon is prevented by designing feature differential regression loss for normalization. This ensures that during the training process, the student network can stably learn similar but better feature representations from the homogeneous teacher network, thereby improving the accuracy of image feature extraction.

[0011] In the teacher-teacher network distillation structure, feature differential regression loss is used to enable the two teacher networks to learn and optimize each other in the process of pursuing the consistency of features obtained by the encoder. This helps to better align and fuse features under different modalities when processing multimodal data, so that the student network can learn a comprehensive visual feature representation and improve the accuracy of multimodal image feature extraction. In the teacher-student network distillation structure composed of a heterogeneous teacher network-student network, the use of a pre-designed feature diversity preservation loss can prevent the gradient explosion phenomenon caused by excessive differences in non-information vectors from the encoder and avoid network collapse. This ensures the stability of the student network when learning features from the heterogeneous teacher network, and can fully absorb the unique feature information captured by the heterogeneous network, while avoiding the inability to train normally due to gradient explosion, thereby improving the ability to extract complex image features, especially for images under complex weather or lighting conditions in the field, which can more accurately extract features and improve image recognition and segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 1 is a flow chart of an image feature extraction method based on a self-supervised triple neural network in one embodiment;

[0013] Figure 2 A schematic diagram of the detailed structure and detailed information of a self-supervised triple neural network in one embodiment;

[0014] Figure 3 A schematic diagram of feature diversity preservation loss in one embodiment. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0016] In one embodiment, Figure 1 As shown, a method for extracting image features based on a self-supervised triple neural network is provided, comprising the following steps:

[0017] Step 102, obtain the domain image to be extracted; construct an image feature extraction model based on a self-supervised triple neural network; the image feature extraction model includes a student network encoder, a homogeneous teacher network encoder with the same structure as the student network encoder, and a heterogeneous teacher network encoder; the student network encoder and the homogeneous teacher network encoder constitute a first teacher-student network distillation structure; the student network encoder and the heterogeneous teacher network encoder constitute a second teacher-student network distillation structure; the homogeneous teacher network encoder and the heterogeneous teacher network encoder construct a teacher-teacher network distillation structure.

[0018] Domain image representation includes images from various fields, such as medical images, including X-ray images, CT (computed tomography) images, MRI (magnetic resonance imaging) images, and ultrasound images; the transportation field includes road monitoring images, vehicle-mounted camera images, etc.

[0019] Step 104, performing distillation training on the first teacher-student network distillation structure, the teacher-teacher network distillation structure, and the second teacher-student network distillation structure according to a pre-designed feature differential regression loss and a pre-designed feature diversity retention loss, respectively, to obtain a trained image feature extraction model.

[0020] Self-supervised triple neural network composed of student network model Homogeneous Teacher Network Model and heterogeneous teacher networks Composed of. In the teacher-student distillation model structure composed of Perform gradient regression calculation and design feature differential regression loss for normalization to prevent gradient disappearance. In the teacher-student network distillation structure, feature diversity preservation loss is used to prevent gradient explosion caused by excessive differences in non-information vectors from the encoder, thus avoiding network collapse. This design enables the student network to continuously improve its feature extraction capabilities during training, thereby improving the performance of the overall model. The improvement direction of feature extraction capability is consistent, and the two different teacher models need to be standardized. This application adopts the mutual distillation method to form a teacher-teacher Distillation network, with The teacher and student distillation networks are consistent, and both pursue the consistency of encoder features, so both use feature differential regression loss. In addition, in order to ensure the alignment of data features of multimodal information, Data augmentation methods adopt different strategies when faced with different modal information.

[0021] like Figure 2 As shown, this application extracts three different views in a self-supervised triple neural network and uses three network encoders for training. Whether they are consistent, they can be divided into isomorphic encoders and heterogeneous encoders ,exist Figure 2 In are respectively expressed as and , where only the encoder No gradient regression is used, and there is no multi-layer perception layer at the end. When calculating the loss, two major loss features are involved: differential regression loss and feature diversity preservation loss ,exist Figure 2 In are respectively expressed as and , where the student-teacher network The loss is calculated using the loss function , the rest of the student-teacher network and teacher-teacher networks Both use The purpose of the self-supervised triple neural network model is to perform multimodal information fusion through knowledge distillation. Through distillation learning, the student network can obtain more advanced feature representations from the teacher network. The gradient backpropagation can avoid the excessive dependence of the student network on the teacher network, and promote the student network to learn autonomously. Combined with the loss function design, it can prevent the model from overfitting and improve the model's adaptability to multimodal data. The self-supervised triple neural network model improves the performance of the student network on specific tasks through a series of carefully designed mechanisms, and enhances its adaptability and generalization when facing multimodal data.

[0022] Step 106: Use the trained image feature extraction model to perform feature extraction on the domain image to be extracted.

[0023] In the above-mentioned image feature extraction method based on self-supervised triple neural network, this application introduces heterogeneous teacher network branches to form a triple heterogeneous network architecture in order to address the shortcomings of traditional Siamese networks in processing multimodal data. This structure can use homogeneous and heterogeneous networks to perform distillation training on the same target network (student network), so that the student network can learn feature representations from different angles, and different network branches can capture their own unique feature information, which is then passed to the student network through distillation training, thereby improving the student network's ability to extract complex multimodal image features, thereby improving the accuracy of image recognition and segmentation. In the teacher-student distillation model structure composed of homogeneous teacher network-student network, the gradient disappearance phenomenon is prevented by designing feature differential regression loss for normalization. This ensures that during the training process, the student network can stably learn similar but better feature representations from the homogeneous teacher network, thereby improving the accuracy of image feature extraction. In the teacher-teacher network distillation structure, feature differential regression loss is used so that the two teacher networks learn and optimize each other in the process of pursuing the consistency of the encoder's acquisition of features. This helps to better align and fuse features from different modalities when processing multimodal data, so that the student network can learn a comprehensive visual feature representation and improve the accuracy of multimodal image feature extraction. In the teacher-student network distillation structure composed of a heterogeneous teacher network and a student network, the use of a pre-designed feature diversity retention loss can prevent the gradient explosion caused by excessive differences in non-information vectors from the encoder and avoid network collapse. This ensures the stability of the student network when learning features from the heterogeneous teacher network, and can fully absorb the unique feature information captured by the heterogeneous network, while avoiding the inability to train normally due to gradient explosion, thereby improving the ability to extract complex image features, especially for images under complex weather or lighting conditions, which can more accurately extract features and improve image recognition and segmentation accuracy.

[0024] In one embodiment, the process of pre-designing feature differential regression loss includes:

[0025] Given a view and , enter the encoder respectively A network consisting of an encoder and an MLP predictor The corresponding features are obtained by training, which is expressed as ; Define regression loss , in the regression loss Based on this, we design feature differential regression loss for:

[0026] ;

[0027] in, Indicates balanced encoder and Network Parameters of the extraction capability.

[0028] In a specific embodiment, It is designed to calculate the similarity between the features predicted by the MLP predictor and the original features, thereby guiding model training. Based on the and The difference between the two, introducing parameters To balance the encoder and network structure The extraction capability of ,and can more accurately evaluate the gap between the perspective information.

[0029] In one embodiment, the regression loss is defined as:

[0030] ;

[0031] in, Represents different variables.

[0032] In one embodiment, the process of pre-designing feature diversity preservation loss includes:

[0033] Define the given two eigenvectors as z and z', whose characteristic dimensions are ,matrix , Used to represent two sets of embedded vectors, where represents the original feature data, Represents the target feature, and the matrix size is , where each row represents the feature vector obtained after encoding the sample image, is the number of samples, is the feature dimension, and the invariance loss is constructed by calculating the square error between the features generated by the model and the target features and normalizing them;

[0034] From the original feature data matrix Subtract the row mean , get the normalized feature data matrix ; Calculate the standard deviation using the normalized feature data matrix, and construct the standard deviation loss based on the standard deviation;

[0035] Transpose the normalized feature data matrix With the normalized feature data matrix Multiply and then divide by the number of samples minus one to get the covariance matrix of the original feature data; define the covariance regularization term as the sum of the squares of the non-diagonal coefficients of the covariance matrix, and construct the correlation loss based on the covariance regularization term;

[0036] The invariance loss, standard deviation loss, and correlation loss are weighted averaged to obtain the feature diversity preservation loss.

[0037] In a specific embodiment, the structure of the feature diversity preservation loss is as follows Figure 3 As shown, the loss consists of three losses, namely, the representation invariance loss of the feature , standard deviation loss and covariance loss . The embedding vector is mainly calculated, and the three losses are used to calculate the similarity within the embedding vector, between the embedding vector groups, and between the embedding vector pairs. Encourage the model to learn the invariance and diversity of representations, which is achieved by weighting the differences between features, the distribution of features themselves, and the relationships within features, so that the model can capture the essential properties of the data. The formal representation of Figure 3 As shown, the loss needs to calculate the representation invariance loss, standard deviation loss and covariance loss of the features separately. Specifically, the invariance loss refers to measuring the similarity of the vector itself by calculating the mean square distance between the embedding vectors. Standard deviation loss, as a difference measure, can be used to keep the embedding vectors of samples different from each other, so that the standard deviation of each embedded variable on a batch of data is above a given threshold. Covariance loss, as the endogenous difference of the embedding vector, can attract each pair of concentrated embedded variables to zero. This term decorrelates each embedded variable to prevent information collapse. The standard deviation and covariance terms are applied independently to the two branches of the architecture, which retains the information of each embedding vector to a certain extent, and further reduces the possibility of information collapse through the endogenous differences of the data. This makes the loss There is no need for the two branch networks to share weights, which makes it possible to use different network structures; compared with other contrast losses, this loss does not require the use of a repository or large-scale comparison samples, which improves the computational efficiency of the model; and does not require vector quantization or predictor modules. This makes the network applicable to a variety of different data types and tasks.

[0038] In one embodiment, the invariance loss is constructed by calculating the square error between the features generated by the model and the target features and normalizing them:

[0039] .

[0040] In a specific embodiment, the feature representation learned by the model is optimized by calculating the square error between the feature generated by the model and the target feature and normalizing it. By using the square error to calculate the similarity of the feature representation, it can be ensured that in data analysis, no single feature has a disproportionate impact on the result because of its large numerical scale. Normalization helps to balance these differences, allowing the model to more fairly evaluate the contribution of each feature.

[0041] In one embodiment, the standard deviation is calculated using the normalized feature data matrix:

[0042] ;

[0043] in, is the normalized feature data matrix No. The mean of the column, is the dimension of the feature, is a constant target value for the standard deviation.

[0044] In one embodiment, the standard deviation loss is constructed based on the standard deviation as:

[0045] .

[0046] In a specific embodiment, using It is used to calculate the standard deviation to quantify the degree of dispersion of characteristic data. is a constant target value for the standard deviation, which encourages the variance within the current batch to be equal in each dimension , to prevent all inputs mapped to the same vector from collapsing, and is usually set to a small positive number. To ensure that the standard deviation loss is within a reasonable range, ReLu operations are performed for all standard deviations less than 1. Finally, the two sets of different embedding vector matrices are weighted averagely to construct the standard deviation loss.

[0047] In one embodiment, the covariance regularization term is defined as the sum of the squares of the off-diagonal coefficients of the covariance matrix:

[0048] ;

[0049] in, is the Kronecker function, Representation variables The covariance matrix of Representation variables In the Row and The covariance values ​​of the columns, Representation variables The values ​​on the diagonal of the covariance matrix of .

[0050] In one embodiment, the correlation loss is constructed based on the covariance regularization term as

[0051] .

[0052] In a specific embodiment, the covariance loss is constructed Focusing on quantifying the linear relationship between features and encouraging them to remain independent in the embedding space, we first calculate the covariance matrix of the feature data, which is obtained by transposing the normalized feature matrix With the normalized feature matrix Multiply them together and then divide by the sample size minus one. Next, define the covariance regularization term for The sum of the squares of the off-diagonal coefficients is:

[0053] ;

[0054] in is the Kronecker function, when hour, ;otherwise, This function ensures that the application only operates on non-diagonal elements. So the correlation loss function yes:

[0055] ;

[0056] in is the matrix dimension, The off-diagonal coefficients are encouraged to be close to 0. By minimizing the defined loss function, the model learns to decorrelate the feature vectors of the embedding layer, which results in each feature vector encoding information independently in the representation layer, reducing the redundancy between features and enhancing the generalization and interpretability of the model.

[0057] In one embodiment, the invariance loss, the standard deviation loss, and the correlation loss are weighted averaged to obtain the feature diversity preservation loss, including:

[0058] The invariance loss, standard deviation loss and correlation loss are weighted averaged to obtain the feature diversity preservation loss:

[0059] ;

[0060] in , and is a hyperparameter that controls the importance of each loss, is the invariance loss, is the standard deviation loss, is the correlation loss.

[0061] In a specific embodiment, the present application uses 9 benchmark datasets to verify the data adaptability of the self-supervised triple neural network (HTNet). Including: MNIST, FashionMNIST, KMNIST, USPS, SVHN, CIFAR10, CIFAR-100, STL-10 and Tiny-ImageNet, where MNIST represents a handwritten digit recognition dataset, FashionMNIST represents a fashion item image dataset, KMNIST represents a Korean handwritten character recognition dataset, USPS represents a country A postal service digital image dataset, SVHN represents a street view house number dataset, CIFAR10 represents a 10-category image dataset of the Advanced Research Institute of Country B, CIFAR-100 represents a 100-category image dataset of the Advanced Research Institute of Country B, STL-10 represents a Stanford micro-image-10-category dataset, and Tiny-ImageNet represents a tiny image network dataset. TriBYOL, BYOL and Sim-Siam are currently the most advanced self-supervised learning methods without negative sample pairs, and VICReg, PIRL-rotation, PIRL-jigsaw and SimCLR are currently the most advanced self-supervised learning methods with negative sample pairs. ImageNet in the experimental results table represents the results of the model test pre-trained on the ImageNet dataset. This application experiments with different downstream tasks, including linear evaluation, transfer learning, multimodal image and text retrieval. Linear evaluation and transfer learning are mainly compared with the above methods. In multimodal image and text retrieval, this application uses the MS-COCO dataset. It should be noted that in the above datasets, this application only uses unlabeled data for training. In the ablation experiment, this application verifies the feature extraction ability and multimodal ability of the model, hides the mutual distillation loss method and triple network structure of the model, and observes the performance of the model.

[0062] A batch size sensitivity study was conducted on the STL-10 dataset, an established benchmark for evaluating the effectiveness of image recognition models. Our method involves a 40-epoch training scenario where we vary the batch size to determine its impact on the training dynamics and final model performance. After the training phase, we perform linear evaluation experiments using a pre-trained ResNet-18 encoder to assess the generalization ability of the model. Through this experimental process, an important finding emerged: the batch size has a profound impact on the training results. Notably, we observed that the loss reduction rate decreases as the batch size increases. This observation is consistent with the seminal findings that hypothesize that larger batch sizes slow down convergence due to increased variance in gradient estimates. To mitigate the adverse effects of large batches on model performance, we strategically employ a linear scaling rule. This rule is a well-established technique in machine learning that involves increasing the learning rate as the batch size grows. Our goal is to maintain an optimal balance between convergence speed and stability of the training process. This adjustment ensures that our model not only performs well during the training phase, but also maintains its robustness and effectiveness when applied to downstream classification tasks. Our approach embodies a meticulous and proactive strategy that allows for complex interactions between batch size, learning rate, and model performance, thus laying the foundation for the successful deployment of our model in real-world applications.

[0063] In this section, the application uses the batch sizes that perform well above to perform linear evaluation on different datasets to determine the feature extraction performance of the HTNet of the application. The application selects three benchmark datasets: CIFAR-10, CIFAR100 and STL-10, which can provide detailed evaluation of the performance of multiple aspects of HTNet. The application trains 80 epochs on the training set, and then fine-tunes the features of the pre-trained ResNet-50 decoder for 200 epochs on the test set, and evaluates every 10 epochs.

[0064] As shown in Table 1, HTNet has a clear performance advantage when the batch size is small. This performance is obvious compared to other self-supervised learning methods, which proves that HTNet can perform better feature extraction with a small sample size. In some cases, HTNet's performance even exceeds the pre-trained model on the ImageNet dataset. In addition, the experiments of this application reveal the challenges faced by methods such as SimSiam. When the batch size is configured to 32, SimSiam encounters training instability, which leads to complete collapse and produces very poor results, such as its accuracy on CIFAR-100 is only 1.00% and its accuracy on STL-10 is only 10.00%. This further emphasizes the robustness of HTNet, which can maintain model performance under various test conditions. The model visualization after linear evaluation in this application under the stl10 dataset shows that the pre-trained images have good perception of different categories in STL10, especially for pictures with small differences, such as deer and horses, which will pay more attention to different parts, such as deer will pay more attention to its horns, while horses will pay more attention to its legs. Table 1 shows the linear evaluation results (%) of HTNet with different mini-batches on three datasets. "B32", "B64" and "B128" represent batch sizes of 32, 64 and 128, respectively. The bold numbers represent the best classification accuracy achieved on the dataset using the self-supervised method. "-" indicates that this method does not work well at this batch size and the experimental results are not recorded.

[0065] Table 1

[0066]

[0067] The results on CIFAR-10, CIFAR-100, and STL-10 in Table 1 collectively confirm the assertion that HTNet performs well in learning nuanced and discriminative representations, outperforming other state-of-the-art SSL methods, especially when trained with different mini-batches. This finding suggests that HTNet can be effectively deployed in environments with limited computational resources and has the potential to become an efficient SSL framework.

[0068] This application extends the evaluation to the field of transfer learning and conducts a comprehensive evaluation of the HTNet framework. This application uses the Tiny-ImageNet dataset for training, freezes the trained encoder, and then tests its effect on eight different downstream datasets. This method can measure the generalized feature expression ability of HTNet in other fields. This application sets the number of training cycles to 80 to give the model enough time to extract and encode meaningful representations from the data, and uses the ResNet50 architecture as the backbone to balance performance and computational efficiency. Throughout the training process, this application keeps the batch size at 128, which achieves a favorable trade-off between convergence speed and generalization ability. In the downstream migration stage, this application fine-tunes the encoder for 200 epochs, and evaluates performance every 10 epochs. Table 2 shows the transfer learning results (%) on different datasets. Different algorithms are used for pre-training on the Tiny ImageNet dataset, and the trained dataset is used for transfer learning on other datasets, where the pre-trained encoder is frozen.

[0069] Table 2

[0070]

[0071] The results of the transfer learning experiments of this application are included in Table 2, which compares the performance of HTNet with other state-of-the-art SSL methods on 8 test datasets side by side. According to the data provided, the transfer learning experimental results of HTNet on multiple benchmark datasets show that it has significant advantages in feature extraction and generalization ability, especially when compared with other self-supervised learning methods such as TriBYOL, Cross, BYOL, and SimSiam. HTNet has shown performance close to or exceeding the ImageNet pre-trained model on datasets such as MNIST, FashionMNIST, KMNIST, and USPS, demonstrating its effectiveness in small sample learning tasks. Although HTNet's performance on the SVHN dataset is slightly lower than the ImageNet pre-trained model, its excellent performance on CIFAR-100 and STL-10 highlights its potential for handling complex visual tasks. These results together confirm the robustness and efficiency of HTNet as a self-supervised learning framework, while also pointing out that further optimization may be needed on specific datasets to improve performance. The data shows that HTNet consistently outperforms its competitors and has a high generalization representation ability. The results show that HTNet is good at learning features that are invariant to changes in datasets and tasks, thereby achieving more effective knowledge transfer. The transfer learning results on various datasets provide convincing evidence for HTNet's ability to learn generalized representations.

[0072] This application extends the application of HTNet to the field of multimodal learning, with a particular focus on the task of matching images and corresponding captions in the MS-Coco 2017 dataset. This dataset provides an ideal testbed for evaluating HTNet's multimodal capabilities in processing visual and textual data. The method of this application involves a Dual-Encoders architecture that uses two different encoders to process image and text modalities. Image Encoder The task is to extract rich visual features from images, while the text encoder The text information from the corresponding title is processed. Table 3 shows the evaluation results (%) of the MS-COCO 5K retrieval task. In all settings, the text encoder is a word embedding followed by a GRU layer.

[0073] Table 3

[0074]

[0075] According to the data in Table 3, HTNet shows excellent performance in image-to-text and text-to-image retrieval tasks, significantly surpassing existing methods such as VSE++, Barlow Twins, and VICReg. HTNet achieved 35.7%, 63.2%, and 78.3% in the R@1, R@5, and R@10 indicators of Image-to-text, and 46.1%, 78.4%, and 85.7% in the corresponding indicators of Text-to-Image, highlighting its obvious advantages in accuracy and efficient information retrieval capabilities. These results not only demonstrate HTNet's powerful multimodal learning ability and robustness, but also demonstrate its advantages in capturing the essential characteristics of image and text data.

[0076] In summary, HTNet shows its superior performance and potential in handling multimodal data retrieval tasks with its advanced self-supervised learning framework. This superior performance can be attributed to the unique architecture of HTNet. In addition, the incorporation of image self-distillation plays a key role in improving its performance. By leveraging self-distillation technology, HTNet is able to extract and improve image representations, thereby providing a more comprehensive understanding of visual data. This additional representation learning layer enables HTNet to surpass the performance of VICReg, especially in tasks that require a detailed understanding of the visual field.

[0077] Under the premise of ensuring the normal use of the model, in order to verify the effectiveness of the triple network structure and loss function proposed in this application, this application divides the network into a heterogeneous network group H-Net, a homogeneous network group I-Net and a non-inter-distilled network group un-HTNet. H-Net consists of a student network and heterogeneous teacher networks And the loss function I-Net consists of student networks and heterogeneous teacher networks And the loss function composition, while un-HTNet lacks mutual distillation loss . This application carried out tasks of image and text retrieval, linear evaluation and transfer learning respectively to demonstrate the multimodal and feature extraction capabilities of the model. The transfer learning used the STL-10 dataset. The number of training and testing rounds was the same as in the previous experiment, and other parameter settings were consistent with HTNet. Table 4 shows the ablation experiments of different modules of HTNet (%). The networks were divided into heterogeneous network group H-Net, homogeneous network group I-Net and non-distilled network group un-HTNet and HTNet for comparison. For the multimodal module, the GRU layer was used as the text information embedding layer. '-' means that the network does not have multimodal capabilities and no related experiments were conducted. The experimental training rounds and batch sizes all selected the best parameters of the previous experiment.

[0078] Table 4

[0079]

[0080] The results shown in Table 4 demonstrate the effectiveness of HTNET in multimodal learning, indicating that HTNET can effectively process and learn both visual and textual data. The superior performance of HTNET in downstream tasks, as well as the enhanced capabilities due to image self-distillation, make it a promising method for widespread application in multimodal learning. The heterogeneous network group in this application is compared and analyzed in Table 4. The results show that HTNET is able to skillfully handle two different types of data (images and text) and is able to learn robust representations that capture the essence of both modalities. The results show that HTNET outperforms the Barlow Twins method in downstream tasks of image and text retrieval, and this superior performance can be attributed to the unique architecture of HTNET. In addition, the incorporation of image self-distillation plays a key role in improving its performance. By leveraging self-distillation, HTNET is able to extract and improve image representations, leading to a more comprehensive understanding of visual data. This additional layer of representation learning enables HTNET to surpass the performance of VICReg, especially in tasks that require a detailed understanding of the visual domain.

[0081] This application is based on the architecture of Triple Network, and completes the self-distillation and mutual distillation tasks of two different viewpoints through a dual heterogeneous teacher-student model. This method improves the performance of self-supervised learning representation learning, enabling self-supervised learning to perform multimodal tasks. This application conducts many experiments on small-scale data, including linear evaluation of different batch sizes, downstream task migration, and multimodal image and text retrieval. The experimental results show that in these data sets, the proposed method is significantly better than the traditional method, and the model can also perform well in small-scale data that is not limited by computing resources.

[0082] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0083] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image feature extraction method based on a self-supervised triple neural network, characterized in that: The method comprises: Acquire a domain image to be extracted; construct an image feature extraction model based on a self-supervised triple neural network; the image feature extraction model includes a student network encoder, a homogeneous teacher network encoder having the same structure as the student network encoder, and a heterogeneous teacher network encoder; the student network encoder and the homogeneous teacher network encoder constitute a first teacher-student network distillation structure; the student network encoder and the heterogeneous teacher network encoder constitute a second teacher-student network distillation structure; the homogeneous teacher network encoder and the heterogeneous teacher network encoder constitute a teacher-teacher network distillation structure; According to a pre-designed feature differential regression loss and a pre-designed feature diversity retention loss, the first teacher-student network distillation structure, the teacher-teacher network distillation structure, and the second teacher-student network distillation structure are respectively distilled and trained to obtain a trained image feature extraction model; Use the trained image feature extraction model to extract features from the domain image to be extracted; The process of pre-designing feature diversity preserving loss includes: Define the given two eigenvectors as z and z', whose characteristic dimension is D, and the matrix , Used to represent two sets of embedded vectors, where represents the original feature data, Represents the target feature, and the matrix size is , where each row represents the feature vector obtained after encoding the sample image, N is the number of samples, D is the feature dimension, and the invariance loss is constructed by calculating the square error between the features generated by the model and the target features and normalizing them; From the original feature data matrix Subtract the row mean , get the normalized feature data matrix ; Calculate the standard deviation using the normalized feature data matrix, and construct the standard deviation loss based on the standard deviation; Transpose the normalized feature data matrix With the normalized feature data matrix Multiply and then divide by the number of samples minus one to obtain the covariance matrix of the original feature data; define the covariance regularization term as the sum of the squares of the non-diagonal coefficients of the covariance matrix, and construct the correlation loss according to the covariance regularization term; The invariance loss, standard deviation loss and correlation loss are weighted averaged to obtain the feature diversity preservation loss.

2. The method according to claim 1, characterized in that: The process of the pre-designed feature differential regression loss includes: Given a view and , enter the encoder respectively A network consisting of an encoder and an MLP predictor The corresponding features are obtained by training, which is expressed as ; Define regression loss , in the regression loss Based on this, we design feature differential regression loss for: ； in, Indicates balanced encoder and Network Parameters of the extraction capability.

3. The method according to claim 2, characterized in that The regression loss is defined as: ； in, Represents different variables.

4. The method according to claim 1, characterized in that: The invariance loss is constructed by calculating the square error between the features generated by the model and the target features and normalizing them: 。 5. The method according to claim 1, characterized in that The standard deviation is calculated using the normalized feature data matrix: ； in, is the normalized feature data matrix No. The mean of the column, is the dimension of the feature, is a constant target value for the standard deviation.

6. The method according to claim 5, characterized in that The standard deviation loss is constructed based on the standard deviation: 。 7. The method according to claim 1, characterized in that The covariance regularization term is defined as the sum of the squares of the off-diagonal coefficients of the covariance matrix: ； in, is the Kronecker function, Representation variables The covariance matrix of Representation variables In the Row and The covariance values ​​of the columns, Representation variables The values ​​on the diagonal of the covariance matrix of .

8. The method according to claim 7, characterized in that The correlation loss is constructed based on the covariance regularization term: 。 9. The method according to claim 1, characterized in that: The invariance loss, standard deviation loss and correlation loss are weighted averaged to obtain the feature diversity preservation loss, including: The invariance loss, standard deviation loss and correlation loss are weighted averaged to obtain the feature diversity retention loss: ； in , and is a hyperparameter that controls the importance of each loss, is the invariance loss, is the standard deviation loss, is the correlation loss.

Images