Mask self-supervision image recognition analysis method of improved DINO model

By introducing masking mechanism and local feature enhancement learning strategy into the DINO model, the shortcomings of the basic DINO model in image fine-grained features and complex background recognition are solved, and the accuracy and model stability of image recognition are significantly improved.

CN118941925APending Publication Date: 2024-11-12NANJING TECH UNIV

Patent Information

Application Number
CN202411102623.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The basic DINO model has information loss and ambiguity problems when processing the fine-grained features and local information of the image, and the recognition accuracy and robustness are insufficient in the absence of scenes or complex backgrounds.

Method used

A masking mechanism and local feature enhancement learning strategy are introduced, and the DINO network structure is optimized through random masking and multi-scale feature extraction to improve the accuracy and generalization ability of image recognition.

Benefits of technology

It significantly improves the accuracy of image recognition and the generalization ability of the model, can capture details and semantic features in the image more accurately, and enhances the recognition performance under complex backgrounds and diverse data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941925A_ABST
    Figure CN118941925A_ABST
Patent Text Reader

Abstract

The invention relates to the fields of deep learning, computer vision and self-supervised learning, in particular to a mask self-supervised image recognition analysis method of an improved DINO model, which comprises the following steps of: firstly, acquiring and processing an image recognition data set, and performing model training by using a public data set (ImageNet-lk, ImageNet 100, CIFAR-10 and CIFAR-100) and a private data set; and secondly, constructing a mask self-supervision model, based on a Vision Transform model, introducing a random mask mechanism, performing mask processing on the image blocks, and improving the model performance through a local feature enhancement strategy. Secondly, training the improved DINO model by using a plurality of data sets, and setting model parameters and a training strategy; and finally, the trained model is evaluated and adjusted, the recognition precision is evaluated through a test set, and parameter optimization is performed according to a result so as to improve the performance of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of deep learning, computer vision and self-supervised learning, and in particular to a masked self-supervised image recognition and analysis method based on an improved DINO model. Background Art

[0002] In recent years, image recognition technology based on self-supervised learning has gradually become a research hotspot in the field of computer vision. Self-supervised learning enables the model to learn rich feature representations from unlabeled data by designing clever tasks and network architectures without the expensive cost of manual annotation. The current popular image recognition methods, such as the YOLO model, often rely on manually annotated labels, which not only takes a lot of time and money, but also the quality of image annotation is uneven, which greatly affects the accuracy of image recognition.

[0003] With the development of deep learning technology, especially the introduction of Transformer model, self-supervised image recognition has shown significant advantages in processing large-scale data. As an emerging self-supervised learning method, DINO (Self-Distillation with No Labels) model uses self-distillation to guide the learning process through the teacher network, so as to effectively learn the semantic information and high-level features in the image without labels.

[0004] However, the basic DINO model suffers from information loss and ambiguity when processing fine-grained features and local information in images. In addition, the accuracy and robustness of the basic DINO model in distinguishing different categories also need to be further improved for image recognition with partially missing or damaged scenes or complex backgrounds.

[0005] Therefore, in order to solve the shortcomings of the above-mentioned basic DINO model, this paper proposes an improved DINO model mask self-supervised image recognition and analysis method, aiming to overcome the limitations of the basic DINO model. Summary of the invention

[0006] In order to improve the shortcomings of the existing DINO model in image recognition and analysis, the present invention proposes a masked self-supervised image recognition and analysis method of the improved DINO model. By introducing a mask mechanism and a local feature enhancement learning strategy, this method can better capture the detail information and semantic features in the image, while improving the recognition accuracy of partially missing and damaged image scenes and complex backgrounds.

[0007] At the same time, the improved DINO network structure is used to optimize the model's training process, enabling it to handle large-scale data in complex scenarios, effectively respond to challenges in image recognition, and further improve the accuracy and generalization ability of image recognition.

[0008] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0009] A masked self-supervised image recognition and analysis method based on an improved DINO model, characterized by comprising the following steps:

[0010] 1) Acquisition and processing of image recognition datasets: Use public datasets (such as ImageNet-1k, ImageNet100, CIFAR-10, CIFAR-100) and private datasets for training, and select corresponding datasets for training according to different image recognition tasks.

[0011] 2) Construction of mask self-supervised model: Based on the Vision Transformer model, a random mask mechanism is introduced to mask image blocks. Normalization, attention layer calculation, residual connection, DropPath operation and MLP layer forward propagation are adopted. Different convolution kernels are used to extract multi-scale features from the input feature map through the local feature enhancement strategy, and finally a masked output tensor is generated.

[0012] 3) Use multiple data sets to train and test the improved DINO model: input the training sets of the public data set and the private data set into the improved DINO model, set the model parameters and training strategy, perform forward propagation and data enhancement operations, and save the model weights after training is completed;

[0013] 4) Evaluate and adjust the improved DINO model: Test the trained model by inputting the test sets of the corresponding public and private datasets, evaluate the recognition accuracy, analyze the problems and bottlenecks based on the model performance, and further adjust the model parameters to improve performance.

[0014] As a further improvement of the present invention, the private data set in step 1) is captured by a dynamic visual sensor camera, which consists of four categories in total, including pedestrians, cars, bicycles, and electric vehicles. 1 to 3 minutes of video are taken by different visual sensor cameras, and then the video is framed, cleaned and classified.

[0015] As a further improvement of the present invention, the normalization formula in step 2) is:

[0016] X norm1 =LayerNorm(x) (1)

[0017] The calculation formula of the attention layer in step 2) is:

[0018] Q, K, V = X norm1 *W q , Xnorm1 *W k ,X norm1 *W v (2)

[0019] The attention matrix is:

[0020]

[0021] The attention output is:

[0022] y=A*V (4)

[0023] The residual connection and DropPath operation in step 2 are as follows:

[0024] x=x+DropPath(y) (5)

[0025] Wherein "x" is normalized and forward propagated through the MLP layer, and residual connection and DropPath operation are performed again; the step 2) is secondary normalized:

[0026] X norm2 =LayerNorm(x) (6)

[0027] Input the normalized x into the MLP layer:

[0028] z 1 =GELU(X norm2 *W 1 +b 1 ) (7)

[0029] z 2 =z 1 *W 2 +b 2 (8)

[0030] Perform residual connection and DropPath operations

[0031] x=x+DropPath(z 2 ) (9)

[0032] The formula of the random mask method in step 2) is:

[0033] x=x*(1-mask.unsqueeze(-1))+fill value *mask.unsequeeze(-1) (10)

[0034] As a further improvement of the present invention, the local feature enhancement strategy in step 2) uses different convolution kernels to perform multi-scale feature extraction on the input feature map, and finally generates a masked output tensor

[0035] The convolution kernel convO is expressed as:

[0036]

[0037] Among them, m is the batch index, d corresponds to the group index in the group convolution, h and w are the height and width indexes, k is the convolution kernel size, which we set to 5 here, in represents the input feature map, and weight represents the convolution kernel weight. For each group, each input channel has a corresponding convolution kernel for convolution operation, and each channel is independent.

[0038] The convolution kernel conv_spatial is expressed as:

[0039]

[0040] Among them, c' is the output channel index, c is the input channel index, ranging from 0 to C-1, where C is the number of input channels, d represents the hole rate, which is 3 here, and the convolution kernel size k is set to 7 here. For each output channel, the features of all input channels are multiplied by the corresponding weights and then summed to obtain the output features of that position. Different from ordinary convolution, the hole convolution adds holes or intervals to the convolution kernel, thereby increasing the receptive field of the convolution kernel without increasing the number of parameters and computational complexity.

[0041] As a further improvement of the present invention, in step 3), the initial model parameters are set to an initial learning rate of 0.0005, the learning rate decreases exponentially with the number of training times, the batch size is 64, the network is trained for 100 epochs, and finally the trained model weights are saved.

[0042] As a further improvement of the present invention, the step 4) uses a linear test model to evaluate the weights of the trained model, analyzes the performance of the model based on the recognition accuracy results, adjusts the model parameters according to the problems and bottlenecks, and further optimizes the model performance.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention uses an improved DINO model for masked self-supervised image recognition analysis. By introducing a mask mechanism and a local feature enhancement strategy, the accuracy of image recognition and the generalization ability of the model are greatly improved. The mask mechanism enables the model to learn more robust feature representations during training, thereby maintaining efficient recognition performance when processing incomplete or noisy image data. The local feature enhancement strategy enables the model to more accurately capture local and global information in the image and reduce information loss by performing multi-scale processing and weight optimization on the feature map.

[0045] In addition, the present invention adopts an improved DINO architecture based on a multi-head self-attention mechanism, which enhances the adaptability and stability of the model on complex backgrounds and diverse image data. By optimizing the mask ratio and feature fusion method, the present invention achieves more efficient feature extraction and representation capabilities, making the model more balanced and superior in performance on different data sets.

[0046] Compared with the traditional DINO model, the improvements in this invention make the model more stable and accurate when processing high-dimensional and complex scene images. Through reasonable mask design and feature enhancement strategy, this invention shows significant advantages in image recognition tasks, significantly improving the training efficiency and recognition accuracy of the model.

[0047] In general, the present invention has achieved significant improvements in image recognition accuracy, model stability and training efficiency, and can better meet the needs of practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 It is a diagram of the steps of mask self-supervised image recognition analysis of the present invention;

[0050] Figure 2 It is a diagram of the improved DINO model mask self-supervised model image recognition analysis method proposed by the present invention;

[0051] Figure 3 It is a schematic diagram of the overall process of the improved DINO model of the present invention;

[0052] Figure 4 This is the network structure diagram of the improved DINO model of the present invention DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] like Figure 1 As shown, the mask self-supervised image recognition and analysis method of the improved DINO model in this embodiment specifically includes the following steps:

[0055] S1. Acquisition and processing of image recognition datasets

[0056] The present invention uses public data sets and supplementary private data sets for training. The public data sets can be directly obtained from the official website. The public data sets used in the present invention include ImageNet-1k, ImageNet100, CIFAR-100, and CIFAR-10. According to different image recognition tasks, different public data sets are selected for training. The ImageNet-1k data set contains 1000 categories, a total of 1281167 training images, 50000 verification images, and 100000 test images, covering most daily objects and meeting most image recognition tasks. ImageNet100 is a sub-dataset of ImageNet, which randomly selects 100 categories from 1000 categories. It can replace the complete ImageNet-1k data set to quickly evaluate model performance. CIFAR-10 contains 10 categories, each category contains 6000 images of size 32×32. CIFAR-100 is a supplement to the CIFAR-10 dataset. It contains 100 categories of images, each category contains 600 pictures. The images in the CIFAR dataset are from real-world scenes and come from different photographers, and there is no duplication in each photo.

[0057] When performing image recognition for a specific scene, if this category does not exist in the public dataset, a private dataset should be collected and organized. The private dataset is captured by a dynamic visual sensor camera and consists of four categories, including pedestrians, cars, bicycles, and electric vehicles. 1 to 3 minutes of video is taken through different visual sensor cameras, and then the video is framed, cleaned and classified.

[0058] S2: Construction of masked self-supervised model

[0059] like Figure 2 As shown, the mask self-supervised model of the present invention is based on the Vision Transformer model, and is an improved mask self-supervised model of the DINO model. It mainly includes two modules. One is to introduce a mask mechanism in the Vision Transformer model. The present invention designs a random mask structure, that is, after the image enters the Transforemer model for block division, each Block is masked. The specific process is to normalize the input tensor "x" and perform self-attention calculation through the attention layer;

[0060] Normalize x:

[0061] X norm1 =LayerNorm(x) (1)

[0062] Attention Layer:

[0063] Q, K, V = X norm1 *W q , X norm1 *W k , X norm1 *W v (2)

[0064] The attention matrix is:

[0065]

[0066] The attention output is:

[0067] y=A*V (4)

[0068] Then perform residual connection and DropPath operation, and add the result to the input "x";

[0069] x=x+DropPath(y) (5)

[0070] Normalize "x" and forward propagate through the MLP layer, and perform residual connection and DropPath operation again;

[0071] Normalize x again:

[0072] X norm2 =LayerNorm(x) (6)

[0073] Input the normalized x into the MLP layer:

[0074] z 1 =GELU(X norm2 *W 1 +b 1 ) (7)

[0075] z 2 =z 1 *W 2 +b 2 (8)

[0076] Perform residual connection and DropPath operations

[0077] x=x+DropPath(z 2 ) (9)

[0078] Call the random mask method to perform random masking to get the final output tensor "x"

[0079] x=x*(1-mask.unsqueeze(-1))+fill value*mask.unsequeeze(-1) (10)

[0080] The specific mask operation is to set a mask ratio of 20%, calculate the number of tokens to be retained, generate a random noise matrix, sort the noise matrix to get the index, then restore the order of the index, retain the index of the number of tokens we need, generate the corresponding mask sequence, and fill the masked part with value Uniform padding, generating masked tensors by restoring the original indices.

[0081] The second is the local feature enhancement learning strategy;

[0082] The present invention processes the input feature map by using two different convolution kernels to extract features from multiple scales, thereby more fully capturing the information of the image.

[0083] The convolution kernel convO is expressed as:

[0084]

[0085] Among them, m is the batch index, d corresponds to the group index in the group convolution, h and w are the height and width indexes, k is the convolution kernel size, which we set to 5 here, in represents the input feature map, and weight represents the convolution kernel weight. For each group, each input channel has a corresponding convolution kernel for convolution operation, and each channel is independent.

[0086] The convolution kernel conv_spatial is expressed as:

[0087]

[0088] Among them, c' is the output channel index, c is the input channel index, ranging from 0 to C-1, where C is the number of input channels, d represents the hole rate, which is 3 here, and the convolution kernel size k is set to 7 here. For each output channel, the features of all input channels are multiplied by the corresponding weights and then summed to obtain the output features of that position. Different from ordinary convolution, the hole convolution adds holes or intervals to the convolution kernel, thereby increasing the receptive field of the convolution kernel without increasing the number of parameters and computational complexity.

[0089] Next, the two attention maps attnl and attn2 obtained in the first step are further processed by conv1 and conv2 respectively. The above two convolutional layers are 1×1 convolutions to achieve channel interaction and information fusion. Then the two processed feature maps are spliced ​​together along the channel dimension, the average and maximum values ​​of each channel are calculated, the two values ​​are spliced ​​together, and the attention weights are generated by the conv_squeeze convolution layer and a sigmoid function. Finally, these weights are applied to the original feature map.

[0090] S3: Train and test the improved DINO model using multiple datasets

[0091] like Figure 3 As shown, the overall process diagram of the mask self-supervision model of the improved DINO model in this embodiment; Figure 4 Network structure diagram for the improved DINO model.

[0092] According to different image recognition tasks, the training sets of public and private datasets are respectively input into the improved DINO model. After the image is enhanced, the vit_tiny function is called to create a VisionTransformer model instance. The model parameters include patch_size=16, embedding dimension embed_dim=192, depth=12, number of attention heads num_heads=3, and MLP ratio mlp_ratio=4. The input image 'x' is passed into the model for forward propagation. The PatchEmbed layer is called to slice the input image into patches of fixed size and linearly embed them. The local feature enhancement learning strategy is used to extract features from multiple scales to more fully capture the image information. The cls_token is added to the embedded patch tokens, and the position encoding is added to each token. The tokens are dropout, and then each Block is masked. After all Blocks are processed, the results are finally normalized, and the image recognition classification information is extracted for final output. Set the initial learning rate to 0.0005. The learning rate value decreases exponentially with the number of training times. Set the batch size to 64. Stop the network training after 100 epochs. Finally, save the trained model weights.

[0093] S4: Evaluate and adjust the improved DINO model

[0094] The weights of the training results are tested, and the corresponding public data sets and private data sets are input into the linear test model to test the final recognition accuracy of the model. Based on the recognition accuracy of the model, the problems and bottlenecks of the model are analyzed, and the parameters are further adjusted according to the performance of the model to improve the model performance.

[0095] Finally, it is noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A masked self-supervised image recognition and analysis method based on an improved DINO model, characterized in that: The following steps are involved: 1) Acquisition and processing of image recognition datasets: Use public datasets (such as ImageNet-1k, ImageNet100, CIFAR-10, CIFAR-100) and private datasets for training, and select corresponding datasets for training according to different image recognition tasks. 2) Construction of mask self-supervised model: Based on the Vision Transformer model, a random mask mechanism is introduced to mask image blocks. Normalization, attention layer calculation, residual connection, DropPath operation and MLP layer forward propagation are adopted. Different convolution kernels are used to extract multi-scale features from the input feature map through the local feature enhancement strategy, and finally a masked output tensor is generated. 3) Use multiple data sets to train and test the improved DINO model: input the training sets of the public data set and the private data set into the improved DINO model, set the model parameters and training strategy, perform forward propagation and data enhancement operations, and save the model weights after training is completed; 4) Evaluate and adjust the improved DINO model: Test the trained model by inputting the test sets of the corresponding public and private datasets, evaluate the recognition accuracy, analyze the problems and bottlenecks based on the model performance, and further adjust the model parameters to improve performance.

2. According to the improved DINO model of claim 1, the masked self-supervised image recognition and analysis method is characterized in that The private data set is constructed according to the required specific scene. The private data set used in the present invention is captured by a dynamic visual sensor camera and consists of four categories in total, including pedestrians, cars, bicycles, and electric vehicles. 1 to 3 minutes of video is taken by different visual sensor cameras, and then the video is framed, cleaned and classified.

3. The masked self-supervised image recognition and analysis method of the improved DINO model according to claim 1, characterized in that: The normalization formula in step 2) is: X norm1 =LayerNorm(x) (1) The calculation formula of the attention layer in step 2) is: Q,K,V=X norm1 *W q ,X norm1 *W k ,X norm1 *W v (2) The attention matrix is: The attention output is: y=A*V (4) The residual connection and DropPath operation in step 2 are as follows: x=x+DropPath(y) (5) The normalized "x" is forward propagated through the MLP layer, and the residual connection and DropPath operation are performed again; The step 2) is secondary normalization: X norm2 =LayerNorm(x) (6) Input the normalized x into the MLP layer: z1=YELLOW(X norm2 *W1+b1) (7) z2=z1*W2+b2 (8) Perform residual connection and DropPath operation x=x+DropPath(z2) (9) The formula for calling the random mask method in step 2) is: x=x*(1-mask.unsqueeze(-1))+fill value *mask.unsequeeze(-1) (10) The local feature enhancement strategy in step 2 uses different convolution kernels to extract multi-scale features from the input feature map, and finally generates a masked output tensor. The convolution kernel conv0 is expressed as: Among them, m is the batch index, d corresponds to the group index in the group convolution, h and w are the height and width indexes, k is the convolution kernel size, which we set to 5 here, in represents the input feature map, and weight represents the convolution kernel weight. For each group, each input channel has a corresponding convolution kernel for convolution operation, and each channel is independent. The convolution kernel conv_spatial is expressed as: Among them, c' is the output channel index, c is the input channel index, ranging from 0 to C-1, where C is the number of input channels, d represents the hole rate, which is 3 here, and the convolution kernel size k is set to 7 here. For each output channel, the features of all input channels are multiplied by the corresponding weights and then summed to obtain the output features of that position. Different from ordinary convolution, the hole convolution adds holes or intervals to the convolution kernel, thereby increasing the receptive field of the convolution kernel without increasing the number of parameters and computational complexity.

4. The masked self-supervised image recognition and analysis method of the improved DINO model according to claim 1, characterized in that: In step 3), the initial model parameters are set to an initial learning rate of 0.0005, the learning rate decreases exponentially with the number of training times, the batch size is 64, the network is trained for 100 epochs, and finally the trained model weights are saved.

5. The masked self-supervised image recognition and analysis method of the improved DINO model according to claim 1, characterized in that: Use the linear test model to evaluate the weights of the trained model, analyze the model performance based on the recognition accuracy results, adjust the model parameters according to the problems and bottlenecks, and further optimize the model performance.

Citation Information

Patent Citations

  • Colorectal cancer focus area lightweight segmentation method based on deep learning

    CN115409846A

  • Multi-scene-oriented intelligent manufacturing defect detection large model training method

    CN118334491A

  • Pollutant high-precision target detection method and system based on river information guidance

    CN118429622A

  • Cluster-based histopathology phenotype representation learning by self-supervised multi-class token hierarchical vision transformer

    WO2024124071A1

Cited By

  • Visual self-supervised learning-based high-altitude operation personnel operation specification detection method

    CN121354034A