A small sample image recognition method based on double gradient reinforcement contrast learning

Abstract

The application discloses a kind of based on double gradient reinforcement contrast learning small sample image recognition method, the method includes: obtaining image to be identified and carries out data enhancement processing, obtains double view sample and mixed sample;Double view sample and mixed sample are input to depth neural network model and gradient training is carried out, and global contrast loss function is constructed minimization;Based on global contrast loss function, gradient back propagation correction processing is carried out to the full connection layer of depth neural network model, and the depth neural network model after correction is constructed;Based on the depth neural network model after correction, image recognition processing is carried out, and image recognition result is obtained.By using the application, the identification accuracy of depth neural network model to image can be improved while reducing the cost of manual data sample labeling.The application can be widely applied to the field of neural network-based image recognition technology as a kind of based on double gradient reinforcement contrast learning small sample image recognition method.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology based on neural networks, and in particular to a few-sample image recognition method based on dual-gradient reinforcement contrastive learning. Background Technology

[0002] Because the external textures of coal and gangue are extremely ambiguous, visual differentiation is difficult. Therefore, in general coal and gangue sorting industrial sites, manual knocking and visual sorting are still the main methods. Current technologies mainly fall into three categories: machine learning based on digital images, manual inspection, and deep learning technology. Since the external characteristics of coal and gangue are extremely similar, manual sorting remains the primary method in industrial sites, leading to inefficiency, a high probability of misjudgment, and potential health hazards. Machine learning based on digital images requires combining manually extracted features with machine learning theory. However, manually extracted features are not optimal, and the resulting models are difficult to adapt to diverse industrial site samples. However, it suffers from poor generalization ability. Currently, self-supervised learning is mainly divided into contrastive learning and generative adversarial learning. Contrastive learning is a type of self-supervised learning used to train a model without labels by setting the similarity between positive and negative samples. This allows the model to mine the intrinsic features of the samples and encode high-order features sufficient to distinguish different objects. In this way, similar and different images can be distinguished without labels. Contrastive learning can avoid the problems of difficult sample acquisition and labeling errors in industrial sites. However, there are training processes, and the gradient vanishing model fails to achieve good fitting ability. Furthermore, current contrastive learning methods suffer from the gradient vanishing problem in the later stages of model training, which is not conducive to model optimization. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention aims to provide a few-sample image recognition method based on dual-gradient reinforcement contrastive learning, which can improve the image recognition accuracy of deep neural network models while reducing the cost of manual annotation of data samples.

[0004] The first technical solution adopted in this invention is: a few-sample image recognition method based on dual-gradient reinforcement contrastive learning, comprising the following steps:

[0005] The image to be identified is acquired and data augmentation processing is performed to obtain dual-view samples and mixed samples;

[0006] The dual-view samples and the mixed samples are input into a deep neural network model for gradient training, and a global contrastive loss function is constructed to minimize the global contrastive loss function.

[0007] Based on the global contrastive loss function, gradient backpropagation correction is performed on the fully connected layers of the deep neural network model to construct the corrected deep neural network model.

[0008] Image recognition results are obtained by performing image recognition processing based on the modified deep neural network model.

[0009] Furthermore, the step of acquiring the image to be recognized and performing data augmentation processing to obtain dual-view samples and mixed samples specifically includes:

[0010] The object to be identified is photographed and processed by a camera to obtain an image of the object to be identified.

[0011] The image to be recognized is read and processed using the OpenCV library function imread to obtain the read image.

[0012] The read image is centrally cropped to obtain the cropped image.

[0013] The cropped image is processed with random noise level, rotation, random cropping, color distortion and Gaussian blur to obtain dual-view samples;

[0014] The dual-view samples are selected and reconstructed to create hybrid samples.

[0015] Furthermore, the step of inputting dual-view samples and mixed samples into a deep neural network model for gradient training to construct a function that minimizes the global contrastive loss function specifically includes:

[0016] Dual-view samples and mixed samples are input into a deep neural network model, which includes a feature extractor, a projector, and a fully connected layer.

[0017] Based on the feature extractor, feature extraction processing is performed on dual-view samples and mixed samples to obtain feature encoding vectors for dual-view samples and feature encoding vectors for mixed samples;

[0018] Based on the projector, a global contrast loss calculation is performed on the feature encoding vectors of dual-view samples and the feature encoding vectors of mixed samples to obtain the linear similarity between dual-view samples and mixed samples.

[0019] Based on a fully connected layer, the global contrast loss function is optimized by combining the linear similarity between dual-view samples and mixed samples, thus constructing a global contrast loss function that minimizes the global contrast loss function.

[0020] Furthermore, the step of performing feature extraction processing on the dual-view samples and mixed samples based on the feature extractor to obtain the feature encoding vectors of the dual-view samples and the mixed samples specifically includes:

[0021] The dual-view samples and the mixed samples are input into the feature extractor, which includes a convolutional layer, a batch normalization layer, an activation function, a max pooling layer, a first residual block, a second residual block, a third residual block, a fourth residual block, and an average pooling layer.

[0022] Convolutional layers, batch normalization layers, activation functions, and max pooling layers are used to perform convolutional and pooling processes on dual-view samples and mixed samples, respectively, to obtain processed dual-view samples and processed mixed samples.

[0023] Based on the residual block network, the processed dual-view samples and the processed mixed samples are subjected to identity mapping to obtain the feature vectors of the dual-view samples and the feature vectors of the mixed samples.

[0024] The residual block network includes a first residual block, a second residual block, a third residual block, and a fourth residual block;

[0025] The feature vectors of the dual-view samples and the feature vectors of the mixed samples are transformed by the average pooling layer to obtain the feature encoding vectors of the dual-view samples and the feature encoding vectors of the mixed samples.

[0026] Furthermore, the step of performing identity mapping processing on the processed dual-view samples and the processed mixed samples based on the residual block network to obtain the feature vectors of the dual-view samples and the mixed samples specifically includes:

[0027] The processed dual-view samples and the processed mixed samples are sequentially input into the first residual block, the second residual block, the third residual block, and the fourth residual block in the residual block network. The first residual block includes a one-layer downsampled residual structure and two-layer skip residual structures. The second residual block includes a one-layer downsampled residual structure and three-layer skip residual structures. The third residual block includes a one-layer downsampled residual structure and five-layer skip residual structures. The fourth residual block includes a one-layer downsampled residual structure and two-layer skip residual structures.

[0028] Furthermore, the downsampling residual structure includes a convolutional layer, a batch normalization layer, an isocyanate layer, and an activation function, wherein:

[0029] Based on convolutional layers and batch normalization layers, feature dimension processing is performed on the image information of the processed dual-view samples and the processed mixed samples to obtain the corresponding multi-channel feature maps.

[0030] The multi-channel feature map is segmented and transformed based on the iso-segmentation layer to obtain the transformed segmented map.

[0031] The transformed segmented image is then subjected to convolutional fusion processing to obtain the downsampled image result.

[0032] Furthermore, the step of segmenting the multi-channel feature map based on the iso-segmentation layer to obtain the transformed segmented map specifically includes:

[0033] The multi-channel feature map is segmented based on the iso-segmentation layer to obtain the first segmentation map, the second segmentation map, the third segmentation map, and the fourth segmentation map;

[0034] The first segmented image is processed without feature transformation to obtain the transformed first segmented image;

[0035] The second segmented image is transformed by convolution to obtain the transformed second segmented image.

[0036] The transformed second and third segmented images are superimposed and convolved to obtain the transformed third segmented image.

[0037] The transformed third and fourth segmented images are superimposed and convolved to obtain the transformed segmented image.

[0038] Furthermore, the specific expression for minimizing the global contrastive loss function is as follows:

[0039] ;

[0040] In the above formula, Indicates the similarity between two-view samples. Indicates the similarity of the mixed samples. Indicates the number of mixed samples. This represents the representation of a batch of data-augmented samples after scaling factor mixing and encoder processing, where the feature vector A1+C1 is used as... hour, It represents A1. It represents C1. Representing the temperature coefficient is beneficial for the model to learn a softer representation. Both represent the representation of samples after being scaled and mixed using a scaling factor from a batch of data-augmented samples, and then encoded; however, their scaling factors are different from those of the previous two samples. The coefficients of 1 are added together. This represents the loss function used to calculate the similarity between two views. This indicates the calculation of the similarity between the mixed sample and the original image, and and The degree of similarity between them This represents the sum of the two, which is provided to the model as a training constraint as the objective function.

[0041] The beneficial effects of the method of this invention are as follows: This invention constructs positive sample pairs by performing dual-view data augmentation on the same sample. After calculating and extracting features, the positive samples are made as similar as possible to each other, and the negative samples are made as dissimilar as possible to each other. This serves as a proxy task, effectively reducing the need for a large number of labeled samples for model training under traditional supervised training. It solves the problem of current technologies using a large number of manually labeled samples and fitting the model with a large amount of labeled data, thus reducing the need for manual labor. By constructing mixed difficult samples, under the original method of measuring contrastive loss, the linear similarity relationship between the mixed samples and the dual-view samples is used as part of the proxy task to calculate and measure the similarity between samples. This is used as a constraint for the objective function, and the model is corrected through the objective function. This effectively improves the gradient degradation problem of existing contrastive learning, enhances sample utilization, and has a higher generalization ability for samples with high ambiguity, thereby improving the recognition accuracy of deep neural network models. Attached Figure Description

[0042] Figure 1 This is a flowchart of the steps of a few-sample image recognition method based on dual-gradient reinforced contrastive learning according to the present invention;

[0043] Figure 2 This is a schematic diagram illustrating the process of image recognition using a neural network structure model according to the present invention;

[0044] Figure 3 This is a structural block diagram of the feature extractor of the present invention;

[0045] Figure 4 This is a schematic diagram of the conventional isodifferential layer operation process of the present invention;

[0046] Figure 5 This is a schematic diagram of the operation flow of the isodifferentiated layer in the feature extractor of the present invention;

[0047] Figure 6 This is a schematic diagram of the structure of the skip-type residual module of the present invention;

[0048] Figure 7 This is a schematic diagram of the structure of the downsampling residual module of the present invention;

[0049] Figure 8 This is a schematic diagram comparing the similarity between dual-view samples and mixed samples according to the present invention;

[0050] Figure 9 This is a schematic diagram of the projector module of the present invention;

[0051] Figure 10 This is a schematic diagram of a coal gangue image field acquisition device in a specific embodiment of the present invention;

[0052] Figure 11This is a schematic diagram of coal gangue images collected in a specific embodiment of the present invention;

[0053] Figure 12 This is a schematic diagram of the image result recognition output by a specific embodiment of the present invention;

[0054] Figure 13 This is a schematic diagram of the process of image identification based on the Argsort function in this invention;

[0055] Figure 14 This is a schematic diagram showing the recognition accuracy curves of applying traditional methods and the deep neural network model applied in this invention. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0057] Reference Figure 1 and Figure 2 This invention provides a few-shot image recognition method based on dual-gradient reinforced contrastive learning, which includes the following steps:

[0058] S1. Acquire the image to be recognized and perform data augmentation processing to obtain dual-view samples and mixed samples;

[0059] Specifically, the present invention uses real coal and gangue data collected in industrial settings, and the collected coal and gangue images are as follows: Figure 11 As shown, the digital image sensor of MV-CA050-12UC, such as Figure 10 As shown, images of coal gangue are acquired on a mine production line. The acquired images are RGB images, which provide more semantic information than grayscale images. The images are read using the OpenCV library function imread. After data acquisition, the CV module performs central cropping on the read images to remove redundant background information. The acquired samples are then augmented with random noise level adjustment, rotation, random cropping, color distortion, and Gaussian blur to generate dual views. Gaussian blur is an image blur filter that calculates the transformation of each pixel in the image using a normal distribution. This blurring process preserves edge effects better than other equal blur filters, effectively achieving good boundary information preservation even in scenarios with reduced crop size.

[0060] In summary, dual-view samples are positive sample pairs formed by two random data augmentations, while mixed samples are obtained by linearly combining dual-view samples according to different mixing ratios.

[0061] S2. Input the dual-view samples and the mixed samples into the deep neural network model for gradient training, and construct the global contrastive loss function that minimizes the global contrastive loss function.

[0062] Specifically, after generating dual views, a batch of views is used to construct mixed hard samples. After passing through the feature extractor proposed in this invention, the data-enhanced dual-view samples and hard samples are encoded respectively. The main idea of ​​the encoder is to nonlinearly map the input image to a high-dimensional space, which can effectively distinguish each instance in the high-dimensional space. The encoder of this invention maps the (3, 32, 32) RGB image and finally outputs the (1, 2048) encoded feature vector. The output encoded features are used as features for the classification layer of the downstream task. In this invention example, the downstream task is coal gangue sorting task.

[0063] S21. An improved residual block structure by incorporating a multi-scale model encoder;

[0064] Specifically, ResNet50, a feedforward residual network and a current mainstream classifier network backbone, mainly consists of four stages, each group comprising (3, 4, 6, 3) blocks, as follows: Figure 4 As shown, its basic blocks consist of 1×1 and 3×3 convolutions, with residual structures embedded outside their input and output. Residual structures are a type of structure that effectively addresses the degradation problem of deep networks, transforming the fitting of multiple nonlinear variations in the input into fitting the residuals between the input and output, thus becoming an identity mapping. However, ResNet50 suffers from a relatively singular information extraction problem; each block contains only a single 3×3 convolution without multi-scale information learning. Therefore, the original 3x3 convolution is changed to... Figure 5 As shown;

[0065] Image information is first processed by convolutional kernels to form multi-channel feature maps, ensuring the image scale remains unchanged. Based on the channel dimension, the image is divided into four feature maps with equal number of channels: l1, l2, l3, and l4. Next, feature map l1 is used as an untransformed feature to obtain y1. Feature map l2 is transformed into k2 (also called y2) through convolution. Feature map l3 is superimposed on k2 and then convolved to transform into y3 (k3). Finally, l4 is superimposed on k3 and then convolved to transform into y4. In effect, the feature information of l2 undergoes 5×5 and 7×7 convolutions to obtain information at different scales. This method achieves the effect of other convolutional kernel sizes by using 3×3 convolutions, reducing a large number of redundant parameters. Finally, y1, y2, y3, and y4 are transformed into... Features at different scales are concatenated and then fused across different dimensions through convolution. Each convolution operation can potentially accept all feature information from its left side. With more channels segmented, more features and more receptive fields are learned. At the same time, multi-scale convolution is beneficial for integrating information at the feature level, increasing the number of information channels and enhancing the information intensity of gradient backpropagation. The multi-scale module divides the data into four equal parts and passes through convolution kernels of different sizes, which can extract sufficient semantic information and avoid the neglect of some original detailed features caused by single convolution. This invention uses this combination as one of the gradient enhancement methods in dual-gradient enhanced contrastive learning, which helps improve the recognition accuracy of samples in the feature extraction of this encoder.

[0066] S22. Improve contrastive learning to enhance gradients and improve model classification performance;

[0067] Specifically, this invention constructs positive sample pairs (vi, vi') by performing two random data augmentations on the samples. A feature encoder then extracts features to form two feature vectors. Since the positive sample pairs are generated from the same image, their inherent semantic information is consistent. Therefore, the purpose of the feature encoder is to extract the feature vectors of the positive sample pairs to make them as similar as possible, while keeping the remaining samples as dissimilar as possible from the original samples, as shown in the following formula:

[0068] ;

[0069] The numerator is the similarity between two positive samples, and the denominator is the similarity between negative samples. (vi, vi') represents the encoded features of the two views of image xi after data augmentation. The inner product of the two is used as the similarity index. sim() represents the inner product between samples. Based on the above objectives, a contrastive loss function is constructed.

[0070] To address the poor classification performance of contrastive learning during training due to gradient degradation, this invention constructs difficult samples through linear combination. This improves the model's sample utilization and enables the model to learn different similarities between the mixed samples and the original samples, enhancing the model's representation ability in fuzzy samples. The gradient enhancement module is mainly divided into two parts: one part enhances the gradient information backpropagation on the network through a multi-scale feature module, and the other part enhances the gradient strength during the gradient backpropagation stage through a hybrid contrastive loss.

[0071] like Figure 8 As shown, after the hybrid sample is constructed, it is also sent to the feature extraction module to obtain feature encoding. The similarity between it and the original sample is γi and 1-γi, respectively. The hard sample and the original sample have γ similarity and (1-γ) similarity. After A1 and C1 construct the hybrid hard sample, the features extracted by the encoder have a similarity in the proportional coefficient with A2 and C2 after data augmentation. At the same time, the hybrid hard sample is also essentially reconstructed by A1 and C1. Therefore, the intrinsic semantic information between the two is the same. The data after half-segmentation is defined as xi and xh. Therefore, the number of hybrid samples is defined as nhybrid, and the hybrid sample is xhybrid and the opposite sample xop.

[0072] Since the original contrastive loss function can only effectively distinguish clear categories without fully exploring the potential effective features of the data, it does not have good feature discrimination for highly ambiguous data such as coal and gangue. Therefore, this invention constructs mixed hard samples and compares the mixed samples with the original samples before mixing to form different similarities. This improves the sample utilization rate and expands the support for the training distribution, which improves the performance of the model in learning highly ambiguous data. At the same time, it optimizes the original contrastive loss function and improves the gradient vanishing problem and gradient degradation problem based on experimental results.

[0073] This invention obtains the encoded features of the original samples by data augmentation, and the encoded features reconstructed through hybridization are vi(hybrid) and vi(op). The similarity between vi(hybrid) and the original samples is γi and (1-γi), respectively. The intrinsic semantic information of the reconstructed samples vi(hybrid) and vi(op) is the same. In order to ensure that the model still maintains its ability to distinguish instances, the contrastive loss between the hybrid samples and the opposite samples is also included as part of the hybrid augmentation loss function. Extensive experiments have shown that the weight gradient backpropagation information in the network is enhanced during the gradient backpropagation stage. The objective function of the global contrastive loss is as follows:

[0074] ;

[0075] In the above formula, Indicates the similarity between two-view samples. Indicates the similarity of the mixed samples. Indicates the number of mixed samples. This represents the representation of a batch of data-augmented samples after scaling factor mixing and encoder processing, where the feature vector A1+C1 is used as... hour, It represents A1. It represents C1. Representing the temperature coefficient is beneficial for the model to learn a softer representation. Both represent the representation of samples after being scaled and mixed using a scaling factor from a batch of data-augmented samples, and then encoded; however, their scaling factors are different from those of the previous two samples. The coefficients of 1 are added together. This represents the loss function used to calculate the similarity between two views. This indicates the calculation of the similarity between the mixed sample and the original image, and and The degree of similarity between them This represents the sum of the two, which is provided to the model as a training constraint as the objective function.

[0076] S23. Based on the feature extractor, perform feature extraction processing on the dual-view samples and the mixed samples to obtain the feature encoding vector of the dual-view samples and the feature encoding vector of the mixed samples;

[0077] Specifically, refer to Figure 3 , Figure 6 and Figure 7 The input image is a 224*224 image. It is first processed by 64 convolutional kernels of 7×7 with a stride of 2, and the output is a 64-channel feature map. N: number of convolutional kernels (equal to the number of channels of convolution), W: width of input matrix, H: height of input matrix, K: convolutional kernel size, P: padding (number of zeros to be padded), S: stride. The input image format is 3*224*224, so W=224, H=224, convolutional kernel N=64, K=7, S=2, P=3.

[0078] width represents the width of the output matrix after convolution, and height represents the height of the output matrix after convolution;

[0079] width = (W - K + 2P) / S + 1 (rounded down) = (224 - 7 + 2 * 3) / 2 + 1 = 112

[0080] height = (H - K + 2P) / S + 1 (rounded down) = (224 - 7 + 2 * 3) / 2 + 1 = 112

[0081] The output after the first convolutional layer is 64*122*122;

[0082] Then, max pooling is performed. With input sizes W=112 and H=112, and keeping the number of kernels constant N=64, K=3, S=2, and P=1, the pooled size is:

[0083] width = (W - K + 2P) / S + 1 (rounded down) = (112 - 3 + 1 * 2) / 2 + 1 = 56

[0084] height = (H - K + 2P) / S + 1 (rounded down) = (112 - 3 + 1 * 2) / 2 + 1 = 56

[0085] The pooling layer output is 65*56*56;

[0086] The processing procedure for the first residual block is as follows:

[0087] Based on the first downsampling residual structure;

[0088] In the first downsampling residual structure, the input image is convolved by the downsampling structure as (1*1, output_channel=64, p=0, s=1).

[0089] width = (W - K + 2P) / S + 1 (rounded down) = (64 - 1 + 0 * 2) / 1 + 1 = 56

[0090] height = (W - K + 2P) / S + 1 (rounded down) = (64 - 1 + 0 * 2) / 1 + 1 = 56

[0091] The output image is 64*56*56; similarly, the downsampling branch outputs the same image but with 256 convolutional kernels, so the image output is 256*56*56; then the output features are divided into four equal parts into 4*(16*56*56) and passed through a convolutional kernel (3*3, output_channel=64, s=1, p=1).

[0092] After convolution and convolution of the information interaction between block 2 and block 3 as described above, as well as the information interaction between block 3 and block 4, the output image is (64*56*56) after being re-integrated by the number of channels. Then, it is passed through a convolution kernel (3*3, output_channel=256, s=1, p=1).

[0093] The final output image is (256, 56, 56), which is then added to the downsampling branch to complete the process.

[0094] In the first skip residual structure, the input image is convolved as (1*1, output_channel=64, s=1, p=0):

[0095] width = (W - K + 2P) / S + 1 (rounded down) = (64 - 1 + 0 * 2) / 1 + 1 = 56

[0096] height = (W - K + 2P) / S + 1 (rounded down) = (64 - 1 + 0 * 2) / 1 + 1 = 56

[0097] The final output image is (64, 56, 56). The image is divided into four equal parts (16, 56, 56) as L1, L2, L3, and L4, respectively. L1 remains unchanged and is also called Y1. L2 is then convolved with (3*3, output_channel=16, s=1, p=1) to output Y2 (16, 56, 56). L3 is added to Y2 and then convolved with (3*3, output_channel=16, s=1, p=1) to output Y3 (16, 56, 56). L4 is added to Y3 and then convolved with (3*3, output_channel=16, s=1, p=1) to output Y4 (16, 56, 56). Y1, Y2, Y3, and Y4 are added together along the channel dimension to output the image (64, 56, 56), which is then convolved with a (3*3, output_channel=256, s=1, p=1) kernel.

[0098] The final output image is (256, 56, 56). Similarly, in the second jump residual structure, the input image is convolved into (1*1, output_channel=64, s=1, p=0), and the final output image is (64, 56, 56).

[0099] The image is divided into four equal parts (16, 56, 56), designated L1, L2, L3, and L4 respectively. L1 remains unchanged and is also called Y1. L2 is then convolved with (3*3, output_channel=16, s=1, p=1) to output Y2 (16, 56, 56). L3 is added to Y2 and then convolved with (3*3, output_channel=16, s=1, p=1) to output Y3 (16, 56, 56). L4 is added to Y3 and then convolved with (3*3, output_channel=16, s=1, p=1) to output Y4 (16, 56, 56). Y1, Y2, Y3, and Y4 are added together along the channel dimension to output the image (64, 56, 56), which is then convolved with a (1*1, output_channel=256, s=1, p=0) kernel to finally output the image (256, 56, 56).

[0100] Based on the second residual block;

[0101] In the first downsampled residual structure, the input image is convolved by the downsampled structure as (1*1, output_channel=128, p=0, s=2):

[0102] width = (W - K + 2P) / S + 1 (rounded down) = (64 - 1 + 0 * 2) / 2 + 1 = 28

[0103] height = (W - K + 2P) / S + 1 (rounded down) = (64 - 1 + 0 * 2) / 2 + 1 = 28

[0104] The output image is 128*28*28;

[0105] The images output by the same downsampling branch are the same, but the number of convolution kernels is 512, so the image output is 512*28*28;

[0106] The image is divided into four equal parts (32, 28, 28), designated as L1, L2, L3, and L4 respectively. L2 is then convolved with (3*3, output_channel=32, s=1, p=1) to output Y2 (32, 28, 28). L3 is added to Y2 and then convolved with (3*3, output_channel=32, s=1, p=1) to output Y3 (32, 28, 28). L4 is added to Y3 and then convolved with (3*3, output_channel=32, s=1, p=1) to output Y3 (32, 28, 28). The output of Y1, Y2, Y3, and Y4 is (32, 28, 28) after passing through a convolution kernel (1*1, output_channel=512, s=1, p=0). The final output image is (512, 28, 28), which is then added to the downsampling branch.

[0107] In the first jump residual structure, the input image is convolved as (1*1, output_channel=128, s=1, p=0), and the final output image is (128, 28, 28).

[0108] The image is divided into four equal parts (32, 28, 28), designated as L1, L2, L3, and L4 respectively. L1 remains unchanged and is also called Y1. L2 is then convolved (3*3, output_channel=32, s=1, p=1) to output Y2 (32, 28, 28). L3 is added to Y2 and then convolved (3*3, output_channel=32, s=1, p=1) to output Y3 (32, 28, 28). L4 is added to Y3 and then convolved... The convolution (3*3, output_channel=32, s=1, p=1) outputs Y4 (32, 28, 28); Y1, Y2, Y3, and Y4 are added together along the channel dimension to output the image (128, 28, 28), which is then passed through a convolution kernel (3*3, output_channel=512, s=1, p=1) to finally output the image (512, 28, 28). The skip branch adds the original input image (512, 28, 28) to the forward output.

[0109] In the second residual skip structure, the input image is convolved as (1*1, output_channel=128, s=1, p=0), and the final output image is (128, 28, 28).

[0110] The image is divided into four equal parts (16, 56, 56), designated L1, L2, L3, and L4 respectively. L1 remains unchanged and is also called Y1. L2 is then convolved with (3*3, output_channel=16, s=1, p=1) to output Y2 (32, 28, 28). L3 is added to Y2 and then convolved with (3*3, output_channel=16, s=1, p=1) to output Y3 (32, 28, 28). L4 is added to Y3 and then convolved with (3*3, output_channel=16, s=1, p=1) to output Y3 (32, 28, 28). The convolution is (3*3, output_channel=16, s=1, p=1) to output Y4 (32, 28, 28). Y1, Y2, Y3, and Y4 are added together along the channel dimension to output the image (128, 28, 28). After passing through the convolution kernel (1*1, output_channel=512, s=1, p=0), the final output image is (512, 28, 28). The skip branch is to add the original input image (512, 28, 28) to the forward output.

[0111] In the third residual skip structure, the input image is convolved as (1*1, output_channel=128, s=1, p=0), and the final output image is (128, 28, 28).

[0112] The image is divided into four equal parts (16, 56, 56), designated L1, L2, L3, and L4 respectively. L1 remains unchanged and is also called Y1. L2 is then convolved with (3*3, output_channel=16, s=1, p=1) to output Y2 (32, 28, 28). L3 is added to Y2 and then convolved with (3*3, output_channel=16, s=1, p=1) to output Y3 (32, 28, 28). L4 is added to Y3 and then convolved with (3*3, output_channel=16, s=1, p=1) to output Y3 (32, 28, 28). The convolution is (3*3, output_channel=16, s=1, p=1) to output Y4 (32, 28, 28). Y1, Y2, Y3, and Y4 are added together along the channel dimension to output the image (128, 28, 28). After passing through the convolution kernel (1*1, output_channel=512, s=1, p=0), the final output image is (512, 28, 28). The skip branch is to add the original input image (512, 28, 28) to the forward output.

[0113] Based on the third residual block, similarly, the input image undergoes a downsampled residual structure and five skip residual structures;

[0114] In the first downsampling residual structure, the input image is convolved by the downsampling structure as (1*1, output_channel=256, p=0, s=2).

[0115] width = (W - K + 2P) / S + 1 (rounded down) = (28 - 1 + 0 * 2) / 2 + 1 = 14

[0116] height = (W - K + 2P) / S + 1 (rounded down) = (28 - 1 + 0 * 2) / 2 + 1 = 14

[0117] The output image is 256*14*14;

[0118] The downsampling branch outputs the same image, but with 1024 convolution kernels, resulting in an image output of 1024*14*14. The image is then divided into four equal parts (64, 14, 14) as L1, L2, L3, and L4, respectively. L2 undergoes a convolution (3*3, output_channel=64, s=1, p=1) to output Y2 (64, 14, 14). L3 is added to Y2 and then convolved with another convolution (3*3, output_channel=64, s=1, p=1) to output Y2 (64, 14, 14). The output of L4 (e.g., el=64, s=1, p=1) is Y3 (64, 14, 14). L4 is added to Y3 and then convolved to get (3*3, output_channel=64, s=1, p=1) and the output is Y4 (64, 14, 14). Y1, Y2, Y3 and Y4 are added together in terms of channel dimension to output an image of (256*14*14), which is then passed through a convolution kernel (1*1, output_channel=1024, s=1, p=0).

[0119] width = (W - K + 2P) / S + 1 (rounded down) = (28 - 1 + 0 * 2) / 1 + 1 = 28

[0120] height = (W - K + 2P) / S + 1 (rounded down) = (28 - 1 + 0 * 2) / 1 + 1 = 28

[0121] The output image is (1024, 14, 14), and the final output image is (1024, 14, 14), which is then added to the downsampling branch to complete the process.

[0122] In the first skip residual structure, the input image is downsampled and the convolution is (1*1, output_channel=256, p=0, s=1).

[0123] width = (W - K + 2P) / S + 1 (rounded down) = (14 - 1 + 0 * 2) / 1 + 1 = 14

[0124] height = (W - K + 2P) / S + 1 (rounded down) = (14 - 1 + 0 * 2) / 1 + 1 = 14

[0125] The output image is 256*14*14. The image is divided into four equal parts (64, 14, 14) as L1, L2, L3, and L4 respectively. L2 is then convolved by (3*3, output_channel=64, s=1, p=1) to output Y2 (64, 14, 14). L3 is added to Y2 and then convolved by (3*3, output_channel=64, s=1, p=1) to output Y3 (64, 14, 14). L4 is added to Y3 and then convolved by (3*3, output_channel=64, s=1, p=1) to output Y4 (64, 14, 14). Y1, Y2, Y3, and Y4 are added together along the channel dimension to output the image (256, 14, 14), which is then convolved by a kernel (1*1, output_channel=1024, s=1, p=0).

[0126] The output image is (1024, 14, 14), and the skip branch is the sum of the original input image (1024, 14, 14) and the forward output.

[0127] In the second skip residual structure, the input image is downsampled and the convolution is (1*1, output_channel=256, p=0, s=1):

[0128] The output image is 256*14*14. The image is divided into four equal parts (64, 14, 14) as L1, L2, L3, and L4 respectively. L2 is then convolved by (3*3, output_channel=64, s=1, p=1) to output Y2 (64, 14, 14). L3 is added to Y2 and then convolved by (3*3, output_channel=64, s=1, p=1) to output Y3 (64, 14, 14). L4 is added to Y3 and then convolved by (3*3, output_channel=64, s=1, p=1) to output Y4 (64, 14, 14). Y1, Y2, Y3, and Y4 are added together along the channel dimension to output the image (256, 14, 14), which is then convolved by a kernel (1*1, output_channel=1024, s=1, p=0).

[0129] The output image is (1024, 14, 14), and the skip branch is the sum of the original input image (1024, 14, 14) and the forward output.

[0130] In the third jump residual structure, the input image is downsampled and the convolution is (1*1, output_channel=256, p=0, s=1).

[0131] The output image is 256*14*14. The image is divided into four equal parts (64, 14, 14) as L1, L2, L3, and L4 respectively. L2 is then convolved by (3*3, output_channel=64, s=1, p=1) to output Y2 (64, 14, 14). L3 is added to Y2 and then convolved by (3*3, output_channel=64, s=1, p=1) to output Y3 (64, 14, 14). L4 is added to Y3 and then convolved by (3*3, output_channel=64, s=1, p=1) to output Y4 (64, 14, 14). Y1, Y2, Y3, and Y4 are added together along the channel dimension to output the image (256, 14, 14), which is then convolved by a kernel (1*1, output_channel=1024, s=1, p=0).

[0132] The output image is (1024, 14, 14), and the skip branch is the sum of the original input image (1024, 14, 14) and the forward output;

[0133] In the fourth jump residual structure, the input image is downsampled and the convolution is (1*1, output_channel=256, p=0, s=1).

[0134] The output image is 256*14*14. The image is divided into four equal parts (64, 14, 14) as L1, L2, L3, and L4 respectively. L2 is then convolved by (3*3, output_channel=64, s=1, p=1) to output Y2 (64, 14, 14). L3 is added to Y2 and then convolved by (3*3, output_channel=64, s=1, p=1) to output Y3 (64, 14, 14). L4 is added to Y3 and then convolved by (3*3, output_channel=64, s=1, p=1) to output Y4 (64, 14, 14). Y1, Y2, Y3, and Y4 are added together along the channel dimension to output the image (256, 14, 14), which is then convolved by a kernel (1*1, output_channel=1024, s=1, p=0).

[0135] The output image is (1024, 14, 14), and the skip branch is the sum of the original input image (1024, 14, 14) and the forward output;

[0136] In the fifth jump residual structure, the input image is downsampled and the convolution is (1*1, output_channel=256, p=0, s=1):

[0137] The output image is 256*14*14. The image is divided into four equal parts (64, 14, 14) as L1, L2, L3, and L4 respectively. L2 is then convolved by (3*3, output_channel=64, s=1, p=1) to output Y2 (64, 14, 14). L3 is added to Y2 and then convolved by (3*3, output_channel=64, s=1, p=1) to output Y3 (64, 14, 14). L4 is added to Y3 and then convolved by (3*3, output_channel=64, s=1, p=1) to output Y4 (64, 14, 14). Y1, Y2, Y3, and Y4 are added together along the channel dimension to output the image (256, 14, 14), which is then convolved by a kernel (1*1, output_channel=1024, s=1, p=0).

[0138] The output image is (1024, 14, 14), and the skip branch is the sum of the original input image (1024, 14, 14) and the forward output.

[0139] The fourth residual block, in stage four, consists of one downsampling residual block and two skip residual blocks.

[0140] In the first downsampled residual structure, the input image is convolved by the downsampled structure as (1*1, output_channel=512, p=0, s=2):

[0141] The output image is 512*7*7. Similarly, the downsampling branch outputs the same image, but with 2048 convolution kernels, resulting in an output image of 2048*7*7. The image is divided into four equal parts (128, 7, 7) as L1, L2, L3, and L4. L2 undergoes a convolution (3*3, output_channel=128, s=1, p=1) to output Y2 (128, 7, 7). L3 is added to Y2 and then convolved (3*3, output_channel=128, s=1, p=1) to output Y2 (128, 7, 7). The output of L4 (channel=128, s=1, p=1) is Y3 (128, 7, 7). L4 is added to Y3 and then convolved to get (3*3, output_channel=128, s=1, p=1) and the output is Y4 (128, 7, 7). Y1, Y2, Y3 and Y4 are added together in terms of channel dimension to output image (512*7*7), and then passed through a convolution kernel (1*1, output_channel=2048, s=1, p=0).

[0142] The output image is (2048, 7, 7), and the final output image is (2048, 7, 7), which is then added to the downsampling branch to complete the process.

[0143] In the first skip residual structure, the input image is downsampled and the convolution is (1*1, output_channel=512, p=0, s=1).

[0144] width = (W - K + 2P) / S + 1 (rounded down) = (7 - 1 + 0 * 2) / 1 + 1 = 7

[0145] height = (W - K + 2P) / S + 1 (rounded down) = (7 - 1 + 0 * 2) / 1 + 1 = 7

[0146] The output image is 512*7*7. It is divided into four equal parts (128, 7, 7) as L1, L2, L3, and L4. L2 is then convolved (3*3, output_channel=128, s=1, p=1) to output Y2 (128, 7, 7). L3 is added to Y2 and then convolved (3*3, output_channel=128, s=1, p=1) to output Y3 (128, 7, 7). L4 is added to Y3 and then convolved... The convolution (3*3, output_channel=128, s=1, p=1) outputs Y4 (128, 7, 7). Y1, Y2, Y3, and Y4 are added together along the channel dimension to output an image (512*7*7). After passing through a convolution kernel (1*1, output_channel=2048, s=1, p=0), the calculation process is similar, and the output image is (2048, 7, 7). The skip branch adds the original input image (2048, 7, 7) to the forward output.

[0147] In the second jump residual structure, the input image is downsampled and the convolution is (1*1, output_channel=512, p=0, s=1), and the output image is 512*7*7;

[0148] The image is divided into four equal parts (128, 7, 7), designated L1, L2, L3, and L4 respectively. L2 is then convolved with (3*3, output_channel=128, s=1, p=1) to output Y2 (128, 7, 7). L3 is added to Y2 and then convolved with (3*3, output_channel=128, s=1, p=1) to output Y3 (128, 7, 7). L4 is added to Y3 and then convolved with... The output of (3*3, output_channel=128, s=1, p=1) is Y4 (128, 7, 7). Y1, Y2, Y3 and Y4 are added together in terms of channel dimension to output an image of (512*7*7). After passing through a convolution kernel (1*1, output_channel=2048, s=1, p=0), the output image is (2048, 7, 7). The skip branch is to add the original input image (2048, 7, 7) to the forward output.

[0149] Finally, the feature vector (2048, 7, 7) is transformed into (2048, 1, 1) through a global average pooling layer, which is the feature encoding vector.

[0150] S24. Based on the projector, perform global contrast loss calculation on the feature encoding vectors of dual-view samples and mixed samples to obtain the linear similarity between dual-view samples and mixed samples.

[0151] Specifically, refer to Figure 9 During the training phase, the feature encoder needs to be combined with the projector. The projector is a multilayer perceptron, and its main structure is shown in the figure. It consists of a fully connected layer (2048, 512) that reduces the dimensionality of the encoded features to 512 dimensions. Then, it is input into the Batch Normalization (BN) layer. After batch normalization, the data distribution is prevented from changing. Then, the ReLU function is used to learn the complex relationships in the data. The dimensionality is reduced again by a fully connected layer (512, 128), and the BN layer is used to maintain the stability of the data distribution. The role of the projector is that since the surrogate task is the basis for model pre-training, directly using the features of the encoder for task training will cause the information extracted by the feature encoder to be mixed with too much information of the surrogate task. Therefore, using the features output by the projector for the surrogate task will leave more surrogate task feature information in the projector, while the feature encoding retains the general feature extraction information.

[0152] S3. Based on the global contrastive loss function, gradient backpropagation correction is performed on the fully connected layers of the deep neural network model to construct the corrected deep neural network model.

[0153] Specifically, this invention uses a training iterator with a batch size of 64 for 600 training rounds. After feeding unlabeled coal gangue images with a batch size of 64, along with constructed mixed hard samples and their counterexamples (hybrid_size 64), into the encoding feature generator and projector, the features are encoded separately, and then the global contrastive loss function is calculated. This loss constraint ensures that positive sample pairs are as similar as possible, negative samples are as dissimilar as possible, and the mixed samples have linear similarity to the original samples. By minimizing the global contrastive loss function, gradient backpropagation of the model weights completes the backbone network training in the pre-training stage. This invention uses stochastic gradient descent to optimize the model parameters, sets the learning rate to 0.003, and sets weight decay to prevent overfitting. The trained model is stored using the torch.save function in pth format.

[0154] After removing the projector, the feature extractor is combined with the fully connected layer as the network for the downstream task. Using `model.load_state_dict()`, the previously obtained pre-trained model parameters are read, and the feature extractor is frozen. This means that in the downstream training task, the feature extractor is no longer trained; only the fully connected layer is trained. The training layer is set to (2048, 2). The 2048-dimensional features are mapped to a binary classification of coal and gangue. This patent uses the Softmax function for classification. The Softmax function is an activation function that converts a number... The value vector is normalized into a probability distribution vector. This vector is then connected to the fully connected layer and the Softmax layer in the network to calculate the probability that the feature vector belongs to a certain class. The model is trained in a supervised manner using a small number of labeled coal gangue images, so that the output of the fully connected layer is adapted to the distribution of coal gangue data. In the downstream task, the Adam optimizer is used for training with a learning rate of 0.003. The cross-entropy function is used as the loss function. The model's correctness is evaluated by measuring the difference between the predicted value and the label value, and this is used as a function to adjust the weights of the fully connected layer for gradient backpropagation.

[0155] S4. Based on the modified deep neural network model, perform image recognition processing to obtain the image recognition result.

[0156] Specifically, refer to Figure 12 and Figure 13 After outputting the probability distribution, the Argsort function in Python is used to label and output the position of the value with the highest probability. A dictionary is built with the pre-defined position 0 representing coal and position 1 representing gangue. The Argsort function is used to identify the elements. Finally, the processed image is output as the original image, and the category is output to the user after being mapped by the dictionary.

[0157] Reference Figure 14 The pre-trained model is fine-tuned after every 100 training rounds, and its accuracy is shown in the line graph below. Figure 13 As shown, its highest accuracy rate can reach 85.4%.

[0158] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A few-sample image recognition method based on dual-gradient reinforcement contrastive learning, characterized in that, Includes the following steps: The image to be identified is acquired and data augmentation processing is performed to obtain dual-view samples and mixed samples; Among them, the dual-view sample is a positive sample pair formed by two random data augmentations, and the mixed sample is obtained by linearly combining the dual-view samples according to different mixing ratio coefficients; ； ； In the above formula, Indicates a mixed sample. This represents a dual-view sample. Indicates the similarity between the mixed sample and the original sample. , This represents the data after splitting into two halves; The dual-view samples and the mixed samples are input into a deep neural network model, which includes a feature extractor, a projector and a fully connected layer. The feature extractor includes a convolutional layer, a batch normalization layer, an activation function, a max pooling layer, a first residual block, a second residual block, a third residual block, a fourth residual block and an average pooling layer. The residual block includes a downsampling residual structure and a skip residual structure; The downsampling residual structure includes a convolutional layer, a batch normalization layer, an isotropic layer, and an activation function; Among them, the image information of the processed dual-view samples and the processed mixed samples are processed by feature dimension processing based on convolutional layers and batch normalization layers to obtain the corresponding multi-channel feature maps. Based on the isomorphic layer, the multi-channel feature map is segmented, superimposed, and transformed by convolution to obtain the transformed segmented map. The transformed segmented image is then subjected to convolutional fusion processing to obtain the downsampled image result. Based on the feature extractor, feature extraction processing is performed on dual-view samples and mixed samples to obtain feature encoding vectors for dual-view samples and feature encoding vectors for mixed samples; Based on the projector, a global contrast loss calculation is performed on the feature encoding vectors of dual-view samples and the feature encoding vectors of mixed samples to obtain the linear similarity between dual-view samples and mixed samples. Based on the fully connected layer, the global contrast loss function is optimized by combining the linear similarity between dual-view samples and mixed samples, and a global contrast loss function that minimizes the global contrast loss function is constructed. The specific expression for minimizing the global contrastive loss function is as follows: ； In the above formula, This represents the loss function used to calculate the similarity between two views. This represents the similarity loss function used to calculate the difference between the mixed sample and the original image, and between the mixed sample and the two views. This represents the sum of the two, which is provided to the model as a training constraint as the objective function; Based on the global contrastive loss function, gradient backpropagation correction is performed on the fully connected layers of the deep neural network model to construct the corrected deep neural network model. Image recognition results are obtained by performing image recognition processing based on the modified deep neural network model.

2. The few-sample image recognition method based on dual-gradient reinforcement contrastive learning according to claim 1, characterized in that, The step of acquiring the image to be identified and performing data augmentation processing to obtain dual-view samples and mixed samples specifically includes: The object to be identified is photographed and processed by a camera to obtain an image of the object to be identified. The image to be recognized is read and processed using the OpenCV library function imread to obtain the read image. The read image is centrally cropped to obtain the cropped image. The cropped image is processed with random noise level, rotation, random cropping, color distortion and Gaussian blur to obtain dual-view samples; The dual-view samples are selected and reconstructed to create hybrid samples.

3. The few-sample image recognition method based on dual-gradient reinforcement contrastive learning according to claim 2, characterized in that, The step of performing feature extraction processing on dual-view samples and mixed samples based on a feature extractor to obtain feature encoding vectors for dual-view samples and mixed samples specifically includes: The dual-view samples and the mixed samples are input into the feature extractor, which includes a convolutional layer, a batch normalization layer, an activation function, a max pooling layer, a first residual block, a second residual block, a third residual block, a fourth residual block, and an average pooling layer. Convolutional layers, batch normalization layers, activation functions, and max pooling layers are used to perform convolutional and pooling processes on dual-view samples and mixed samples, respectively, to obtain processed dual-view samples and processed mixed samples. Based on the residual block network, the processed dual-view samples and the processed mixed samples are subjected to identity mapping to obtain the feature vectors of the dual-view samples and the feature vectors of the mixed samples. The residual block network includes a first residual block, a second residual block, a third residual block, and a fourth residual block; The feature vectors of the dual-view samples and the feature vectors of the mixed samples are transformed by the average pooling layer to obtain the feature encoding vectors of the dual-view samples and the feature encoding vectors of the mixed samples.

4. The few-sample image recognition method based on dual-gradient reinforcement contrastive learning according to claim 3, characterized in that, The step of performing identity mapping on the processed dual-view samples and the processed mixed samples based on the residual block network to obtain the feature vectors of the dual-view samples and the mixed samples specifically includes: The processed dual-view samples and the processed mixed samples are sequentially input into the first residual block, the second residual block, the third residual block, and the fourth residual block in the residual block network. The first residual block includes a one-layer downsampled residual structure and two-layer skip residual structures. The second residual block includes a one-layer downsampled residual structure and three-layer skip residual structures. The third residual block includes a one-layer downsampled residual structure and five-layer skip residual structures. The fourth residual block includes a one-layer downsampled residual structure and two-layer skip residual structures.

5. The few-sample image recognition method based on dual-gradient reinforcement contrastive learning according to claim 4, characterized in that, The step of segmenting the multi-channel feature map based on the iso-segmentation layer to obtain the transformed segmented map specifically includes: The multi-channel feature map is segmented based on the iso-segmentation layer to obtain the first segmentation map, the second segmentation map, the third segmentation map, and the fourth segmentation map; The first segmented image is processed without feature transformation to obtain the transformed first segmented image; The second segmented image is transformed by convolution to obtain the transformed second segmented image. The transformed second and third segmented images are superimposed and convolved to obtain the transformed third segmented image. The transformed third and fourth segmented images are superimposed and convolved to obtain the transformed segmented image.

6. The few-sample image recognition method based on dual-gradient reinforcement contrastive learning according to claim 5, characterized in that, The specific expression for minimizing the global contrastive loss function is as follows: ； ； In the above formula, Indicates the similarity between two-view samples. Indicates the similarity of the mixed samples. Indicates the number of mixed samples. This represents the representation of a batch of data-augmented samples after scaling and mixing with a coefficient, processed by an encoder, where the feature vector A1+C1 is used as... hour, It represents A1. It represents C1. Representing the temperature coefficient is beneficial for the model to learn a softer representation. Both represent the representation of samples after being scaled and mixed using a scaling factor from a batch of data-augmented samples, and then encoded; however, their scaling factors are different from those of the previous two samples. The coefficients of 1 are added together. This represents the loss function used to calculate the similarity between two views. This indicates the calculation of the similarity between the mixed sample and the original image, and and The degree of similarity between them This represents the sum of the two, which is provided to the model as a training constraint as the objective function.

Images