Image recognition method based on multi-view knowledge distillation

Abstract

The present invention relates to the field of data processing technology, and in particular to an image recognition method based on multi-view knowledge distillation, comprising: obtaining multiple imaging views of a single target at different viewing angles through an imaging device; performing parallel feature extraction of a convolution auxiliary branch and an attention auxiliary branch on a teacher network for multiple views; inputting the auxiliary features generated by the two branches in parallel into a multi-view alignment weighting module for feature fusion between corresponding positions of the multiple views; predicting the category probability of the target under a single viewing angle and multiple viewing angles using the fused feature vector and using real labels for supervision; performing prototype feature extraction of multiple views by a student model; and training the student network under the supervision of real labels and teacher predicted labels using a teacher-student training framework to obtain a lightweight multi-view model. The problem of large number of parameters, high training and reasoning costs, and slow reasoning speed in the current multi-view recognition model is solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an image recognition method based on multi-view knowledge distillation. Background Art

[0002] Most of the current image recognition methods are based on traditional single-view methods, which results in a single perspective for image information acquisition. The recognition accuracy of the target is affected by the quality of a single imaging. Especially in the field of 3D modeling and medical diagnosis, single-view imaging is difficult to represent the image depth in the 3D modeling field well. In medical diagnosis, single-view imaging may be affected by artifacts and noise, or there may be organ occlusion problems. These problems result in single-view medical diagnosis not having high credibility and accuracy. In the field of 3D modeling and medical diagnosis of targets, in order to ensure the accuracy of the effect, people usually use multiple perspectives and multiple observations to achieve target modeling or medical target diagnosis. In the study of multi-view recognition methods, most studies use a multi-model method to deal with the increased image stream. Due to the increase in image streams and the increase in model size, the number of model parameters and the amount of calculation increase exponentially. The reasoning of the model often requires a large amount of time and computational overhead, which brings challenges to the multi-view real-time image recognition task.

[0003] Therefore, the traditional single-view recognition method has the problems of poor recognition effect and low credibility, and the traditional multi-view recognition method has the problems of large model parameters, high training and inference costs, and difficulty in meeting real-time requirements. Lightweighting of multi-view models through knowledge distillation has become an effective way to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to provide an image recognition method based on multi-view knowledge distillation, aiming to solve the problems of low recognition efficiency, high computational cost and poor real-time performance of existing multi-view recognition technologies.

[0005] To achieve the above object, the present invention provides an image recognition method based on multi-view knowledge distillation, comprising:

[0006] Acquire multiple imaging views of a single target at different viewing angles through an imaging device;

[0007] The teacher network performs parallel feature extraction of convolution auxiliary branches and attention auxiliary branches on multiple views;

[0008] The auxiliary features generated by the two branches are input into the multi-view alignment weighting module in parallel to perform feature fusion between corresponding positions of multiple views;

[0009] Use the fused feature vector to predict the category probability of the target in single view and multi-view and use the real label for supervision;

[0010] The student model extracts prototype features from multiple views respectively, generates a prototype-based multi-view feature vector, performs cross-view joint prediction, and obtains the student network prediction probability under single view and multi-view.

[0011] The teacher-student training framework is used to train the student network under the supervision of true labels and teacher predicted labels to obtain a lightweight multi-view model.

[0012] The specific steps of obtaining multiple imaging views of a single target at different viewing angles by an imaging device include:

[0013] Use X-ray, MRI or ultrasound to obtain images of a single target from multiple perspectives;

[0014] Multi-view imaging is desensitized, sensitive information contained in the image is cropped, and the masked automatic codec is used to predict and fill in the pixels of the missing area.

[0015] The specific steps of performing parallel feature extraction of convolution auxiliary branches and attention auxiliary branches on multiple views by the teacher network include:

[0016] Performing a covering convolution with a step size of 4 on the multiple views, mapping the features of the image blocks using a dilated convolution kernel with a size greater than the step size, extracting the association between each image block and the corresponding adjacent features, and generating mapping features;

[0017] The mapping features are arranged according to the spatial position into a feature vector of H×W×C and input into a multi-stage convolution branch. The mapping features are arranged in sequence into a feature sequence consisting of N single-dimensional features and input into a multi-stage attention branch for parallel feature extraction. The extracted parallel features are expressed as:

[0018] F c =CNN(R(Patch(F)))

[0019] F a =Attention(Patch(F))

[0020] where F c represents the output features of the CNN branch, F a Represents the output features of the Attention branch, F represents the input features of the image, R represents the reshape operation, and Patch represents the image block feature mapping performed by the covering convolution.

[0021] The specific steps of inputting the auxiliary features generated by the two branches in parallel into the multi-view alignment weighting module to perform feature fusion between corresponding positions of the multi-views include:

[0022] The specific process of the multi-view alignment weighting module for feature fusion between corresponding positions of multiple views is as follows: the parallel feature vectors (2×3×C) from the two branches are spliced ​​in the channel dimension and input into the fully connected layer to predict w1 and w2, and w1 and w2 are used as weighted values ​​for alignment fusion calculation, which is expressed as F=w1F c +w2F a .

[0023] The specific steps of using the fused feature vector to predict the category probability of the target in single view and multi-view and using the real label for supervision include:

[0024] The fused feature vector is input in parallel to the single view prediction branch and the overall classifier to predict the single view category and the multi-view category, and the labels are used for supervision.

[0025] The specific steps of extracting prototype features from multiple views respectively and generating a prototype-based multi-view feature vector for cross-view joint prediction to obtain the student network prediction probability under single view and multiple view include:

[0026] Use lightweight convnext-tiny as the backbone network to extract feature maps, use global convolution to generate M prototypes of length C, use the prototype and feature map to calculate cosine similarity and perform maximum pooling operations to obtain prototype features of length M;

[0027] The single-view category vector and multi-view category probability are predicted serially. The single-view category vector is first predicted by a multi-layer perceptron, and the category vector is input into the classifier to predict the multi-view comprehensive category.

[0028] The specific steps of using the teacher-student training framework to train the student network under the supervision of the real label and the teacher's predicted label to obtain a lightweight multi-view model include:

[0029] The teacher prediction label is obtained by data augmentation based on the prediction probability of the teacher model. Represents the output of the teacher model. When the distillation temperature coefficient is τ, the teacher's prediction result is defined as:

[0030] The true label is used as the hard label, the label predicted by the teacher network is used as the soft label, and the student network is trained with dual supervision using the single-view label and the multi-view comprehensive label. The loss of multi-view knowledge distillation is defined as:

[0031]

[0032] Where N is the number of views, α, β, c1, c2 are adjustable parameters, is the cosine similarity loss between the i-th single view label and its soft label, is the cosine similarity loss between the i-th single view label and its hard label. The recognition accuracy of the student model is improved through knowledge distillation, and a streamlined model that can be deployed to edge devices is obtained.

[0033] The present invention discloses an image recognition method based on multi-view knowledge distillation, which realizes the extraction of local features and long-distance dependencies in multi-views through convolution-assisted learning and attention-assisted learning of multi-view imaging, realizes the compression of multi-view models through knowledge distillation, improves the performance and reasoning efficiency of the model, and solves the problems of large number of parameters, high training and reasoning costs, and slow reasoning speed of current multi-view recognition models, greatly reduces the amount of calculation of multi-view models, simplifies the decision-making process of multi-view models, and improves the effect and performance of multi-view recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0035] Figure 1 It is a flow chart of the image recognition method based on multi-view knowledge distillation of the present invention.

[0036] Figure 2 It is a flow chart of the present invention for obtaining multiple imaging views of a single target at different viewing angles through an imaging device.

[0037] Figure 3 It is a flowchart of the teacher network of the present invention performing parallel feature extraction of convolution auxiliary branches and attention auxiliary branches on multiple views.

[0038] Figure 4 It is a flow chart of the student model of the present invention extracting prototype features from multiple views respectively, generating a multi-view feature vector based on the prototype, performing cross-view joint prediction, and obtaining the student network prediction probability under single perspective and multi-perspective.

[0039] Figure 5 It is a flowchart of the present invention using the teacher-student training framework to train the student network under the supervision of real labels and teacher predicted labels to obtain a lightweight multi-view model.

[0040] Figure 6 It is a model structure diagram of the image recognition method based on multi-view knowledge distillation of the present invention.

[0041] Figure 7 It is a training framework diagram of the multi-view knowledge distillation of the present invention. DETAILED DESCRIPTION

[0042] See also Figure 1 to Figure 7 , an image recognition method based on multi-view knowledge distillation of the present invention comprises:

[0043] The S1 teacher network performs parallel feature extraction of convolution auxiliary branches and attention auxiliary branches on multiple views;

[0044] The specific steps include:

[0045] S11 uses X-ray, MRI or ultrasound to obtain images of a single target from multiple perspectives:

[0046] S12 desensitizes multi-view imaging, crops sensitive information contained in the image, and uses the masked automatic codec to predict and fill in pixels in missing areas.

[0047] The S2 teacher network performs parallel feature extraction of convolution auxiliary branches and attention auxiliary branches on multiple views;

[0048] The specific steps include:

[0049] S21 performs a covering convolution with a step size of 4 on the multiple views, and maps the features of the image blocks using a dilated convolution kernel with a size greater than the step size, so as to extract the association between each image block and the corresponding adjacent features and generate mapping features;

[0050] S22 arranges the mapping features into a feature vector of H×W×C according to the spatial position and inputs it into a multi-stage convolution branch, arranges the mapping features into a feature sequence composed of N single-dimensional features in sequence and inputs it into a multi-stage attention branch to perform parallel feature extraction. The extracted parallel features are expressed as:

[0051] F c =CNN(R(Patch(F)))

[0052] F a =Attention(Patch(F))

[0053] where F c represents the output features of the CNN branch, F a Represents the output features of the Attention branch, F represents the input features of the image, R represents the reshape operation, and Patch represents the image block feature mapping performed by the covering convolution.

[0054] S3 inputs the auxiliary features generated by the two branches into the multi-view alignment weighting module in parallel to perform feature fusion between corresponding positions of multiple views;

[0055] The specific steps include:

[0056] S31 concatenates the parallel feature vectors (2×3×C) from the two branches in the channel dimension and inputs them into the fully connected layer to predict w1 and w2. Using w1 and w2 as weighted values, the positional fusion calculation is expressed as F=w1F c +w2F a .

[0057] S4 uses the fused feature vector to predict the category probability of the target in single view and multi-view and uses the real label for supervision;

[0058] The specific steps include:

[0059] S41 inputs the fused feature vector into the single view prediction branch and the general classifier in parallel to predict the single view category and the multi-view category, and uses the label for supervision.

[0060] The S5 student model extracts prototype features from multiple views respectively, generates a prototype-based multi-view feature vector, performs cross-view joint prediction, and obtains the student network prediction probability under single view and multi-view.

[0061] The specific steps include:

[0062] S51 uses lightweight convnext-tiny as the backbone network to extract feature maps, uses global convolution to generate M prototypes of length C, and uses the prototypes and feature maps to calculate cosine similarity and perform maximum pooling operations to obtain prototype features of length M;

[0063] S52 performs serial prediction on the single-view category vector and the multi-view category probability. The single-view category vector is first predicted by a multi-layer perceptron, and the category vector is input into the classifier to predict the multi-view comprehensive category.

[0064] S6 uses the teacher-student training framework to train the student network under the supervision of the real labels and the teacher's predicted labels to obtain a lightweight multi-view model;

[0065] The specific steps include:

[0066] The teacher prediction label in S61 is obtained by data enhancement based on the prediction probability of the teacher model. Represents the output of the teacher model. When the distillation temperature coefficient is τ, the teacher's prediction result is defined as:

[0067] S62 uses the real label as the hard label and the label predicted by the teacher network as the soft label. The student network is trained with dual supervision using the single-view label and the multi-view comprehensive label. The loss of multi-view knowledge distillation is defined as:

[0068]

[0069] Where N is the number of views, α, β, c1, c2 are adjustable parameters, is the cosine similarity loss between the i-th single view label and its soft label, is the cosine similarity loss between the i-th single view label and its hard label. The recognition accuracy of the student model is improved through knowledge distillation, and a streamlined model that can be deployed to edge devices is obtained.

[0070] A person skilled in the art can understand that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0071] The present invention discloses an image recognition method based on multi-view knowledge distillation, which realizes the extraction of local features and long-distance dependencies in multi-views through convolution-assisted learning and attention-assisted learning of multi-view imaging, realizes the compression of multi-view models through knowledge distillation, improves the performance and reasoning efficiency of the model, and solves the problems of large number of parameters, high training and reasoning costs, and slow reasoning speed of the current multi-view recognition model, greatly reduces the amount of calculation of the multi-view model, simplifies the decision-making process of the multi-view model, and improves the effect and performance of multi-view recognition.

[0072] What is disclosed above is only one or more preferred embodiments of the present application, and cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image recognition method based on multi-view knowledge distillation, characterized in that: include: Acquire multiple imaging views of a single target at different viewing angles through an imaging device; The teacher network performs parallel feature extraction of convolution auxiliary branches and attention auxiliary branches on multiple views; The auxiliary features generated by the two branches are input into the multi-view alignment weighting module in parallel to perform feature fusion between corresponding positions of multiple views; Use the fused feature vector to predict the category probability of the target in single view and multi-view and use the real label for supervision; The student model extracts prototype features from multiple views respectively, generates a prototype-based multi-view feature vector, performs cross-view joint prediction, and obtains the student network prediction probability under single view and multi-view. The teacher-student training framework is used to train the student network under the supervision of the real labels and the teacher's predicted labels to obtain a lightweight multi-view model. The specific steps of the teacher network performing parallel feature extraction of convolution auxiliary branches and attention auxiliary branches on multiple views include: Performing a covering convolution with a step size of 4 on the multiple views, mapping the features of the image blocks using a dilated convolution kernel with a size greater than the step size, extracting the association between each image block and the corresponding adjacent features, and generating mapping features; The mapping features are arranged according to the spatial position into a feature vector of H×W×C and input into a multi-stage convolution branch. The mapping features are arranged in sequence into a feature sequence consisting of N single-dimensional features and input into a multi-stage attention branch for parallel feature extraction. The extracted parallel features are expressed as: F c =CNN(R(Patch(F))) F a =Attention(Patch(F)) where F c represents the output features of the CNN branch, F a represents the output features of the Attention branch, F represents the input features of the image, R represents the reshape operation, and Patch represents the image block feature mapping performed by the coverage convolution; The specific steps of inputting the auxiliary features generated by the two branches in parallel into the multi-view alignment weighting module to perform feature fusion between corresponding positions of the multi-views include: The parallel feature vectors (2×3×C) from the two branches are concatenated in the channel dimension and input into the fully connected layer to predict w1 and w2. The positional fusion calculation is performed using w1 and w2 as weighted values, expressed as F=w1F c +w2F a ; The specific steps of using the teacher-student training framework to train the student network under the supervision of the real label and the teacher's predicted label to obtain a lightweight multi-view model include: The teacher prediction label is obtained by data augmentation based on the prediction probability of the teacher model. Represents the output of the teacher model. When the distillation temperature coefficient is τ, the teacher's prediction result is defined as: The true label is used as the hard label, the label predicted by the teacher network is used as the soft label, and the student network is trained with dual supervision using the single-view label and the multi-view comprehensive label. The loss of multi-view knowledge distillation is defined as: Where N is the number of views, α, β, c1, c2 are adjustable parameters, is the cosine similarity loss between the i-th single view label and its soft label, is the cosine similarity loss between the i-th single view label and its hard label. The recognition accuracy of the student model is improved through knowledge distillation, and a streamlined model that can be deployed to edge devices is obtained.

2. The image recognition method based on multi-view knowledge distillation according to claim 1, characterized in that: The specific steps of obtaining multiple imaging views of a single target at different viewing angles by an imaging device include: Use X-ray, MRI or ultrasound to obtain images of a single target from multiple perspectives; Multi-view imaging is desensitized, sensitive information contained in the image is cropped, and the masked automatic codec is used to predict and fill in the pixels of the missing area.

3. The image recognition method based on multi-view knowledge distillation according to claim 1, characterized in that: The specific steps of using the fused feature vector to predict the category probability of the target in single view and multi-view and using the real label for supervision include: The fused feature vector is input in parallel to the single view prediction branch and the overall classifier to predict the single view category and the multi-view category, and the labels are used for supervision.

4. The image recognition method based on multi-view knowledge distillation according to claim 1, characterized in that: The specific steps of extracting prototype features from multiple views respectively and generating a prototype-based multi-view feature vector for cross-view joint prediction to obtain the student network prediction probability under single view and multi-view include: Use lightweight convnext-tiny as the backbone network to extract feature maps, use global convolution to generate M prototypes of length C, use the prototype and feature map to calculate cosine similarity and perform maximum pooling operations to obtain prototype features of length M; The single-view category vector and multi-view category probability are predicted serially. The single-view category vector is first predicted by a multi-layer perceptron, and the category vector is input into the classifier to predict the multi-view comprehensive category.

Images