Training method of image recognition model, image recognition method, equipment, medium and product

Abstract

The invention relates to the technical field of deep learning and image recognition, and provides a training method of an image recognition model, an image recognition method, equipment, a medium and a product, and the training method comprises the steps: carrying out the knowledge distillation training of a second image recognition model based on a training sample set and a first image recognition model, the method comprises the following steps: executing image information decoupling on a first recognition result of a first image recognition model and a second recognition result of a second image recognition model to obtain first decoupling information and second decoupling information; comparing the first decoupling information with the second decoupling information to obtain overall feature loss and detail feature loss; and training a second image recognition model based on the overall feature loss, the detail feature loss and the knowledge distillation loss. The problem that the knowledge distillation effect is poor due to the fact that texture detail information is lost can be solved, the overall feature loss and the detail feature loss are considered respectively, and the knowledge distillation effect is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of deep learning and image recognition, and particularly relates to a training method of an image recognition model, an image recognition method, equipment, a medium and a product. BACKGROUND

[0002] Knowledge distillation (KD) is a model compression technology, which migrates the knowledge of a large teacher model to a lightweight student model, so as to realize efficient deployment on resource-limited devices (such as mobile terminals and edge computing nodes). The core idea of this technology is to make the student model imitate the output distribution of the teacher model (as soft labels), so as to inherit its generalization ability, and significantly improve the deployment efficiency of edge devices.

[0003] However, in the knowledge distillation technology in the field of computer vision, the traditional feature distillation mainly performs knowledge migration based on structural contour information of an image. In this process, textural detail information of the image is suppressed, and this part of information may be lost in the distillation, thereby resulting in poor knowledge distillation effect. SUMMARY

[0004] The present disclosure provides a training method of an image recognition model, an image recognition method, equipment, a medium and a product, to at least solve the problem that the knowledge distillation effect is poor due to the loss of textural detail information in the related art which mainly leads to the structural contour information of an image. The technical solutions of the present disclosure are as follows: According to a first aspect of the present disclosure, a training method of an image recognition model is provided, the training method comprising: performing knowledge distillation training on a second image recognition model based on a training sample set and a first image recognition model to obtain a trained second image recognition model, wherein the training sample set comprises a plurality of sample images, and the first image recognition model and the second image recognition model are used to recognize target objects in the sample images, wherein in each training round, the second image recognition model is trained by: performing image information decoupling on a first recognition result of the first image recognition model in a current training round and a second recognition result of the second image recognition model in the current training round respectively to obtain first decoupling information corresponding to the first recognition result and second decoupling information corresponding to the second recognition result, wherein each of the first decoupling information and the second decoupling information comprises overall feature information for representing overall features of an image and detail feature information for representing detail features of the image; obtaining an overall feature loss corresponding to the overall feature information and a detail feature loss corresponding to the detail feature information by comparing the first decoupling information and the second decoupling information; and training the second image recognition model based on the overall feature loss, the detail feature loss, and a knowledge distillation loss of the knowledge distillation training.

[0005] As an example, for each of the first recognition result and the second recognition result, the image information decoupling is performed by: in a spatial dimension, transforming the current recognition result to a frequency domain for each channel to obtain first frequency domain features; in a channel dimension, transforming the current recognition result to the frequency domain to obtain second frequency domain features; and separating image overall features and image detail features from the first frequency domain features and the second frequency domain features respectively to obtain decoupling information corresponding to the current recognition result.

[0006] As an example, the separating the image overall feature and the image detail feature from the first frequency domain feature and the second frequency domain feature respectively to obtain the decoupling information corresponding to the current recognition result comprises: separating spatial overall information and spatial detail information from the first frequency domain feature by using a first binary mask; separating channel overall information and channel detail information from the second frequency domain feature by using a second binary mask; taking the spatial overall information and the channel overall information as overall feature information corresponding to the current recognition result, and taking the spatial detail information and the channel detail information as detail feature information corresponding to the current recognition result, wherein the spatial overall information represents an image part with concentrated energy in the current recognition result, the spatial detail information represents an image part with dispersed energy in the current recognition result, the channel overall information represents an overall description of color or brightness or luminance of the current recognition result, and the channel detail information represents an image part with color change or brightness change or luminance change in the current recognition result.

[0007] As an example, the spatial overall information is represented by a spatial direct current component, the spatial detail information is represented by a spatial alternating current component, the channel overall information is represented by a channel direct current component, and the channel detail information is represented by a channel alternating current component, wherein the spatial direct current component and the channel alternating current component satisfy an orthogonal constraint.

[0008] As an example, the training the second image recognition model based on the overall feature loss, the detail feature loss, and the knowledge distillation loss of the knowledge distillation training comprises: weighting the overall feature loss and the detail feature loss respectively; and training the second image recognition model based on the weighted overall feature loss, the weighted detail feature loss, and the knowledge distillation loss.

[0009] As an example, the overall feature information of each of the first decoupling information and the second decoupling information includes spatial overall information and channel overall information, the detailed feature information of each of the first decoupling information and the second decoupling information includes spatial detailed information and channel detailed information, wherein the spatial overall information represents an image part with energy dispersion in the current recognition result, the spatial detailed information represents an image part with energy concentration in the current recognition result, the channel overall information represents an overall description of color or brightness or luminance of the current recognition result, and the channel detailed information represents an image part with color change or brightness change or luminance change in the current recognition result, wherein the overall feature loss includes a first loss for the spatial overall information between the first decoupling information and the second decoupling information and a second loss for the channel overall information between the first decoupling information and the second decoupling information, and the detailed feature loss includes a third loss corresponding to the spatial detailed information between the first decoupling information and the second decoupling information and a fourth loss corresponding to the channel detailed information between the first decoupling information and the second decoupling information, wherein a weight parameter for the first loss is greater than a weight parameter for the second loss, and a weight parameter for the third loss is greater than a weight parameter for the fourth loss.

[0010] According to a second aspect of the present disclosure, an image recognition method is provided, including: obtaining a target image; inputting the target image into an image recognition model, identifying a target object in the target image by using the image recognition model to obtain a recognition result, wherein the image recognition model is trained according to the training method of the image recognition model of the present disclosure.

[0011] According to a third aspect of the present disclosure, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions, wherein the processor-executable instructions, when executed by the processor, cause the processor to perform the training method of the image recognition model of the present disclosure or the image recognition method of the present disclosure.

[0012] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can perform the training method of the image recognition model of the present disclosure or the image recognition method of the present disclosure.

[0013] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising computer executable instructions which, when executed by at least one processor, implement the training method of the image recognition model or the image recognition method according to the present disclosure.

[0014] The technical solutions provided by the present disclosure bring at least the following beneficial effects: According to the image recognition model training and image recognition scheme, in the knowledge distillation training process, the image information of the recognition results output by the first image recognition model and the second image recognition model can be decoupled, the information for representing the overall features of the image and the information for representing the detailed features of the image are separated from each recognition result, and the overall feature loss and the detailed feature loss are respectively determined by comparing the information of the corresponding features, so that the knowledge distillation training can be performed by considering the overall feature loss and the detailed feature loss respectively, to avoid the problem of unbalanced learning of the overall features (such as structural contour) and the detailed features (such as texture details) of the image by the model, and the effect of knowledge distillation can be improved, so as to improve the image detail recognition ability of the second image recognition model.

[0015] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an improper limitation on the present disclosure.

[0017] Figure 1 is a schematic flowchart of a training method of an image recognition model according to an exemplary embodiment of the present disclosure.

[0018] Figure 2 is a schematic flowchart of obtaining decoupled information in a training method of an image recognition model according to an exemplary embodiment of the present disclosure.

[0019] Figure 3 is a schematic block diagram of a training method of an image recognition model according to an exemplary embodiment of the present disclosure.

[0020] Figure 4 is a schematic flowchart of obtaining decoupled information by using a binary mask in a training method of an image recognition model according to an exemplary embodiment of the present disclosure.

[0021] Figure 5 is a schematic block diagram of obtaining decoupled information in a spatial dimension in a training method of an image recognition model according to an exemplary embodiment of the present disclosure.

[0022] Figure 6 FIG. 1 is a schematic block diagram of decoupling information on a channel dimension in a training method of an image recognition model according to an example embodiment of the present disclosure.

[0023] Figure 7 FIG. 2 is a schematic flowchart of an application example of the training method of the image recognition model according to the example embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] For those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings.

[0025] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following example embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0026] It should be noted that "at least one of a plurality of items" appearing in the present disclosure means that it includes "any one of the plurality of items", "a combination of any multiple of the plurality of items", and "all of the plurality of items". For example, "including at least one of A and B" includes the following three alternatives: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step one and step two" means the following three alternatives: (1) performing step one; (2) performing step two; (3) performing step one and step two.

[0027] As described previously, in the conventional knowledge distillation scheme for image recognition, there is a problem that the structural contour information of the image is dominated, the texture detail information is lost, and the effect of knowledge distillation is poor.

[0028] In view of the above problems, the example embodiments of the present disclosure propose a training method of an image recognition model, an image recognition method, a training device of an image recognition model, an electronic device, a computer-readable storage medium, and a computer program product, which can solve or at least alleviate the above problems.

[0029] In a first aspect of the example embodiments of the present disclosure, a training method of an image recognition model is provided.

[0030] The training method of the image recognition model according to the example embodiments of the present disclosure can be applied to a training scenario of the image recognition model, for example, a software can be loaded on a user terminal, a user can input a model training instruction on the user terminal, the user terminal can obtain a trained image recognition model by performing the training method of the image recognition model according to the example embodiments of the present disclosure, and the image recognition model can be used to recognize an image, for example, to recognize a target object in the image, for image classification or image segmentation, etc.

[0031] The user terminal described above can be, for example, a tablet computer, a notebook computer, a digital assistant, a wearable device, etc., however, the implementation scenario of the above method is only an example scenario, and the training method of the image recognition model according to the example embodiments of the present disclosure can also be applied to other application scenarios, for example, a user can also request a server to train a model through a network on a user terminal (for example, a mobile phone, a desktop computer, a tablet computer, etc.), and the server can complete the request by performing the training method of the image recognition model according to the example embodiments of the present disclosure, here, the server can be a standalone server, a server cluster, a cloud computing platform, or a virtualization center.

[0032] The training method of the image recognition model according to the example embodiments of the present disclosure can decouple the recognition result output by the model in knowledge distillation to separately focus on the training loss of the overall features and the detailed features of the image, avoid the problem of uneven learning of the overall features and the detailed features of the image by the model, and improve the effect of knowledge distillation, thereby improving the recognition ability of the image recognition model for image details.

[0033] The training method of the image recognition model according to the example embodiments of the present disclosure will be described below with reference to the accompanying drawings. Figures 1 to 7 Examples of the training method of the image recognition model according to the example embodiments of the present disclosure are described below.

[0034] In the training method, the second image recognition model can be trained by knowledge distillation based on the training sample set and the first image recognition model to obtain a trained second image recognition model.

[0035] Here, the training sample set can include a plurality of sample images, and the first image recognition model and the second image recognition model are used to recognize a target object in the sample images.

[0036] As an example, the format, size, and image content of the sample images can be arbitrary, and the target object can be any preset content in the image. For example, the sample images can be tissue section images, such as cell pathology section images in medicine, plant cell section images in plant research, fungus images in eukaryote research, etc., and the target object can be a target tissue or structure in the tissue section images, such as a diseased cell, a plant cell, a cell nucleus, etc.

[0037] In addition, the training sample set can further include sample labels, which are used to mark the target object to be recognized in each sample image. In the training process, the sample labels can be used to compare the recognition result output by the second image recognition model to determine a part of the training loss, which will be described in detail below.

[0038] The first image recognition model and the second image recognition model can be used to recognize the target object in the image, and can also classify the image or segment the image based on the recognition result, for example, segmenting the target object from the image.

[0039] Here, the first image recognition model can be pre-trained (for example, can be trained according to any training method), and the second image recognition model can be a model to be trained. In the training method according to the embodiment of the present disclosure, the first image recognition model can be used as a teacher model to perform knowledge transfer to the second image recognition model to train the second image recognition model; the second image recognition model can be used as a student model, which can be trained using the prediction result of the teacher model.

[0040] As an example, the first image recognition model and the second image recognition model can be models with the same architecture (hereinafter also referred to as "isomorphic models"), or can also be models with different architectures (hereinafter also referred to as "heterogeneous models"), for example, the first image recognition model can be a model based on Swin-T (Swin Transformer) architecture, and the second image recognition model can be a model based on ResNet18 (Residual Network 18) architecture; as another example, the first image recognition model can be a model based on a convolutional network combined with a multilayer perceptron (Convolutional Neural Network-Multilayer Perceptron, CNN-MLP) architecture, and the second image recognition model can be a model based on a Transformer-MobileNet architecture.

[0041] The training method of the image recognition model according to the embodiment of the present disclosure can be combined with the existing training scheme of the image recognition model based on knowledge distillation to optimize the process of knowledge distillation. As an example, the existing training scheme of the image recognition model based on knowledge distillation can include but is not limited to ViT-CNN migration scheme, etc.

[0042] As Figure 1 shown, in each training round, the second image recognition model can be trained in the following way: In step S110, image information decoupling can be performed on the first recognition result of the first image recognition model in the current training round and the second recognition result of the second image recognition model in the current training round, respectively, to obtain first decoupling information corresponding to the first recognition result and second decoupling information corresponding to the second recognition result.

[0043] Specifically, the sample image can be input into the first image recognition model and the second image recognition model respectively, so that the sample image can be recognized by the first image recognition model and the second image recognition model respectively, and a first recognition result output from the first image recognition model and a second recognition result output from the second image recognition model are obtained.

[0044] As an example, the first recognition result and the second recognition result can be in the form of feature maps. In this example, before performing image information decoupling on the first recognition result and the second recognition result, the spatial size alignment of the feature maps of the first recognition result and the second recognition result can be performed. For example, the two feature maps can be aligned to a preset resolution (such as 7×7 resolution) through adaptive pooling.

[0045] In step S110 , each of the first decoupling information and the second decoupling information may include global feature information for representing global features of the image and detail feature information for representing detail features of the image.

[0046] Here, the overall image features may include, for example, the structural outline, main color, main brightness / shade and other features of the image, and the overall feature information may be used to describe at least one of the above-mentioned overall image features; the image detail features may include, for example, the image texture, lines, edges, color changes, brightness / shade changes and other features, and the detail feature information may be used to describe at least one of the above-mentioned image detail features.

[0047] In this step, the overall feature information and the detail feature information of the image can be decoupled from the first recognition result and the second recognition result, respectively, so as to be used for subsequently determining the loss in overall feature and detail feature of the image between the two recognition results.

[0048] As an example, in step S110, for each of the first recognition result and the second recognition result, image information decoupling may be performed in the following manner: like Figure 2 As shown, in step S210, the current recognition result can be transformed into the frequency domain for each channel in the spatial dimension to obtain a first frequency domain feature.

[0049] Specifically, the first recognition result and the second recognition result may be subjected to characteristic spectrum conversion in the spatial dimension respectively, so as to convert the first recognition result and the second recognition result from the pixel domain to the frequency domain.

[0050] As an example, for each recognition result (which can be the first recognition result or the second recognition result), a two-dimensional discrete cosine transform (2D DCT) can be performed on a channel-by-channel basis to obtain a first frequency domain feature. Specifically, a transformation from a pixel domain to a frequency domain can be performed by the following equation (1): (1) wherein, represents the first frequency domain feature in the transformed feature map at position (u, v) on channel c, represents the original recognition result (e.g., a feature map output by a model), represents a height of an image, represents a width of the image, represents a pixel in the height of the image, represents a pixel in the width of the image, represents a pixel position in the height direction of the transformed feature map, represents a pixel position in the width direction of the transformed feature map, wherein, for example, as shown in (u = 0, v = 0) corresponds to a low-frequency structural component (Structural Contours), and the remaining frequency coefficients correspond to high-frequency textural components (Textural Details). Figure 5 At step S220, the current recognition result can be transformed to a frequency domain on a channel dimension to obtain a second frequency domain feature.

[0051] Specifically, a feature spectrum conversion on a channel dimension can be performed on the first recognition result and the second recognition result respectively, to convert the first recognition result and the second recognition result from a pixel domain to a frequency domain.

[0052] As an example, for each recognition result of a channel vector , a one-dimensional discrete cosine transform (1D DCT) can be applied along a channel axis to obtain a second frequency domain feature. Specifically, a transformation from a pixel domain to a frequency domain can be performed by the following equation (2):

[0053] (2) wherein, represents the second frequency domain feature of the transformed feature map on channel , represents the original recognition result (e.g., a feature map output by a model), represents a total number of channels of an image, wherein, for example, as shown in Figure 6 ​​​As shown in Figure 2, k=0 corresponds to semantic information, and k>0 corresponds to instance-specific features.

[0054] The above steps S210 and S220 can be performed as follows, for example: Figure 3 As shown, the first recognition result and the second recognition result can be transformed into the frequency domain by feature projection in the spatial dimension and channel dimension respectively. Figure 3 In the equation, “⊥” indicates that the spatial dimension and the channel dimension are orthogonal.

[0055] In addition, despite Figure 2 It is shown that step S210 is performed first and then step S220 is performed, but the embodiments of the present disclosure are not limited thereto. Step S220 may be performed first and then step S210, or the two may be performed in parallel.

[0056] In step S230 , the overall image features and the image detail features may be separated from the first frequency domain features and the second frequency domain features, respectively, to obtain decoupling information corresponding to the current recognition result.

[0057] In this step, if Figure 3 As shown, the overall image features and image detail features can be separated from the first frequency domain features and the second frequency domain features respectively by filtering and other methods. In this way, the overall image features and image detail features in each frequency domain feature can be decoupled, and the loss calculation can be performed separately for the overall features and detail features subsequently.

[0058] As an example, a binary mask can be used to separate frequency domain features. Figure 4 As shown, the decoupling information corresponding to the current recognition result can be obtained through the following steps: In step S410, a first binary mask may be used to separate the spatial overall information and the spatial detail information from the first frequency domain feature.

[0059] Here, the spatial overall information represents the image part where energy is concentrated in the current recognition result, such as large color blocks in the image; the spatial detail information represents the image part where energy is dispersed in the current recognition result, such as edges, textures, noise, etc. in the image.

[0060] Specifically, for the first frequency domain feature in the spatial dimension, the structural contours and the textural details can be distinguished according to the high and low frequencies. For example, the low frequency information of the frequency u, v = 0 can be determined as the spatial overall information (which can also be referred to as a low frequency structural contours component); the high frequency information of the frequency u, v > 0 can be determined as the spatial detail information (which can also be referred to as a high frequency textural details component), so that in the example that the first image recognition model and the second image recognition model are heterogeneous models, the differences in the frequency spectrum of different architectures can be eliminated to enable the calculation of the loss in the spatial overall information and the spatial detail information between the two models.

[0061] As an example, the spatial overall information can be represented by a spatial direct component (DC-spatial component), and the spatial detail information can be represented by a spatial alternative component (AC-spatial component). Figure 5 An example of separating the spatial DC component and the spatial AC component from the first frequency domain feature by using the first binary mask is shown as Figure 5 As shown, the spatial overall information can be represented as: , wherein, The first binary mask is represented, for example, the coefficient (u = 0, v = 0) can be reserved; the spatial detail information can be represented as: .

[0062] In step S420, the channel overall information and the channel detail information can be separated from the second frequency domain feature by using the second binary mask.

[0063] Here, the channel overall information represents the overall description of the color or the brightness or the luminance of the current recognition result, for example, a large color block, a smooth brightness change, or no brightness change; the channel detail information represents the image part in which the color change or the brightness change or the luminance change exists in the current recognition result.

[0064] Specifically, for the second frequency domain feature in the channel dimension, the global semantic information and the instance-specific features can be distinguished according to the high and low frequencies. For example, the information of k = 0 can be determined as the channel overall information (which can also be referred to as a semantic information component); the information of k > 0 can be determined as the channel detail information (which can also be referred to as an instance-specific features component), here, by further considering the channel dimension in the spatial dimension, the single-dimensional decoupling limitation can be broken through, and the distillation effect can be further improved.

[0065] As an example, the channel overall information is represented by a Channel Direct Component (DC-channel), and the channel detail information is represented by a Channel Alternative Component (AC-channel). For example, the channel overall information can be represented as: wherein, represents a second binary mask, for example, the coefficient k = 0 can be reserved, representing the semantic information at the class level; the channel detail information can be represented as: corresponding to the encoded instance-level feature with k > 0.

[0066] In addition, the above spatial direct component and channel alternative component can satisfy an orthogonal constraint, for example, satisfying the orthogonal relationship Fdc spatial,Fac channel = 0, so that the cross-dimension information redundancy can be eliminated.

[0067] In step S430, the spatial overall information and the channel overall information can be taken as the overall feature information corresponding to the current recognition result, and the spatial detail information and the channel detail information can be taken as the detail feature information corresponding to the current recognition result.

[0068] Referring back to Figure 1 , in step S120, the overall feature loss corresponding to the overall feature information and the detail feature loss corresponding to the detail feature information can be obtained by comparing the first decoupled information and the second decoupled information.

[0069] For example, as Figure 3 indicated, the first decoupled information and the second decoupled information in the frequency domain can be inverse transformed back to the pixel domain respectively, and based on the first decoupled information and the second decoupled information in the pixel domain, the above overall feature loss and detail feature loss can be calculated.

[0070] As an example, in the example that the overall feature information of each decoupled information includes the spatial overall information and the channel overall information, the overall feature loss can be represented as wherein, represents a loss, , represents a spatial dimension, represents a channel dimension, represents overall information (for example, a DC component). In the example that the detail feature information of each decoupled information includes the spatial detail information and the channel detail information, the detail feature loss can be represented as , represents detail information (for example, an AC component).

[0071] At step S130, the second image recognition model can be trained based on the overall feature loss, the detail feature loss, and the knowledge distillation loss of the knowledge distillation training.

[0072] In this step, the decoupled overall feature loss and the detail feature loss calculated above can be combined with the original loss of the knowledge distillation training as the final prediction loss (e.g. Figure 3 MSE loss in (3)).

[0073] In an example, the overall feature loss, the detail feature loss, and the knowledge distillation loss of the knowledge distillation training can be directly added as the final prediction loss.

[0074] In another example, the overall feature loss and the detail feature loss can also be weighted, and the weighted overall feature loss, the weighted detail feature loss, and the knowledge distillation loss can be added as the final prediction loss.

[0075] Specifically, the step S130 can include: weighting the overall feature loss and the detail feature loss respectively; and training the second image recognition model based on the weighted overall feature loss, the weighted detail feature loss, and the knowledge distillation loss.

[0076] Here, the overall feature loss and the detail feature loss can be weighted by adjustable weight parameters.

[0077] As an example, the knowledge distillation loss can include an original task loss and a base distillation loss, the base distillation loss can be obtained by comparing the second recognition result and the first recognition result (e.g. can be understood as a soft label), and the original task loss can be obtained by comparing the second recognition result and a sample label (e.g. can be understood as a real label).

[0078] In this example, the final prediction loss can be represented by the following formula (3): (3) wherein, Lpred represents the final prediction loss (or total loss), Lorig represents the original task loss in the knowledge distillation loss, Lbase represents the base distillation loss in the knowledge distillation loss, e.g. base Vanella KD loss, Loverall represents the overall feature loss, Ldetail represents the detail feature loss, wherein the loss for spatial overall information can optimize structure contour alignment, the loss for channel detail information can strengthen detail feature transfer. and respectively represent weight parameters for weighting the overall feature loss and the detail feature loss, which can be preset.

[0079] In the above process, by introducing the adjustable weight parameters, the distillation strength of the overall information and the detail information can be dynamically regulated, and the problem of the suppression of the detail information caused by the dominance of the overall information is further solved.

[0080] In addition, as an example, as described above, the overall feature loss can include a first loss for the spatial overall information between the first decoupled information and the second decoupled information and a second loss for the channel overall information between the first decoupled information and the second decoupled information, and the detail feature loss can include a third loss for the spatial detail information between the first decoupled information and the second decoupled information and a fourth loss for the channel detail information between the first decoupled information and the second decoupled information. In this case, the weight parameters can be represented as , wherein is used to weight the first loss, is used to weight the third loss, is used to weight the second loss, is used to weight the fourth loss.

[0081] Here, the weight parameter for the first loss is greater than the weight parameter for the second loss , and the weight parameter for the third loss is greater than the weight parameter for the fourth loss .

[0082] In this way, by constraining the size relationship between the weight parameters of different losses, the dominant effect of the overall information can be suppressed, and the knowledge transfer capability of the detail information can be enhanced.

[0083] For example, when the first image recognition model is a visual Transformer architecture (such as Swin-T), setting = 0.03 can suppress its low-frequency dominant effect (the DC energy ratio of ViT is more than 70%); when the second image recognition model is a lightweight CNN architecture (such as MobileNetV2), setting = 0.07 can enhance the instance detail transfer capability.

[0084] In addition, in this step S130, the specific optimization goal of the loss function is to minimize the difference between the output prediction of the second image recognition model and the real label, and the weight parameters of the second image recognition model can be updated based on the optimization goal to train the second image recognition model.

[0085] Figure 7A schematic flowchart showing an application example of a training method of an image recognition model according to an example embodiment of the present disclosure is shown.

[0086] As shown in Figure 7 At step S710, the first recognition result and the second recognition result can be obtained, and at step S720, DCT transformation and orthogonal constraint decoupling of the first recognition result and the second recognition result in the spatial dimension and the channel dimension can be performed, respectively, wherein for the spatial dimension, 2D DCT can be performed to obtain spatial DC components representing structure outlines and spatial AC components representing texture details; and for the channel dimension, 1D DCT can be performed to obtain channel DC components representing category semantics and channel AC components representing instance features.

[0087] At step S730, the losses corresponding to the respective components obtained at step S720 can be weighted to achieve dynamic spectral balance, and at step S740, the weighted losses can be fused with the original task loss and the basic distillation loss to obtain a final prediction loss.

[0088] At step S750, the parameters of the second image recognition model can be updated based on the final prediction loss to train the second image recognition model. The above steps S710 to S750 can be performed for multiple rounds to achieve multiple rounds of training, and finally a trained second image recognition model can be output, which can be a lighter model compared to the first image recognition model, for example, the number of parameters of the second image recognition model can be much smaller than that of the first image recognition model.

[0089] The training method of the image recognition model according to the embodiment of the present disclosure can overcome the cross-architecture feature mismatch problem caused by spectral distribution imbalance in traditional knowledge distillation through frequency domain decoupling, so that the final output lightweight student model (i.e. the above-mentioned second image recognition model) can be directly deployed on edge devices to support image classification, object detection, semantic segmentation and other visual tasks.

[0090] In addition, the training method of the image recognition model according to the embodiment of the present disclosure effectively solves the problem of semantic-instance feature imbalance in knowledge distillation for image recognition through spatial and channel dual-dimensional frequency decoupling and dynamic weight regulation, providing a new paradigm for lightweight model deployment. Specifically, in this method, when updating the student parameters through backpropagation, the introduction of dynamic weight parameters to adjust gradient distribution can further solve the cross-architecture spectral mismatch problem, and through the hierarchical loss function, the dual-dimensional decoupling loss is dynamically fused with the original task loss and the basic distillation loss, and through backpropagation gradient optimization, the student model is guided to retain its own efficiency advantage while absorbing the representation ability of the teacher model.

[0091] In addition, in the knowledge distillation of the related art, in the case where the teacher model and the student model are heterogeneous models, due to the different extraction capabilities and extraction manners of the models with different architectures on information, and the differences in the energy distribution of the outputs of the models on the frequency spectrum, the cross-architecture learning effect is lost. To this end, by using the training method according to the embodiments of the present disclosure, the information of different features is decoupled at the feature level, the key information can be learned respectively, so as to balance the distillation learning between different features, and improve the distillation effect of the heterogeneous models.

[0092] In a second aspect of the example embodiments of the present disclosure, an image recognition method is provided, which can include: obtaining a target image; inputting the target image into an image recognition model, and recognizing a target object in the target image by using the image recognition model to obtain a recognition result, wherein the image recognition model is trained by using the training method of the image recognition model according to the embodiments of the present disclosure, for example, can be the trained second image recognition model described above.

[0093] The specific steps and details of the image recognition method according to the example embodiments of the present disclosure can be similarly implemented and understood with reference to the training method of the image recognition model described in the first aspect above, and accordingly, the image recognition method has the same or similar beneficial effects as the training method of the image recognition model described above, which will not be repeated here.

[0094] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, which includes: a processor; and a memory for storing processor-executable instructions, wherein the processor-executable instructions, when executed by the processor, cause the processor to perform the training method of the image recognition model according to the embodiments of the present disclosure or the image recognition method according to the embodiments of the present disclosure.

[0095] As an example, the electronic device is not necessarily a single device, but can also be any collection of devices or circuits that can individually or jointly execute the above instructions (or instruction sets). The electronic device can also be part of an integrated control system or system manager, or can be configured as a server that interfaces with a local or remote server (e.g., via wireless transmission).

[0096] In the electronic device, the processor can include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. As an example but not limitation, the processor can also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0097] The processor can execute instructions or code stored in the memory, where the memory can also store data. The instructions and data can also be sent and received over a network via the network interface device, where the network interface device can employ any known transmission protocol.

[0098] The memory can be integral to the processor, such as RAM or flash memory disposed within an integrated circuit microprocessor, etc. Further, the memory can include a separate device, such as an external disk drive, memory array, or other storage device usable by any database system. The memory and processor can be operatively coupled, or can communicate with each other, such as through an I / O port, network connection, etc., such that the processor can read files stored in the memory.

[0099] Further, the electronic device can also include a video display (such as a liquid crystal display) and a user interface interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device can be connected to each other via a bus and / or network.

[0100] In an example embodiment, a computer-readable storage medium can also be provided, which, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the training method of an image recognition model or the image recognition method as described in the example embodiments above. The computer-readable storage medium can be, for example, a memory including instructions, and optionally, the computer-readable storage medium can be: a read-only memory (ROM), a random access memory (RAM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, a BD-R, a BD-R LTH, a BD-RE, a Blu-ray or an optical disc memory, a hard disk drive (HDD), a solid state drive (SSD), a card-type memory such as a multimedia card, a secure digital (SD) card, or an extreme digital (XD) card, a magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a solid state disk, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and provide the computer program and any associated data, data files, and data structures to a processor or a computer so that the processor or the computer can execute the computer program. The computer program in the computer-readable storage medium described above can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc., and in addition, in one example, the computer program and any associated data, data files, and data structures are distributed over a networked computer system so that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0101] According to an example embodiment of the disclosure, a computer program product including computer executable instructions that, when executed by at least one processor, implement a training method of an image recognition model or an image recognition method according to an example embodiment of the disclosure can also be provided.

[0102] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the disclosure be construed as including any patents, patent applications, publications, publications, or other disclosure of the prior art that are referred to by citation above. The specification and examples given herein are to be considered illustrative and not restrictive. The scope of the disclosure is to be determined by the appended claims.

[0103] Furthermore, it is to be understood that, unless otherwise expressly provided herein, any of the steps of the processes described herein can be performed in any order. Furthermore, it is to be understood that, although the steps of the above-described embodiments have been described in conjunction with the above-described examples, other steps and / or examples can modify, augment, or omit one or more of the steps described above.

[0104] It is to be understood that the embodiments of the present disclosure are not limited to the precise structures herein described and illustrated above, and that various modifications and changes can be made without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A training method for an image recognition model, characterized in that: The training method comprises: Based on the training sample set and the first image recognition model, the second image recognition model is trained by knowledge distillation to obtain a trained second image recognition model, wherein the training sample set includes a plurality of sample images, and the first image recognition model and the second image recognition model are used to identify target objects in the sample images. In each training round, the second image recognition model is trained in the following manner: performing image information decoupling on a first recognition result of the first image recognition model in a current training round and a second recognition result of the second image recognition model in the current training round, respectively, to obtain first decoupled information corresponding to the first recognition result and second decoupled information corresponding to the second recognition result, wherein each of the first decoupled information and the second decoupled information includes overall feature information for characterizing overall features of the image and detail feature information for characterizing detail features of the image; Obtaining an overall feature loss corresponding to the overall feature information and a detail feature loss corresponding to the detail feature information by comparing the first decoupling information and the second decoupling information; The second image recognition model is trained based on the overall feature loss, the detail feature loss, and the knowledge distillation loss of the knowledge distillation training.

2. The training method according to claim 1, characterized in that For each of the first recognition result and the second recognition result, performing the image information decoupling in the following manner: In the spatial dimension, for each channel, the current recognition result is transformed into the frequency domain to obtain the first frequency domain feature; In the channel dimension, the current recognition result is transformed into the frequency domain to obtain the second frequency domain feature; The overall image features and the image detail features are separated from the first frequency domain features and the second frequency domain features respectively to obtain decoupling information corresponding to the current recognition result.

3. The training method according to claim 2, characterized in that The separating the overall image feature and the detail image feature from the first frequency domain feature and the second frequency domain feature respectively to obtain decoupling information corresponding to the current recognition result includes: Using a first binary mask, separate the spatial overall information and the spatial detail information from the first frequency domain features; using a second binary mask, separate the channel overall information and the channel detail information from the second frequency domain features; The spatial overall information and the channel overall information are used as the overall feature information corresponding to the current recognition result, and the spatial detail information and the channel detail information are used as the detail feature information corresponding to the current recognition result, Among them, the spatial overall information represents the image part with concentrated energy in the current recognition result, the spatial detail information represents the image part with dispersed energy in the current recognition result, the channel overall information represents the overall description of the color, brightness or luminance of the current recognition result, and the channel detail information represents the image part with color change, brightness change or brightness change in the current recognition result.

4. The training method according to claim 3, characterized in that The spatial overall information is represented by the spatial DC component, the spatial detail information is represented by the spatial AC component, the channel overall information is represented by the channel DC component, and the channel detail information is represented by the channel AC component. The spatial DC component and the channel AC component satisfy an orthogonal constraint.

5. The training method according to claim 1, wherein: The training of the second image recognition model based on the overall feature loss, the detail feature loss, and the knowledge distillation loss of the knowledge distillation training includes: Weighting the overall feature loss and the detail feature loss respectively; The second image recognition model is trained based on the weighted overall feature loss, the weighted detail feature loss, and the knowledge distillation loss.

6. The training method according to claim 5, characterized in that The overall feature information of each decoupling information in the first decoupling information and the second decoupling information includes spatial overall information and channel overall information, and the detail feature information of each decoupling information in the first decoupling information and the second decoupling information includes spatial detail information and channel detail information. The spatial overall information represents the image portion with dispersed energy in the current recognition result, the spatial detail information represents the image portion with concentrated energy in the current recognition result, the channel overall information represents the overall description of the color, brightness or luminance of the current recognition result, and the channel detail information represents the image portion with color change, brightness change or luminance change in the current recognition result. The overall feature loss includes: a first loss of the spatial overall information between the first decoupled information and the second decoupled information, and a second loss of the channel overall information between the first decoupled information and the second decoupled information. The detail feature loss includes: a third loss corresponding to the spatial detail information between the first decoupling information and the second decoupling information, and a fourth loss corresponding to the channel detail information between the first decoupling information and the second decoupling information. Among them, the weight parameter for the first loss is greater than the weight parameter for the second loss, and the weight parameter for the third loss is greater than the weight parameter for the fourth loss.

7. An image recognition method, characterized in that: The image recognition method comprises: Acquire the target image; The target image is input into an image recognition model, and the target object in the target image is recognized using the image recognition model to obtain a recognition result, wherein the image recognition model is trained according to the image recognition model training method according to any one of claims 1 to 6.

8. An electronic device, characterized in that: The electronic device comprises: processor; memory for storing processor-executable instructions, When the processor executable instructions are executed by the processor, they prompt the processor to execute the training method of the image recognition model according to any one of claims 1 to 6 or the image recognition method according to claim 7.

9. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the training method of the image recognition model according to any one of claims 1 to 6 or the image recognition method according to claim 7.

10. A computer program product comprising computer executable instructions, characterized in that: When the computer-executable instructions are executed by at least one processor, the training method of the image recognition model according to any one of claims 1 to 6 or the image recognition method according to claim 7 is implemented.

Images