Image recognition model training methods, image recognition methods, equipment, media and products

Abstract

This disclosure relates to the fields of deep learning and image recognition technology, and provides a training method, image recognition method, device, medium, and product for an image recognition model. The training method includes: training a second image recognition model using knowledge distillation based on a training sample set and a first image recognition model. The training is performed by: decoupling image information from a first recognition result of the first image recognition model and a second recognition result of the second image recognition model to obtain first decoupling information and second decoupling information; comparing the first and second decoupling information to obtain an overall feature loss and a detail feature loss; and training the second image recognition model based on the overall feature loss, detail feature loss, and knowledge distillation loss. This disclosure can solve the problem of poor knowledge distillation performance due to the loss of texture detail information by considering both overall feature loss and detail feature loss separately, thereby improving the effectiveness of knowledge distillation.

Description

Technical Field

[0001] This disclosure relates to the fields of deep learning and image recognition technology, and in particular to a training method for an image recognition model, an image recognition method, an apparatus, a medium, and a product. Background Technology

[0002] Knowledge distillation (KD) is a model compression technique that enables efficient deployment on resource-constrained devices (such as mobile devices and edge computing nodes) by transferring knowledge from a large teacher model to a lightweight student model. The core idea of ​​this technique is to allow the student model to mimic the output distribution of the teacher model (as a soft label), thereby inheriting its generalization ability and significantly improving the deployment efficiency on edge devices.

[0003] However, in the knowledge distillation technique in the field of computer vision, traditional feature distillation relies primarily on the structural contours of the image for knowledge transfer. In this process, the textural details of the image are suppressed, and this information may be lost during distillation, resulting in poor knowledge distillation performance. Summary of the Invention

[0004] This disclosure provides a training method for an image recognition model, an image recognition method, an apparatus, a medium, and a product, to at least solve the problem in related technologies where knowledge distillation is ineffective due to the reliance on structural contour information of images and the loss of texture detail information. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of this disclosure, a training method for an image recognition model is provided. The training method includes: performing knowledge distillation training on a second image recognition model based on a training sample set and a first image recognition model to obtain a trained second image recognition model. The training sample set includes multiple sample images. The first image recognition model and the second image recognition model are used to identify target objects in the sample images. In each training round, the second image recognition model is trained by: performing image information decoupling on a first recognition result of the first image recognition model in the current training round and a second recognition result of the second image recognition model in the current training round, respectively, to obtain first decoupling information corresponding to the first recognition result and second decoupling information corresponding to the second recognition result. Each decoupling information in the first and second decoupling information includes overall feature information for characterizing the overall features of the image and detailed feature information for characterizing the detailed features of the image; comparing the first and second decoupling information to obtain an overall feature loss corresponding to the overall feature information and a detailed feature loss corresponding to the detailed feature information; and training the second image recognition model based on the overall feature loss, the detailed feature loss, and the knowledge distillation loss from the knowledge distillation training.

[0006] As an example, for each of the first and second recognition results, the image information decoupling is performed in the following manner: in the spatial dimension, for each channel, the current recognition result is transformed to the frequency domain to obtain the first frequency domain feature; in the channel dimension, the current recognition result is transformed to the frequency domain to obtain the second frequency domain feature; the overall image features and the image detail features are separated from the first and second frequency domain features respectively to obtain the decoupling information corresponding to the current recognition result.

[0007] As an example, the step of separating the overall image features and image detail features from the first frequency domain features and the second frequency domain features respectively to obtain decoupling information corresponding to the current recognition result includes: using a first binary mask to separate spatial overall information and spatial detail information from the first frequency domain features; using a second binary mask to separate channel overall information and channel detail information from the second frequency domain features; using the spatial overall information and the channel overall information as overall feature information corresponding to the current recognition result, and using the spatial detail information and the channel detail information as detail feature information corresponding to the current recognition result, wherein the spatial overall information represents the image portion with concentrated energy in the current recognition result, the spatial detail information represents the image portion with dispersed energy in the current recognition result, the channel overall information represents the overall description of the color, brightness, or luminance of the current recognition result, and the channel detail information represents the image portion in the current recognition result where there are color changes, brightness changes, or luminance changes.

[0008] As an example, the overall spatial information is represented by a spatial DC component, the detailed spatial information is represented by a spatial AC component, the overall channel information is represented by a channel DC component, and the detailed channel information is represented by a channel AC component, wherein the spatial DC component and the channel AC component satisfy an orthogonal constraint.

[0009] As an example, training the second image recognition model based on the overall feature loss, the detail feature loss, and the knowledge distillation loss trained by the knowledge distillation includes: weighting the overall feature loss and the detail feature loss respectively; and training the second image recognition model based on the weighted overall feature loss, the weighted detail feature loss, and the knowledge distillation loss.

[0010] As an example, the overall feature information of each decoupling information in the first and second decoupling information includes spatial overall information and channel overall information; the detailed feature information of each decoupling information includes spatial detail information and channel detail information. The spatial overall information represents the energy-dispersed image portion of the current recognition result; the spatial detail information represents the energy-concentrated image portion of the current recognition result; the channel overall information represents the overall description of the color, brightness, or luminance of the current recognition result; and the channel detail information represents the image portion of the current recognition result where color, brightness, or luminance changes occur. The overall feature loss includes: a first loss for the overall spatial information between the first decoupling information and the second decoupling information, and a second loss for the overall channel information between the first decoupling information and the second decoupling information. The detailed feature loss includes: a third loss corresponding to the spatial detailed information between the first decoupling information and the second decoupling information, and a fourth loss corresponding to the channel detailed information between the first decoupling information and the second decoupling information. The weight parameter for the first loss is greater than the weight parameter for the second loss, and the weight parameter for the third loss is greater than the weight parameter for the fourth loss.

[0011] According to a second aspect of this disclosure, an image recognition method is provided, the image recognition method comprising: acquiring a target image; inputting the target image into an image recognition model, using the image recognition model to recognize a target object in the target image, and obtaining a recognition result, wherein the image recognition model is trained according to the image recognition model training method described in this disclosure.

[0012] According to a third aspect of this disclosure, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing processor-executable instructions, wherein, when executed by the processor, the processor causes the processor to perform a training method for an image recognition model according to this disclosure or an image recognition method according to this disclosure.

[0013] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform a training method for an image recognition model according to this disclosure or an image recognition method according to this disclosure.

[0014] According to a fifth aspect of this disclosure, a computer program product is provided, comprising computer-executable instructions that, when executed by at least one processor, implement a training method for an image recognition model according to this disclosure or an image recognition method according to this disclosure.

[0015] The technical solution provided in this disclosure brings at least the following beneficial effects:

[0016] According to the image recognition model training and image recognition scheme disclosed herein, during the knowledge distillation training process, the image information of the recognition results output by the first image recognition model and the second image recognition model can be decoupled. Information used to characterize the overall features of the image and information used to characterize the detailed features of the image can be separated from each recognition result. By comparing the information of the corresponding features, the overall feature loss and the detailed feature loss can be determined respectively. Thus, knowledge distillation training can be performed while considering the overall feature loss and the detailed feature loss respectively, so as to avoid the problem of the model's unbalanced learning of the overall features (e.g., structural contours) and detailed features (e.g., texture details) of the image. This can improve the effect of knowledge distillation, thereby improving the ability of the second image recognition model to recognize image details.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0019] Figure 1 This is a schematic flowchart of a training method for an image recognition model according to an exemplary embodiment of the present disclosure.

[0020] Figure 2 This is a schematic flowchart illustrating the method for obtaining decoupling information in the training of an image recognition model according to an exemplary embodiment of the present disclosure.

[0021] Figure 3 This is a schematic block diagram of a training method for an image recognition model according to an exemplary embodiment of the present disclosure.

[0022] Figure 4 This is a schematic flowchart illustrating the method for obtaining decoupling information using a binary mask in the training of an image recognition model according to an exemplary embodiment of the present disclosure.

[0023] Figure 5 This is a schematic block diagram illustrating the process of obtaining decoupled information in the spatial dimension in a training method for an image recognition model according to an exemplary embodiment of the present disclosure.

[0024] Figure 6 This is a schematic block diagram illustrating the process of obtaining decoupled information in the channel dimension in a training method for an image recognition model according to an exemplary embodiment of the present disclosure.

[0025] Figure 7 This is a schematic flowchart illustrating an application example of a training method for an image recognition model according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0026] In order to enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0027] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0028] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0029] As mentioned earlier, traditional knowledge distillation schemes for image recognition suffer from poor results due to their reliance on structural contour information and loss of texture details.

[0030] In view of the above problems, exemplary embodiments of this disclosure provide a training method for an image recognition model, an image recognition method, a training apparatus for an image recognition model, an electronic device, a computer-readable storage medium, and a computer program product, which can solve or at least alleviate the above problems.

[0031] In a first aspect of an exemplary embodiment of this disclosure, a method for training an image recognition model is provided.

[0032] The training method for the image recognition model according to the exemplary embodiments of the present disclosure can be applied to the training scenario of the image recognition model. For example, software can be loaded on the user terminal, and the user can input model training instructions on the user terminal. The user terminal can obtain the trained image recognition model by executing the training method for the image recognition model according to the exemplary embodiments of the present disclosure. The image recognition model can be used to recognize images, such as recognizing target objects in the image for image classification or image segmentation.

[0033] The aforementioned user terminal can be such as a tablet computer, laptop computer, digital assistant, wearable device, etc. However, the implementation scenario of the above method is only an example scenario. The training method of the image recognition model according to the exemplary embodiment of this disclosure can also be applied to other application scenarios. For example, the user can request the training model from the server via the network on the user terminal (e.g., mobile phone, desktop computer, tablet computer, etc.). The server can complete the request by executing the training method of the image recognition model according to the exemplary embodiment of this disclosure. Here, the server can be an independent server, a server cluster, a cloud computing platform, or a virtualization center.

[0034] The image recognition model training method according to the exemplary embodiments of the present disclosure can decouple the image information of the recognition result output by the model in knowledge distillation, so as to focus on the training loss of the overall features and details of the image separately, avoid the problem of the model's unbalanced learning of the overall features and details of the image, improve the effect of knowledge distillation, and thus improve the image recognition model's ability to recognize image details.

[0035] The following will refer to Figures 1 to 7 Examples of training methods for image recognition models according to embodiments of the present disclosure are described.

[0036] In this training method, the second image recognition model can be trained by knowledge distillation based on the training sample set and the first image recognition model to obtain the trained second image recognition model.

[0037] Here, the training sample set may include multiple sample images, and the first image recognition model and the second image recognition model are used to identify the target objects in the sample images.

[0038] As an example, the format, size, and content of the sample image can be arbitrary, and the target object can be any preset content in the image. For example, the sample image can be a tissue section image, such as a cytopathological section image in medicine, a plant cell section image in plant research, or a fungal image in eukaryotic research; the target object can be the target tissue or structure in the tissue section image, such as diseased cells, plant cells, or cell nuclei.

[0039] In addition, the training sample set may also include sample labels, which are used to annotate the target object to be identified in each sample image. During training, the sample labels can be used to compare with the recognition results output by the second image recognition model to determine a portion of the training loss, which will be described in detail below.

[0040] The first image recognition model and the second image recognition model can be used to identify target objects in images, and can also classify or segment images based on the recognition results, such as segmenting target objects from images.

[0041] Here, the first image recognition model can be pre-trained (e.g., it can be trained according to any training method), and the second image recognition model can be a model to be trained. In the training method according to the embodiments of this disclosure, the first image recognition model can be used as a teacher model to transfer knowledge to the second image recognition model in order to train the second image recognition model; the second image recognition model can be used as a student model, and the prediction results of the teacher model can be used to train the student model.

[0042] As an example, the first image recognition model and the second image recognition model can be models with the same architecture (hereinafter also referred to as "homogeneous models"), or they can be models with different architectures (hereinafter also referred to as "heterogeneous models"). For example, the first image recognition model can be a model based on the Swin-T (Swin Transformer) architecture, and the second image recognition model can be a model based on the ResNet18 (Residual Network 18) architecture; or, for another example, the first image recognition model can be a model based on a convolutional network combined with a multilayer perceptron (CNN-MLP) architecture, and the second image recognition model can be a model based on the Transformer-MobileNet architecture.

[0043] The training method for the image recognition model according to embodiments of this disclosure can be combined with existing training schemes for knowledge distillation-based image recognition models to optimize the knowledge distillation process. As an example, existing training schemes for knowledge distillation-based image recognition models may include, but are not limited to, the ViT-CNN transfer learning scheme.

[0044] like Figure 1 As shown, in each training round, the second image recognition model can be trained in the following way:

[0045] In step S110, image information decoupling can be performed on the first recognition result of the first image recognition model in the current training round and the second recognition result of the second image recognition model in the current training round to obtain the first decoupling information corresponding to the first recognition result and the second decoupling information corresponding to the second recognition result.

[0046] Specifically, sample images can be input into a first image recognition model and a second image recognition model respectively, so that the sample images can be recognized by the first image recognition model and the second image recognition model respectively, to obtain a first recognition result output from the first image recognition model and a second recognition result output from the second image recognition model.

[0047] As an example, the first and second recognition results can be in the form of feature maps. In this example, before performing image information decoupling on the first and second recognition results, the spatial dimensions of the feature maps can be aligned. For example, the two feature maps can be aligned to a preset resolution (such as 7×7 resolution) through adaptive pooling.

[0048] In step S110, each of the first decoupling information and the second decoupling information may include overall feature information for characterizing the overall features of the image and detailed feature information for characterizing the detailed features of the image.

[0049] Here, the overall image features may include, for example, features such as the image's structural outline, main color, and main brightness / duration. The overall feature information can be used to describe at least one of the above-mentioned overall image features. The image detail features may include, for example, features such as the image's texture, lines, edges, color changes, and brightness / duration changes. The detail feature information can be used to describe at least one of the above-mentioned image detail features.

[0050] In this step, the overall feature information and detail feature information of the image can be decoupled from the first recognition result and the second recognition result, respectively, so as to determine the loss of the overall feature and detail feature of the image between the two recognition results.

[0051] As an example, in step S110, for each of the first and second recognition results, image information decoupling can be performed in the following manner:

[0052] like Figure 2 As shown, in step S210, the current recognition result can be transformed to the frequency domain for each channel in the spatial dimension to obtain the first frequency domain feature.

[0053] Specifically, the first and second recognition results can be transformed in spatial dimension by performing feature spectrum transformation on the first and second recognition results respectively, transforming the first and second recognition results from the pixel domain to the frequency domain.

[0054] As an example, for each recognition result (It can be the first recognition result or the second recognition result), and a two-dimensional discrete cosine transform (2D DCT) can be performed channel by channel to obtain the first frequency domain feature. Specifically, it can be transformed from the pixel domain to the frequency domain by the following equation (1):

[0055] (1)

[0056] in, This represents the first frequency domain feature at position (u,v) in the feature map after transformation on channel c. This represents the original recognition result (e.g., the feature map output by the model). Indicates the height of the image. Indicates the width of the image. The height of the image is represented by the first... 1 pixel, The width of the image is represented by the first... 1 pixel, This represents the pixel position along the height direction in the transformed feature map. This represents the pixel position along the width direction in the transformed feature map, where, for example, ... Figure 5 As shown, (u=0,v=0) corresponds to low-frequency structural components, while the other frequency coefficients correspond to high-frequency texture components.

[0057] In step S220, the current recognition result can be transformed to the frequency domain in the channel dimension to obtain the second frequency domain feature.

[0058] Specifically, feature spectrum transformation can be performed on the first and second recognition results in the channel dimension to transform the first and second recognition results from the pixel domain to the frequency domain.

[0059] As an example, the channel vector for each recognition result A one-dimensional discrete cosine transform (1D DCT) can be applied along the channel axis to obtain the second frequency domain features. Specifically, the pixel domain can be transformed to the frequency domain using the following equation (2):

[0060] (2)

[0061] in, This indicates the transformed feature map in the channel. The second frequency domain feature, This represents the original recognition result (e.g., the feature map output by the model). This represents the total number of channels in the image, where, for example, ... Figure 6 As shown, k=0 corresponds to semantic information, and k>0 corresponds to instance-specific features.

[0062] The above steps S210 and S220 can be, for example, as follows: Figure 3 As shown, the first and second recognition results can be transformed to the frequency domain through feature projection in the spatial and channel dimensions, respectively. Figure 3 In the diagram, “⊥” indicates that the spatial dimension and the channel dimension are orthogonal.

[0063] Furthermore, despite Figure 2 The illustration shows that step S210 is executed first and then step S220 is executed. However, the embodiments of this disclosure are not limited to this. Step S220 may be executed first and then step S210 may be executed, or the two may be executed in parallel.

[0064] In step S230, the overall image features and the detailed image features can be separated from the first frequency domain features and the second frequency domain features respectively to obtain decoupling information corresponding to the current recognition result.

[0065] In this step, such as Figure 3 As shown, the overall image features and image detail features can be separated from the first frequency domain features and the second frequency domain features by filtering and other methods. In this way, the overall image features and image detail features in each frequency domain feature can be decoupled, and the loss can be calculated separately for the overall features and detail features.

[0066] As an example, binary masks can be used to separate frequency domain features. For example... Figure 4 As shown, the decoupling information corresponding to the current recognition result can be obtained through the following steps:

[0067] In step S410, the spatial overall information and spatial detail information can be separated from the first frequency domain features using the first binary mask.

[0068] Here, spatial global information refers to the image portion with concentrated energy in the current recognition result, such as large color blocks in the image; spatial detail information refers to the image portion with dispersed energy in the current recognition result, such as edges, textures, and noise in the image.

[0069] Specifically, for the first frequency domain feature in the spatial dimension, structural contours and texture details can be distinguished based on their frequency levels. For example, low-frequency information with frequencies u,v=0 can be identified as overall spatial information (also known as low-frequency structural contours); high-frequency information with frequencies u,v>0 can be identified as spatial detail information (also known as high-frequency texture details). In this way, in examples where the first and second image recognition models are heterogeneous models, the differences in the spectrum between the different architectures can be eliminated, allowing the calculation of the loss of spatial overall information and spatial detail information between the two models.

[0070] As an example, overall spatial information can be represented by spatial direct components (DC-spatial), while detailed spatial information can be represented by spatial alternating components (AC-spatial). Figure 5 An example is shown where the spatial DC component and spatial AC component are separated from the first frequency domain feature using a first binary mask, such as... Figure 5 As shown, the overall spatial information can be represented as: ,in, This represents the first binary mask, for example, which can preserve coefficients (u=0, v=0); spatial detail information can be represented as: .

[0071] In step S420, the overall channel information and channel detail information can be separated from the second frequency domain features using the second binary mask.

[0072] Here, the overall channel information represents the overall description of the color, brightness, or luminance of the current recognition result, such as large color blocks, smooth brightness changes, or no brightness changes; the detailed channel information represents the image portion in the current recognition result that has color, brightness, or luminance changes.

[0073] Specifically, for the second frequency domain features in the channel dimension, global semantic information and instance-specific features can be distinguished based on their frequency. For example, information with k=0 can be identified as overall channel information (also known as semantic information component); information with k>0 can be identified as channel detail information (also known as instance-specific features component). Here, by further considering the channel dimension in the spatial dimension, the limitations of single-dimensional decoupling can be overcome, and the distillation effect can be further improved.

[0074] As an example, overall channel information is represented by the channel direct components (DC-channel), while detailed channel information is represented by the channel alternating components (AC-channel). For instance, overall channel information can be represented as: ,in, This represents the second binary mask, for example, which can preserve coefficient k=0 and represent category-level semantic information; channel details can be represented as: This corresponds to the encoded instance-level features for k>0.

[0075] Furthermore, the aforementioned spatial DC component and channel AC component can satisfy orthogonal constraints, such as orthogonal relationships. Fdc spatial, Fac channel =0, thus eliminating cross-dimensional information redundancy.

[0076] In step S430, the overall spatial information and the overall channel information can be used as the overall feature information corresponding to the current recognition result, and the spatial detail information and the channel detail information can be used as the detail feature information corresponding to the current recognition result.

[0077] Return to reference Figure 1 In step S120, the overall feature loss corresponding to the overall feature information and the detailed feature loss corresponding to the detailed feature information can be obtained by comparing the first decoupling information and the second decoupling information.

[0078] For example, such as Figure 3 As shown, the first and second decoupling information in the frequency domain can be inversely transformed back to the pixel domain, and the overall feature loss and detail feature loss can be calculated based on the first and second decoupling information in the pixel domain.

[0079] As an example, in an example where the overall feature information of each decoupled information includes both spatial and channel-level overall information, the overall feature loss can be expressed as: ,in, Indicates loss, , Indicates spatial dimension, Indicates channel dimension, Representing overall information (e.g., DC components). In each example where the detailed feature information of the decoupled information includes spatial detail information and channel detail information, the detail feature loss can be expressed as... , Represents detailed information (such as AC components).

[0080] In step S130, a second image recognition model can be trained based on the overall feature loss, the detailed feature loss, and the knowledge distillation loss trained by knowledge distillation.

[0081] In this step, the decoupled global feature loss and detailed feature loss calculated above can be combined with the original loss from knowledge distillation training as the final prediction loss (e.g., Figure 3 MSE loss in the middle.

[0082] In one example, the overall feature loss, the detailed feature loss, and the knowledge distillation loss from knowledge distillation training can be directly added together as the final prediction loss.

[0083] In another example, the overall feature loss and the detailed feature loss can be weighted, and the weighted overall feature loss, the weighted detailed feature loss, and the knowledge distillation loss can be added together to obtain the final prediction loss.

[0084] Specifically, step S130 may include: weighting the overall feature loss and the detail feature loss respectively; and training a second image recognition model based on the weighted overall feature loss, the weighted detail feature loss, and the knowledge distillation loss.

[0085] Here, adjustable weight parameters can be used to weight the overall feature loss and the detailed feature loss.

[0086] As an example, knowledge distillation loss can include the original task loss and the basic distillation loss. The basic distillation loss can be obtained by comparing the second recognition result with the first recognition result (e.g., which can be understood as a soft label), and the original task loss can be obtained by comparing the second recognition result with the sample label (e.g., which can be understood as a true label).

[0087] In this example, the final prediction loss can be represented by the following equation (3):

[0088] (3)

[0089] in, This represents the final predicted loss (or total loss). This represents the original task loss in the knowledge distillation loss. This represents the basic distillation loss in knowledge distillation loss, such as the basic Vanella KD loss. Indicates the overall feature loss. This represents the loss of detailed features, specifically the loss of overall spatial information. It can optimize structural contour alignment and compensate for the loss of channel detail information. It can enhance the transfer of detailed features. and These represent the weighting parameters used to weight the overall feature loss and the detail feature loss, respectively, and they can be preset.

[0090] In the above process, by introducing adjustable weight parameters, the distillation intensity of overall information and detailed information can be dynamically controlled, further solving the problem of detailed information being suppressed due to the dominance of overall information.

[0091] Furthermore, as an example, as described above, the overall feature loss may include a first loss for the spatial overall information between the first and second decoupling information and a second loss for the channel overall information between the first and second decoupling information. The detailed feature loss may include a third loss for the spatial detailed information between the first and second decoupling information and a fourth loss for the channel detailed information between the first and second decoupling information. In this case, the weight parameters may be expressed, for example, as follows: ,in, Used to weight the first loss. Used to weight the third loss. Used to weight the second loss. Used to weight the fourth loss.

[0092] Here, regarding the first loss The weight parameters are greater than those for the second loss. The weight parameters for the third loss The weight parameters are greater than those for the fourth loss. The weight parameters.

[0093] In this way, by constraining the relationship between the weight parameters of different losses, the dominant effect of overall information can be suppressed, while the knowledge transfer capability of detailed information can be enhanced.

[0094] For example, when the first image recognition model is a visual Transformer architecture (e.g., Swing-T), set =0.03 can suppress its low-frequency dominance effect (ViT's DC energy accounts for over 70%); when the second image recognition model is a lightweight CNN architecture (e.g., MobileNetV2), setting =0.07, which can enhance the ability to migrate instance details.

[0095] Furthermore, in step S130, the specific optimization objective of the loss function is to minimize the difference between the output prediction of the second image recognition model and the true label. Based on this optimization objective, the weight parameters of the second image recognition model can be updated to train the second image recognition model.

[0096] Figure 7A schematic flowchart illustrating an application example of a training method for an image recognition model according to an exemplary embodiment of the present disclosure is shown.

[0097] like Figure 7 As shown, in step S710, a first recognition result and a second recognition result can be obtained, and in step S720, the first recognition result and the second recognition result can be subjected to spatial dimension and channel dimension DCT transformation and orthogonal constraint decoupling, respectively. Specifically, for the spatial dimension, 2D DCT can be performed to obtain the spatial DC component representing the structural contour and the spatial AC component representing the texture details; for the channel dimension, 1D DCT can be performed to obtain the channel DC component representing the category semantics and the channel AC component representing the instance features.

[0098] In step S730, the losses corresponding to each component obtained in step S720 can be weighted to achieve dynamic spectral balance. In step S740, the weighted loss can be fused with the original task loss and the basic distillation loss to obtain the final predicted loss.

[0099] In step S750, the parameters of the second image recognition model can be updated based on the final prediction loss to train the second image recognition model. Steps S710 to S750 can be executed in multiple rounds to achieve multiple rounds of training, ultimately outputting a trained second image recognition model. This second image recognition model can be a lighter model than the first image recognition model; for example, the number of parameters in the second image recognition model can be much smaller than the number of parameters in the first image recognition model.

[0100] The training method for the image recognition model according to the embodiments of this disclosure can overcome the cross-architecture feature mismatch problem caused by the imbalance of spectrum distribution in traditional knowledge distillation by frequency domain decoupling, so that the final output is a lightweight student model (i.e. the second image recognition model mentioned above). The trained model can be directly deployed on edge devices to support visual tasks such as image classification, object detection, and semantic segmentation.

[0101] Furthermore, the training method for the image recognition model according to embodiments of this disclosure effectively solves the problem of semantic-instance feature imbalance in knowledge distillation for image recognition by decoupling spatial and channel frequencies and dynamically adjusting weights, providing a new paradigm for lightweight model deployment. Specifically, in this method, when updating student parameters during backpropagation, the gradient allocation is adjusted by introducing dynamic weight parameters, which can further solve the cross-architecture spectrum mismatch problem. Moreover, through a hierarchical loss function, the two-dimensional decoupling loss is dynamically fused with the original task loss and the basic distillation loss. Through backpropagation gradient optimization, the student model is guided to absorb the representational capabilities of the teacher model while retaining its own efficiency advantages.

[0102] Furthermore, in the knowledge distillation of related technologies, when the teacher model and student model are heterogeneous models, the different architectures of the models have different information extraction capabilities and methods, and the energy distribution of the model outputs on the spectrum also differs, resulting in a loss of cross-architecture learning effect. To address this, the training method according to the embodiments of this disclosure is adopted. By decoupling the information of different features at the feature level, key information can be learned separately, thereby balancing the distillation learning between different features and improving the distillation effect of heterogeneous models.

[0103] In a second aspect of an exemplary embodiment of the present disclosure, an image recognition method is provided, which may include: acquiring a target image; inputting the target image into an image recognition model; using the image recognition model to recognize a target object in the target image and obtaining a recognition result, wherein the image recognition model is trained according to the image recognition model training method of the embodiments of the present disclosure, for example, it may be the second image recognition model trained as described above.

[0104] The specific steps and details of the image recognition method according to the exemplary embodiments of this disclosure can be implemented and understood similarly with reference to the training method of the image recognition model described in the first aspect above, and accordingly, the image recognition method has the same or similar beneficial effects as the training method of the image recognition model described above, which will not be repeated here.

[0105] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing processor-executable instructions, wherein, when executed by the processor, the processor-executable instructions cause the processor to perform a training method for an image recognition model according to an embodiment of the present disclosure or an image recognition method according to an embodiment of the present disclosure.

[0106] As an example, an electronic device does not necessarily have to be a single device; it can also be a collection of any means or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. An electronic device can also be part of an integrated control system or system manager, or can be configured to interface with a server, either locally or remotely (e.g., via wireless transmission).

[0107] In electronic devices, processors may include central processing units (CPUs), graphics processing units (GPUs), programmable logic devices, dedicated processor systems, microcontrollers, or microprocessors. By way of example and not limitation, processors may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0108] The processor can execute instructions or code stored in memory, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transport protocol.

[0109] Memory can be integrated with the processor; for example, RAM or flash memory can be housed within an integrated circuit microprocessor. Alternatively, memory can comprise a separate device, such as an external disk drive, storage array, or other storage device that can be used by any database system. Memory and processor can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor to read files stored in the memory.

[0110] In addition, electronic devices may include video displays (such as liquid crystal displays) and user interaction interfaces (such as keyboards, mice, touch input devices, etc.). All components of the electronic device can be interconnected via buses and / or networks.

[0111] In an exemplary embodiment, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the training method or image recognition method of the image recognition model as described in the exemplary embodiment above. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: a read-only memory (ROM), a random access memory (RAM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), flash memory, non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, a BD-R, or a BD-R... LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0112] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, the computer program product including computer-executable instructions, which, when executed by at least one processor, implement a training method or an image recognition method for an image recognition model according to exemplary embodiments of the present disclosure.

[0113] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0114] Furthermore, it should be noted that although several examples of each step have been described above with reference to the specific accompanying drawings, it should be understood that the embodiments of this disclosure are not limited to the combinations given in the examples. The steps appearing in different drawings can be combined, and the execution order of each step can be changed, which will not be exhaustive here.

[0115] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for an image recognition model, characterized in that, The training method includes: Based on the training sample set and the first image recognition model, the second image recognition model is trained using knowledge distillation to obtain a trained second image recognition model. The training sample set includes multiple sample images, and the first and second image recognition models are used to identify target objects in the sample images. In each training round, the second image recognition model is trained in the following manner: Image information decoupling is performed on the first recognition result of the first image recognition model in the current training round and the second recognition result of the second image recognition model in the current training round, respectively, to obtain first decoupling information corresponding to the first recognition result and second decoupling information corresponding to the second recognition result. Each decoupling information in the first decoupling information and the second decoupling information includes overall feature information for characterizing the overall features of the image and detailed feature information for characterizing the detailed features of the image. By comparing the first decoupling information and the second decoupling information, the overall feature loss corresponding to the overall feature information and the detailed feature loss corresponding to the detailed feature information are obtained. The second image recognition model is trained based on the overall feature loss, the detailed feature loss, and the knowledge distillation loss trained by knowledge distillation.

2. The training method according to claim 1, characterized in that, For each of the first and second recognition results, the image information decoupling is performed in the following manner: In the spatial dimension, for each channel, the current recognition result is transformed to the frequency domain to obtain the first frequency domain feature; In the channel dimension, the current recognition result is transformed to the frequency domain to obtain the second frequency domain feature; The overall image features and image detail features are separated from the first frequency domain features and the second frequency domain features, respectively, to obtain decoupling information corresponding to the current recognition result.

3. The training method according to claim 2, characterized in that, The step of separating the overall image features and image detail features from the first frequency domain features and the second frequency domain features, respectively, to obtain decoupling information corresponding to the current recognition result includes: Using a first binary mask, spatial overall information and spatial detail information are separated from the first frequency domain features; using a second binary mask, channel overall information and channel detail information are separated from the second frequency domain features. The overall spatial information and the overall channel information are used as the overall feature information corresponding to the current recognition result, and the spatial detail information and the channel detail information are used as the detail feature information corresponding to the current recognition result. Wherein, the spatial overall information represents the image portion with concentrated energy in the current recognition result, the spatial detail information represents the image portion with dispersed energy in the current recognition result, the channel overall information represents the overall description of the color, brightness, or luminance of the current recognition result, and the channel detail information represents the image portion in the current recognition result where there are color, brightness, or luminance changes.

4. The training method according to claim 3, characterized in that, The overall spatial information is represented by a spatial DC component, the detailed spatial information is represented by a spatial AC component, the overall channel information is represented by a channel DC component, and the detailed channel information is represented by a channel AC component. The spatial DC component and the channel AC component satisfy an orthogonal constraint.

5. The training method according to claim 1, characterized in that, The training of the second image recognition model based on the overall feature loss, the detailed feature loss, and the knowledge distillation loss trained by knowledge distillation includes: The overall feature loss and the detailed feature loss are weighted separately; The second image recognition model is trained based on the weighted overall feature loss, the weighted detailed feature loss, and the knowledge distillation loss.

6. The training method according to claim 5, characterized in that, The overall feature information of each decoupling information in the first and second decoupling information includes overall spatial information and overall channel information. The detailed feature information of each decoupling information in the first and second decoupling information includes spatial detail information and channel detail information. Wherein, the overall spatial information represents the image portion with dispersed energy in the current recognition result, the spatial detail information represents the image portion with concentrated energy in the current recognition result, the overall channel information represents the overall description of the color, brightness, or luminance of the current recognition result, and the channel detail information represents the image portion in the current recognition result where there are color, brightness, or luminance variations. The overall feature loss includes: a first loss of the overall spatial information between the first decoupling information and the second decoupling information, and a second loss of the overall channel information between the first decoupling information and the second decoupling information. The detailed feature loss includes: a third loss corresponding to the spatial detail information between the first decoupling information and the second decoupling information, and a fourth loss corresponding to the channel detail information between the first decoupling information and the second decoupling information. The weight parameter for the first loss is greater than the weight parameter for the second loss, and the weight parameter for the third loss is greater than the weight parameter for the fourth loss.

7. An image recognition method, characterized in that, The image recognition method includes: Acquire the target image; The target image is input into an image recognition model, and the target object in the target image is identified using the image recognition model to obtain a recognition result. The image recognition model is trained by the image recognition model training method according to any one of claims 1 to 6.

8. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store processor-executable instructions. When the processor executes the processor, it causes the processor to perform the training method of the image recognition model according to any one of claims 1 to 6 or the image recognition method according to claim 7.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the training method of the image recognition model according to any one of claims 1 to 6 or the image recognition method according to claim 7.

10. A computer program product comprising computer-executable instructions, characterized in that, When the computer-executable instructions are executed by at least one processor, they implement the training method of the image recognition model according to any one of claims 1 to 6 or the image recognition method according to claim 7.

Images