Intelligent object recognition method, device, and equipment based on long-tail dynamic weight adjustment

Abstract

This application relates to the field of equipment and item recognition technology, and discloses an intelligent item recognition method and apparatus based on long-tail dynamic weight adjustment. The method includes: acquiring an image sample of an item to be recognized; inputting the image sample into a trained item recognition model; the item recognition model is a student model trained based on differential distillation constraints applied to sample frequencies of different item categories; and outputting the category recognition result of the item to be classified. Since the item recognition model is a student model trained based on differential distillation constraints applied to sample frequencies of different item categories, distillation constraints of different intensities can be applied to item categories with different sample numbers, thereby enhancing the student model's learning ability for item categories with fewer samples, mitigating the adverse effects of uneven distribution of category samples on recognition performance, and improving the accuracy of item recognition.

Description

Technical Field

[0001] This application relates to the field of equipment and item recognition technology, such as a smart item recognition method, device, and equipment based on long-tail dynamic weight adjustment. Background Technology

[0002] Currently, with the development of smart home technology, smart devices are gradually becoming an important part of the smart home scenario. Item recognition is the foundation for smart devices to realize higher-level services such as inventory management, analysis, recommendation, and automatic reminders. Therefore, there is a strong practical application demand for accurately identifying items within devices.

[0003] To address the aforementioned needs, a method for object recognition has been disclosed, employing a convolutional neural network to extract features and classify object images. Considering the limited computing resources at the edge of smart devices, a knowledge distillation method is also disclosed, comprising: constructing a teacher model and a student model; and using the output of the teacher model or intermediate layer features to distill and train the student model, thereby reducing the number of model parameters and computational load while maintaining recognition accuracy as much as possible.

[0004] In the process of implementing the embodiments of this disclosure, at least the following problems were found in the related art:

[0005] While related technologies can improve the recognition capabilities of lightweight student models through knowledge distillation, a long-tail distribution exists because some common items have a relatively large number of samples due to their high frequency of occurrence, while some rare items have a relatively small number of samples due to their low frequency of occurrence. For this type of long-tailed data, the student model tends to prioritize learning the features of categories with a larger number of samples during training, while insufficiently learning the features of categories with a smaller number of samples. This results in poor recognition performance for the tail categories, thus affecting the overall recognition performance.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0008] This disclosure provides an intelligent item recognition method, apparatus, and device based on long-tail dynamic weight adjustment to improve the recognition accuracy of tail-category items in real-world scenarios.

[0009] In some embodiments, the intelligent item recognition method based on long-tail dynamic weight adjustment includes: acquiring image samples of the item to be recognized; inputting the image samples into a trained item recognition model; the item recognition model is trained as follows: acquiring the distillation classification joint loss, the frequency domain branch distillation loss after dynamic weight adjustment, and the spatial branch distillation loss; performing weighted fusion of the distillation classification joint loss, the frequency domain branch distillation loss after dynamic weight adjustment, and the spatial branch distillation loss to obtain the total training loss, and training a student model based on the total training loss, and using the trained student model as the item recognition model; wherein, the frequency domain branch distillation loss after dynamic weight adjustment is obtained by implementing differentiated weight adjustment based on the frequency domain branch distillation loss of different item category samples; and outputting the category recognition result of the item to be recognized.

[0010] Optionally, obtaining an image sample of the current object to be identified includes:

[0011] The acquired image of the object to be identified is scaled to a preset size;

[0012] The image of the item of the preset size is normalized to a preset numerical range in the following way to obtain the image sample of the item to be identified:

[0013] ;

[0014] in, These are the normalized pixel values. These are the original pixel values. and These are the maximum and minimum pixel values ​​in the image, respectively.

[0015] Optionally, the frequency domain branch distillation loss after dynamic weight adjustment is obtained in the following way:

[0016] Count the number of samples corresponding to each item category in the training set;

[0017] The item categories are divided into long-tail levels based on the number of samples corresponding to each item category;

[0018] Different distillation weights are assigned to different item categories based on the long-tail classification results;

[0019] The frequency domain branch distillation loss of the student model is adjusted using the distillation weights to obtain the dynamically weighted frequency domain branch distillation loss.

[0020] Optionally, the step of performing long-tail ranking of the item categories based on the number of samples corresponding to each item category includes:

[0021] Sort the number of samples for each item category;

[0022] The head category is determined based on the preset head proportion, the tail category is determined based on the preset tail proportion, and all other item categories, excluding the head and tail categories, are classified as the middle category.

[0023] Optionally, assigning different distillation weights to different item categories based on the long-tail ranking results includes:

[0024] Assign a first distillation weight to the head category; assign a second distillation weight to the middle category; assign a third distillation weight to the tail category; wherein the third distillation weight is greater than the first distillation weight, and / or the third distillation weight is greater than the second distillation weight.

[0025] Optionally, the frequency domain branch distillation loss of the student model is adjusted using the distillation weights, including:

[0026] During training, read the class labels of the current batch of samples;

[0027] Determine the long-tail level corresponding to the item category of the current batch of samples based on the category label;

[0028] The distillation weight corresponding to the current batch of samples is obtained based on the long-tail level.

[0029] The frequency domain branch distillation loss is weighted using the distillation weights to obtain the weighted frequency domain branch distillation loss.

[0030] Optionally, if the distillation loss is a frequency domain branch distillation loss, then the weighted frequency domain branch distillation loss is calculated according to the following formula:

[0031] ;

[0032] in, This represents the weighted frequency domain branching distillation loss. Indicates the first yi The distillation weights corresponding to the categories of each sample. Let N represent the frequency domain branch distillation loss corresponding to the i-th sample, and N represent the batch size.

[0033] Optionally, the total training loss Calculate according to the following formula:

[0034] ;

[0035] in, For total training losses, For the combined loss of distillation classification, This refers to the frequency domain branch distillation loss after dynamic weight adjustment. For spatial branching distillation losses, These are the weighting coefficients for the distillation classification joint loss, the frequency domain branch distillation loss after dynamic weight adjustment, and the spatial branch distillation loss, respectively.

[0036] In some embodiments, the smart item recognition device based on long-tail dynamic weight adjustment includes a processor and a memory storing program instructions, the processor being configured to execute the smart item recognition method based on long-tail dynamic weight adjustment as described above when running the program instructions.

[0037] In some embodiments, the device includes: a device body; and the smart object recognition device based on long-tail dynamic weight adjustment as described above, which is installed on the device body.

[0038] The intelligent item recognition method, apparatus, and device based on long-tail dynamic weight adjustment provided in this disclosure can achieve the following technical effects:

[0039] This embodiment of the disclosure acquires image samples of the item to be identified and inputs these image samples into a trained item recognition model. The model can then directly output the category identification result of the item, thereby meeting the application requirements of automatic item recognition on the device side. Since the item recognition model is a student model trained based on differentiated distillation constraints applied to different item category sample frequencies, distillation constraints of varying strengths can be applied to item categories with different sample numbers. This enhances the student model's learning ability for item categories with fewer samples, mitigates the adverse effects of uneven category sample distribution on recognition performance, and improves the accuracy of item recognition.

[0040] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0041] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein:

[0042] Figure 1 This is a schematic diagram of the implementation environment of an embodiment of this disclosure;

[0043] Figure 2 This is a flowchart illustrating the first intelligent item recognition method based on long-tail dynamic weight adjustment provided in this disclosure embodiment;

[0044] Figure 3 This is a schematic diagram of the training process of the item recognition model in an embodiment of this disclosure;

[0045] Figure 4 This is a schematic diagram of the training process of frequency domain branch distillation of the student model in an embodiment of this disclosure;

[0046] Figure 5 This is a schematic diagram of the training process of spatial branch distillation of the student model in an embodiment of this disclosure;

[0047] Figure 6 This is a schematic diagram illustrating the process of constructing the total training loss to train the student model in an embodiment of this disclosure;

[0048] Figure 7 This is a flowchart illustrating the second intelligent item recognition method based on long-tail dynamic weight adjustment provided in this disclosure embodiment;

[0049] Figure 8 This is a schematic diagram of an intelligent item recognition device based on long-tail dynamic weight adjustment provided in an embodiment of this disclosure. Detailed Implementation

[0050] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0051] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0052] Unless otherwise stated, the term "multiple" means two or more.

[0053] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0054] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0055] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0056] With the development of smart home technology, smart devices have gone beyond mere storage and organization, gradually acquiring functions such as item recognition, inventory management and analysis, usage recommendations, and expiration reminders. Item recognition is fundamental to these functions. Only by accurately identifying items within the device can higher-level applications such as inventory management, usage suggestions, and replenishment reminders be further supported.

[0057] Currently, most object recognition methods for smart devices employ deep learning models for feature extraction and category identification of object images. To adapt to application scenarios with limited computing and storage resources at the edge of smart devices, it is typically necessary to lightweight the model while maintaining recognition accuracy. Knowledge distillation, a commonly used model compression method, utilizes a teacher model to guide student model training, enabling student models to achieve good recognition performance with a smaller parameter scale. Therefore, it is widely used in edge-side visual recognition tasks.

[0058] However, directly applying traditional knowledge distillation methods to smart device item recognition scenarios still presents the following problems. Item data in smart devices often exhibits a significant class imbalance, with some common items having a large number of samples due to high usage frequency, while some less common items have only a few samples due to low occurrence frequency, thus forming a long-tail distribution. Traditional distillation methods typically apply the same strength of distillation constraints to different categories, making it difficult to specifically adjust the distillation process based on the differences in sample numbers between different categories. This causes student models to tend to favor categories with larger sample numbers during training, while learning the categories with smaller sample numbers insufficiently, thereby affecting the recognition performance of the tail categories.

[0059] Based on this, this embodiment proposes an intelligent item recognition method based on long-tail dynamic weight adjustment. In this method, the distillation process is differentiated by the sample frequency of different item categories, so that categories with fewer samples receive stronger distillation enhancement during training. This improves the item recognition performance of the student model in scenarios with imbalanced category distribution, and is particularly beneficial for improving the recognition ability of tail categories.

[0060] Figure 1 This is a schematic diagram illustrating the implementation environment of an embodiment of this disclosure. For example... Figure 1 As shown, the implementation environment may include an image acquisition device 100, a device-side object recognition device 200, a network device 300, and a training server 400.

[0061] The image acquisition device 100 is used to acquire images of items within the device. The image acquisition device 100 can be installed inside the device body and is used to acquire image samples of items to be identified in the device's storage area. The image acquisition device 100 can be a camera module, such as an RGB camera, or other image sensing devices capable of acquiring images of items; this disclosure does not limit the specific device used.

[0062] The device-side item recognition device 200 is used to perform recognition processing on image samples acquired by the image acquisition device 100. A pre-trained item recognition model can be deployed in the device-side item recognition device 200, and inference can be performed on the item images based on the item recognition model to obtain the item category recognition result. The device-side item recognition device 200 can be installed in the device body or can be an edge computing device communicatively connected to the device body. The device-side item recognition device 200 may include a processor, a memory, and program instructions stored in the memory and executable on the processor.

[0063] Network device 300 is used to enable data communication between the device-side object recognition device 200 and the training server 400. Network device 300 can be a router, gateway, or other network communication device, and this embodiment does not limit this. The device-side object recognition device 200 can establish a communication connection with the training server 400 through network device 300 to achieve model download, model update, or data interaction.

[0064] The training server 400 is used to train the item recognition model. The training server 400 can train the student model based on differential distillation constraints according to the sample frequency of different item categories to obtain a trained item recognition model. The training server 400 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center; this embodiment of the disclosure does not limit this.

[0065] It should be understood that Figure 1 The number of image acquisition device 100, device-side object recognition device 200, network device 300, and training server 400 shown is merely illustrative. Depending on actual needs, any number of these devices can be used. For example, one training server 400 can correspond to multiple device-side object recognition devices 200, and these devices 200 can be deployed in different devices.

[0066] It should be noted that the device-oriented item recognition method provided in this embodiment can be jointly implemented by a training server 400 and a device-side item recognition device 200. The training server 400 is used to train the item recognition model, and the device-side item recognition device 200 is used to deploy the trained item recognition model and obtain the item recognition result based on the item image collected by the image acquisition device 100. Correspondingly, the relevant devices in this embodiment can also be respectively set in the training server 400 and the device-side item recognition device 200.

[0067] Figure 2 This is a flowchart illustrating the first intelligent item recognition method based on long-tail dynamic weight adjustment provided in this disclosure embodiment, applied to... Figure 1 The implementation environment shown.

[0068] Combination Figure 2 As shown, the smart object recognition method includes:

[0069] Step S201: Obtain an image sample of the current object to be identified.

[0070] An image sample of the object to be identified refers to the image data of the object acquired by the image acquisition device and used for the current identification task. This image sample contains appearance information of the object to be identified, such as visual features like outline, structure, texture, color, or surface morphology, and serves as the basic data for subsequent inference by the object recognition model.

[0071] Here, by invoking an image acquisition device installed inside the device, images of items in the storage area are captured, resulting in image samples of the items to be identified. To ensure subsequent recognition effectiveness, the image acquisition device can periodically acquire images or trigger acquisition when the user opens or closes the device door, or when the internal state of the device changes. Images of items in the current storage area can be directly obtained on the device, providing raw input data for subsequent model inference. This step enables the solution to perform recognition in real-world device usage scenarios, rather than relying on offline input data, thus laying the foundation for real-time item recognition on the device.

[0072] Step S202: Input the image samples into the trained item recognition model; the item recognition model is trained as follows: obtain the distillation classification joint loss, the frequency domain branch distillation loss after dynamic weight adjustment, and the spatial branch distillation loss; perform weighted fusion of the distillation classification joint loss, the frequency domain branch distillation loss after dynamic weight adjustment, and the spatial branch distillation loss to obtain the total training loss, and train the student model based on the total training loss, and use the trained student model as the item recognition model; wherein, the frequency domain branch distillation loss after dynamic weight adjustment is obtained by applying differentiated weight adjustment to the frequency domain branch distillation loss based on the sample frequency of different item categories.

[0073] An object recognition model refers to a student model that has completed parameter training and can be deployed for inference. This model receives object image samples as input and outputs the corresponding object category recognition result. In this scheme, this model is not a model obtained through ordinary training, but rather a student model trained using differential distillation constraints. This student model can employ a lightweight network, such as MobileNetV3-Small, and is deployed to the edge of smart devices after training.

[0074] The sample frequency of different item categories refers to the distribution of the number of samples corresponding to each item category in the training set, reflecting the frequency of occurrence of different item categories in the training data. Categories with higher sample frequencies usually correspond to head categories, while categories with lower sample frequencies usually correspond to tail categories.

[0075] Differential weight adjustment refers to applying different levels of weights to the frequency domain branch distillation loss based on the differences in sample frequencies among different item categories, so that different categories receive varying degrees of learning reinforcement during distillation training. The core principle is that instead of applying the same distillation intensity to all categories, different distillation weights are assigned to categories with different sample sizes.

[0076] Here, the acquired image samples are input into the trained item recognition model, which then performs forward inference. The trained item recognition model is a student model trained based on differential distillation constraints applied to the sample frequencies of different item categories. In other words, during model training, the number of samples corresponding to different item categories is first counted, and then different constraint strengths are applied to the distillation process based on the differences in sample frequencies, thus obtaining the trained student model. During the deployment and recognition phase, this trained student model is directly used to perform category inference on the input image.

[0077] Step S203: Output the category identification result of the item to be identified.

[0078] Category recognition results refer to the results obtained by the trained object recognition model after performing inference on the input image samples of the object to be recognized. In this scheme, the category recognition results can at least include the object category prediction results.

[0079] Here, the prediction results corresponding to each item category output by the item recognition model are read, and the item category corresponding to the current item image is determined based on the prediction results; this category is then identified as the item recognition result. Furthermore, the item recognition result can also be output to the device's inventory management, expiration reminder, or health recommendation modules for subsequent function calls.

[0080] In this embodiment, image samples of the item to be identified are first acquired, then input into a trained item recognition model for forward inference, and the category identification result of the item to be classified is output. Since the item recognition model is a student model trained based on differential distillation constraints using sample frequencies of different item categories, the distillation training process can be specifically adjusted according to the differences in the number of samples for different categories. This enhances the student model's learning ability for categories with fewer samples, alleviates recognition bias caused by uneven distribution of category samples, and improves item recognition performance.

[0081] Since images acquired under different acquisition conditions may differ in resolution, size, and pixel distribution, the images of the objects to be identified can be preprocessed before inputting them into the trained object recognition model to ensure that the input images meet the model's input requirements.

[0082] Further, in step S201, obtaining an image sample of the current item to be identified includes:

[0083] The acquired image of the object to be identified is scaled to a preset size;

[0084] After normalizing the image of an item of a preset size to a preset numerical range using the following method, an image sample of the item to be identified is obtained:

[0085] (1)

[0086] in, These are the normalized pixel values. These are the original pixel values. and These are the maximum and minimum pixel values ​​in the image, respectively.

[0087] Here, the acquired image of the object to be recognized is first scaled to a preset size. The preset size can be pre-set according to the input dimension of the trained object recognition model, for example, it can be 224×224. By scaling the image, original images of different sizes can be uniformly adjusted to an input size acceptable to the model, thereby ensuring consistency in the subsequent inference process.

[0088] After scaling, the image of the item at the preset size can be further normalized. Specifically, this normalization can be performed based on the maximum value among the image's pixel values. and minimum value The above formula is used to perform a linear mapping on each pixel value, transforming the original pixel value I into a normalized pixel value. This method maps image pixel values ​​to a uniform numerical range, thereby reducing data fluctuations between different image samples due to differences in brightness, contrast, or acquisition environment. For example, the preset numerical range for normalization can be the [0,1] interval.

[0089] Thus, by performing image scaling and normalization, an image sample of the object to be identified can be obtained, and this image sample can be used as input to the trained object recognition model. This ensures that the object images acquired by the device meet the model's input size requirements, and also keeps the image data distribution during deployment as consistent as possible with that during training, thereby improving the stability and accuracy of object recognition.

[0090] In some embodiments, the trained object recognition model can be deployed on an edge device at the device end, so that the acquired object images can be directly recognized and processed on the device side. Specifically, the training process of the object recognition model is described below with reference to specific embodiments.

[0091] Figure 3 This is a schematic diagram of the training process of the item recognition model in an embodiment of this disclosure.

[0092] The trained object recognition model is deployed on edge devices at the device level; combined with Figure 3 As shown, the frequency domain branch distillation loss of the item recognition model after dynamic weight adjustment is obtained as follows:

[0093] Step S301: Count the number of samples corresponding to each item category in the training set.

[0094] It can iterate through all training samples in the training set and count the number of samples corresponding to each item category; after obtaining the number of samples for each item category, it can also sort the sample counts in descending order to obtain the category distribution in the training set.

[0095] Step S302: Divide the item categories into long-tail levels based on the number of samples corresponding to each item category.

[0096] The head and tail proportions can be preset, and the head, middle, and tail categories can be determined based on the number of samples for each sorted item category. Categories with the largest number of samples can be classified as head categories, categories with the largest number of samples as tail categories, and the remaining categories as middle categories.

[0097] Step S303: Assign different distillation weights to different item categories based on the long-tail classification results.

[0098] Step S304: The frequency domain branch distillation loss of the student model is adjusted using the distillation weights to obtain the frequency domain branch distillation loss after dynamic weight adjustment.

[0099] In each training batch, the category label of the current batch sample is read, the long-tail level corresponding to the item category to which the current sample belongs is determined based on the category label, and the distillation weight corresponding to the sample is further obtained; then, the distillation weight is used to weight the frequency domain branch distillation loss corresponding to the current sample, so as to obtain the weighted frequency domain branch distillation loss.

[0100] In some alternative implementations, the number of samples corresponding to each item category can be sorted in descending order before performing long-tail ranking.

[0101] In some alternative implementations, the weighting of the frequency domain branch distillation loss can be dynamically performed within each training batch based on the class label of the current batch of samples.

[0102] Further, in step S302, the item categories are divided into long-tail levels based on the number of samples corresponding to each item category, including:

[0103] Sort the number of samples for each item category;

[0104] The head category is determined based on the preset head proportion, the tail category is determined based on the preset tail proportion, and all other item categories, excluding the head and tail categories, are classified as the middle category.

[0105] For example, the number of samples in the K categories counted in the training set. ,in For category indexing, sorted in descending order by sample count: ;

[0106] in Indicates the sorted order of the first... Larger sample size It is the head category with the most samples. It is the tail category with the fewest samples.

[0107] Set head proportions and tail proportions The categories are divided into three levels: head categories The top sample size Categories; Central Category It's in the middle. Categories, tail categories It is the last one with the fewest samples There are several categories.

[0108] By using the above-mentioned long-tail classification method, the item categories can be divided into head categories, middle categories, and tail categories according to the category distribution in the training set. This provides a basis for subsequently assigning different distillation weights according to different long-tail levels, which is beneficial to enhancing the learning ability of the student model for categories with fewer samples and mitigating the adverse effects of long-tail distribution on item recognition performance.

[0109] Further, in step S303, different distillation weights are assigned to different item categories based on the long-tail classification results, including:

[0110] Assign a first distillation weight to the head category; assign a second distillation weight to the middle category; assign a third distillation weight to the tail category; wherein the third distillation weight is greater than the first distillation weight, and / or the third distillation weight is greater than the second distillation weight.

[0111] Here, by making the third distillation weight corresponding to the tail category greater than the first distillation weight corresponding to the head category, and / or greater than the second distillation weight corresponding to the middle category, the tail category, with its smaller sample size, can obtain stronger loss constraints during the frequency domain branch distillation process. This enhances the student model's ability to learn low-frequency contour and structural information of the tail category on the one hand, and alleviates the problem of excessive bias towards the head category during training on the other, thereby improving the student model's ability to recognize categories with scarce samples and mitigating the adverse effects of long-tail distribution on item recognition performance.

[0112] For example, distillation weights are assigned to different item categories based on the long-tail ranking results described above. :

[0113] (2)

[0114] By classifying the tail The corresponding distillation weight is set to 2.0, while the head category... and central category Setting the corresponding distillation weight to 1.0 allows the tail category to receive a higher loss constraint strength than the head and middle categories during distillation training. This enhances the student model's ability to learn the frequency domain features corresponding to the tail category, especially low-frequency contours and structural information. Simultaneously, it appropriately suppresses excessive bias towards the head and middle categories during overall training, thereby improving the recognition ability of scarce categories and mitigating the class imbalance problem caused by long-tail distribution.

[0115] Further, in step S304, during the training process, the frequency domain branch distillation loss of the student model is adjusted using the distillation weights, including:

[0116] During training, read the class labels of the current batch of samples;

[0117] Determine the long-tail level corresponding to the item category of the current batch of samples based on the category label;

[0118] Obtain the distillation weight corresponding to the current batch of samples based on the long-tail level;

[0119] The frequency domain branch distillation loss is weighted using the distillation weights to obtain the weighted frequency domain branch distillation loss.

[0120] Here, the category label indicates the item category to which the i-th sample belongs; a mapping relationship between "category label - long-tail level" can be pre-established; after completing the category frequency statistics and long-tail level classification, each item category is correspondingly labeled as a head category, middle category, or tail category. During training, when the category label of a sample is read, the long-tail level corresponding to the item category to which the sample belongs can be determined according to the mapping relationship. Then, according to the pre-established mapping relationship between "long-tail level - distillation weight", after determining the long-tail level to which the current sample belongs, the distillation weight corresponding to the sample can be obtained by looking up a table or index. The distillation loss of each sample in the current batch is calculated separately, and then the distillation weight corresponding to the sample is multiplied by the frequency domain branch distillation loss to obtain the weighted frequency domain branch distillation loss of the sample; further, the weighted losses of each sample in the batch are summed or aggregated to obtain the weighted frequency domain branch distillation loss.

[0121] Specifically, the frequency domain branch distillation loss after weight adjustment is calculated according to the following formula:

[0122] (3)

[0123] in, This represents the weighted frequency domain branching distillation loss. Indicates the first yi Distillation weights corresponding to each sample category Let N represent the frequency domain branch distillation loss corresponding to the i-th sample, and N represent the batch size.

[0124] By using the above method, differentiated distillation constraints can be implemented for samples of different long-tail levels during training, thereby enabling tail categories with fewer samples to obtain stronger distillation intensity during training.

[0125] The distillation loss described above is the frequency domain branch distillation loss. Accordingly, before adjusting the distillation loss according to the distillation weights, the frequency domain branch distillation loss can be obtained first. Since the frequency domain branch distillation loss is used to characterize the difference between the student model and the teacher model in terms of frequency domain feature representation, the specific implementation of frequency domain branch distillation is explained below to facilitate understanding of the process of obtaining the weighted frequency domain branch distillation loss.

[0126] Figure 4 This is the training process of frequency domain branch distillation of the student model in this embodiment of the disclosure.

[0127] like Figure 4 As shown, frequency domain branch distillation is performed on student features to transfer knowledge related to the overall outline and texture frequencies of the object from teacher features. This is achieved through the following steps:

[0128] Step S401: Perform frequency domain transformation on the features representing the overall outline, structure and texture of the object in the student features to obtain the frequency domain coefficient matrix.

[0129] Student features refer to the feature representations output by the student model after training samples of equipment and items are input into it. These features contain information related to the overall outline, structure, and texture of the items and are the direct processing objects of frequency domain branch distillation.

[0130] Frequency domain transformation refers to the process of mapping student features from the spatial domain to the frequency domain. In this scheme, the frequency domain transformation is preferably implemented through two-dimensional discrete cosine transform (DCT) to separate different frequency components from spatial features.

[0131] The frequency domain coefficient matrix refers to the spectral representation of student features obtained after frequency domain transformation, used to characterize the response at different frequency positions. The frequency domain coefficient matrix is ​​the foundation for subsequent low-frequency / high-frequency partitioning and frequency response extraction.

[0132] Thus, through step S401, the student features originally located in the spatial domain can be mapped to the frequency domain, thereby allowing different frequency components in the student features to be explicitly expressed. This provides the conditions for subsequent separate processing of low-frequency and high-frequency information, which is beneficial for the student model to further learn frequency domain knowledge related to the overall outline, structure, and texture of the object.

[0133] Step S402: Extract the low-frequency coefficients corresponding to the low-frequency positions and the high-frequency coefficients corresponding to the high-frequency positions from the frequency domain coefficient matrix according to the preset frequency positions.

[0134] Preset frequency positions refer to multiple key frequency positions pre-selected in the frequency domain coefficient matrix, which are used to extract frequency information of a specific frequency band from the frequency domain coefficient matrix.

[0135] Low-frequency coefficients refer to the coefficients located in the low-frequency band of the frequency domain coefficient matrix, used to characterize the overall outline and structural information of an object. High-frequency coefficients refer to the coefficients located in the high-frequency band of the frequency domain coefficient matrix, used to characterize the detailed texture information of an object.

[0136] Step S402 allows for further functional differentiation of different frequency components in the frequency domain coefficients, explicitly separating low-frequency information representing the overall contour and structure from high-frequency information representing detailed texture. This avoids mixing of different frequency components and facilitates differentiated response extraction and knowledge transfer for different frequency knowledge in subsequent operations.

[0137] Step S403: Extract sample frequency response based on low-frequency and high-frequency coefficients to transfer frequency domain knowledge from teacher characteristics.

[0138] The frequency response of an object refers to information that reflects the intensity of a student's response at a corresponding frequency location, calculated based on low-frequency and high-frequency coefficients.

[0139] Contour response refers to the response results of each channel at low-frequency positions, used to characterize the overall contour and structural information of an object. Texture response refers to the response results of each channel at high-frequency positions, used to characterize the detailed texture information of an object.

[0140] Frequency domain knowledge refers to the knowledge related to the overall outline, structure, and texture frequencies of an object contained in the teacher's features. The goal of frequency domain bifurcation distillation is to enable student models to learn this knowledge.

[0141] Step S403 further transforms the low-frequency and high-frequency coefficients into frequency response representations usable for distillation constraints. This allows the student model to move beyond frequency component decomposition and learn frequency domain knowledge from the teacher model, focusing on contour and texture responses. This enhances the student model's ability to represent the overall contour, structure, and texture frequency information of an object and provides a foundation for subsequent calculations of frequency domain distillation loss.

[0142] Further, in step S401, obtaining the frequency domain coefficient matrix includes:

[0143] The student features are pooled to obtain a pooled feature map with a preset spatial size;

[0144] Perform DCT on the pooled feature map to obtain the frequency domain coefficient matrix.

[0145] Here, the student features are first pooled to obtain a pooled feature map with a preset spatial size. Specifically, pooling can be performed on the feature map output by the student model to uniformly map student features at different scales to a pooled feature map with a preset spatial size, thus facilitating subsequent frequency domain analysis at a uniform scale. This method reduces the impact of differences in the size of the original feature maps on the frequency domain transformation results.

[0146] In some alternative implementations, the pooling process can be adaptive average pooling. For example, adaptive average pooling can be used to unify student features to a 7×7 size, reducing computational complexity.

[0147] Then, a Directional Transformation (DCT) is performed on the pooled feature map to obtain the frequency domain coefficient matrix. Specifically, DCT can be performed on each channel of the pooled feature map to transform it from the spatial domain to the frequency domain, thereby obtaining the corresponding frequency domain coefficient matrix. Different positions in the frequency domain coefficient matrix correspond to different frequency components, used to characterize the response of the student features in different frequency bands. Among them, low-frequency components are used to characterize the overall outline and structural information of the object, while high-frequency components are used to characterize the detailed texture information of the object.

[0148] The above method allows for the explicit expression of overall contour information and detailed texture information, which were originally mixed in spatial domain features, in the frequency domain, thus providing a basis for the subsequent extraction of low-frequency and high-frequency information respectively.

[0149] In some alternative implementations, DCT can be performed separately on each channel of the student features to obtain the frequency domain coefficient representation for each channel. DCT is used to transform the student features from the spatial domain to the frequency domain to separate low-frequency contours and high-frequency textures. The low-frequency components reflect the overall structure and shape of the object and are more robust to occlusion and lighting changes; the high-frequency components reflect detailed textures and help distinguish similar ingredients.

[0150] Specifically, the process of obtaining the frequency domain coefficient matrix described above can be achieved in the following way:

[0151] Let the pooling feature map corresponding to the student features be... ,in The number of feature channels, For feature map height, Given the feature map width, the frequency domain representation is obtained through DCT transformation. The frequency domain coefficient matrix mentioned above:

[0152] (4)

[0153] in This represents the two-dimensional discrete cosine transform, using an orthogonal normalization method. The specific calculation formula is as follows:

[0154] (5)

[0155] in The channel index represents the current channel being calculated. Represents the frequency index in the vertical direction. Represents the horizontal frequency index; Represents spatial coordinates, indicating the pixel position in the feature map; Pooled feature maps corresponding to student features in the channel ,Location The value at; For the transformed frequency domain features in the channel ,frequency The value at that location.

[0156] In some embodiments, the first step may be to Pooling processing is performed to obtain pooled feature maps. And then Performing DCT to obtain frequency domain features This is the frequency domain coefficient matrix mentioned above.

[0157] Furthermore, in step S402, after completing the DCT frequency decomposition, in order to extract more valuable frequency information for subsequent distillation from the frequency domain coefficient matrix, multi-spectral feature extraction can be further performed.

[0158] Specifically, multiple key frequency positions can be pre-selected, covering at least the low-frequency and high-frequency bands. Based on the coordinates of these key frequency positions in the frequency domain coefficient matrix, the corresponding frequency domain coefficients are read from the matrix. The frequency domain coefficients corresponding to the key frequency positions located in the low-frequency band are taken as low-frequency coefficients, and the frequency domain coefficients corresponding to the key frequency positions located in the high-frequency band are taken as high-frequency coefficients. The preset frequency positions consist of multiple pre-selected key frequency positions, covering at least the low-frequency and high-frequency bands. Key frequency positions located in the low-frequency band correspond to low-frequency coefficients representing the overall outline and structure of the object, while key frequency positions located in the high-frequency band correspond to high-frequency coefficients representing the detailed texture of the object.

[0159] In this embodiment, the top16 method is used to pre-select 16 key frequency positions, which cover key frequency bands from low frequency to high frequency. The frequency selection method is based on FCANet, and a frequency position index table based on 7×7 feature map is given, as shown in Table 1.

[0160] Table 1. Index of Top 16 Frequency Positions

[0161]

[0162] Table 1 provides 16 preset frequency positions. These 16 frequency positions are divided into at least two sets: the low-frequency position set represents the overall outline and structure of the object, and the high-frequency position set represents the detailed texture of the object.

[0163] For example, the set of high-frequency locations in Table 1 (By serial number): 3, 4, 8, 9, 10, 12, 13, 15; Low-frequency location set in Table 1 (In order of number): 1, 2, 5, 6, 7.

[0164] In some implementations, frequency positions belonging to low and mid-low frequencies in the key frequency positions can be classified as low frequency positions, frequency positions belonging to high and mid-high frequencies can be classified as high frequency positions, and mid-frequency positions can be classified as low frequency positions or high frequency positions as needed, or treated as transitional frequency bands alone.

[0165] Read the corresponding values ​​from the frequency domain coefficient matrix according to the coordinates of the preset frequency position. This is used to obtain the frequency domain coefficient matrix of a specific channel. Then, for each position in the top16 index table Read the value at that position directly. ;

[0166] like Belongs to the low-frequency location set If so, this value is used as the low-frequency coefficient;

[0167] like Belongs to the high-frequency location set If so, then this value is used as a high-frequency coefficient.

[0168] Using the above method, based on the key frequency positions shown in Table 1, low-frequency coefficients representing the overall outline and structure of the object and high-frequency coefficients representing the detailed texture of the object can be extracted from the frequency domain coefficient matrix, thus providing a basis for subsequent extraction of the object's frequency response.

[0169] In step S403, the sample frequency response is extracted based on the low-frequency coefficient and the high-frequency coefficient, including:

[0170] Select the discrete cosine basis functions corresponding to the low-frequency coefficients and the high-frequency coefficients;

[0171] Multiply each channel feature map by its corresponding discrete cosine basis function and sum them in the spatial dimension;

[0172] Based on the summation results, the contour response of each channel at low frequency positions and the texture response at high frequency positions are obtained.

[0173] Specifically, for each preset frequency position, the pooled student feature map can be projected using the discrete cosine basis function corresponding to that frequency position to obtain the response intensity of each channel at that frequency position. The response intensity corresponding to a frequency position in the low-frequency band is used as the contour response, and the response intensity corresponding to a frequency position in the high-frequency band is used as the texture response. In this way, information related to the overall contour and structure of the object, as well as information related to the detailed texture of the object, can be represented separately in the student features, thus providing a foundation for subsequent frequency domain knowledge transfer.

[0174] For example, the process of extracting the frequency response of the above-mentioned items can be implemented in the following way:

[0175] Select the discrete cosine basis functions corresponding to the low-frequency and high-frequency coefficients. For any preset frequency position (u, v), a corresponding two-dimensional discrete cosine basis function can be constructed. .

[0176] Then, the feature maps of each channel of the pooled student feature map are multiplied element-wise by the corresponding discrete cosine basis functions, and summed in the spatial dimension to obtain the frequency response intensity of each channel at the corresponding frequency position. The frequency response intensity of the c-th channel at the frequency position (u,v) can be expressed as:

[0177] (6)

[0178] in, is the frequency response value of the c-th channel, which measures the similarity between the characteristics of the c-th channel and the frequency pattern. It is a scalar, and the larger the value, the stronger the response of the channel at this frequency. For the predefined frequency position corresponding to the c-th channel, The pooled feature value is the feature value at position (x,y) in channel c; Indicates frequency position The corresponding discrete cosine basis functions are located in space. The value at that location.

[0179] Finally, the frequency response intensity corresponding to the frequency position in the low frequency band is taken as the contour response, and the frequency response intensity corresponding to the frequency position in the high frequency band is taken as the texture response, thus obtaining the object's frequency response.

[0180] Among them, based on low-frequency location set For all low-frequency locations The responses at low-frequency locations are calculated using the corresponding DCT basis functions. These response values ​​collectively characterize the response of the c-th channel in low-frequency mode, providing information related to the overall contour and structure of the object. Therefore, these responses at low-frequency locations are collectively referred to as the contour response.

[0181] Based on high-frequency location set For all high-frequency positions The responses at high-frequency locations are calculated using the corresponding DCT basis functions. These response values ​​collectively characterize the response of the c-th channel in high-frequency modes, providing information related to the detail texture of the object. Therefore, these responses at high-frequency locations are collectively referred to as texture responses.

[0182] Furthermore, after obtaining the pooled student feature map... After determining the low-frequency coefficients corresponding to low-frequency positions, the high-frequency coefficients corresponding to high-frequency positions, and the preset frequency position coordinates corresponding to these frequency positions, for each preset frequency position coordinate... Each of these has a corresponding two-dimensional DCT basis function, which are essentially detectors of different frequency modes.

[0183] Here, for any preset frequency position Constructing DCT basis functions Used to detect specific frequency components:

[0184] (7)

[0185] In some embodiments, the above formula can also be expressed by the following formula:

[0186] (8)

[0187] in, Frequency position The corresponding DTC basis functions are located in space. The value at; Spatial location coordinates, represents the orthogonal normalization coefficients of DCT; H and W represent the feature map sizes after pooling.

[0188] Using the above methods, we can extract the frequency response related to the overall outline and structure of the object as well as the frequency response related to the detailed texture of the object, providing a foundation for subsequent frequency domain knowledge transfer.

[0189] Furthermore, after extracting the sample frequency response based on the low-frequency and high-frequency coefficients, the following is also included:

[0190] The contour response and texture response of each channel are combined to form a multi-spectral feature vector;

[0191] The multi-spectral feature vector is input into the channel attention module to generate the attention weights for each channel.

[0192] Student features are obtained by weighting them using attention weights.

[0193] Here, the frequency response intensity obtained in the above embodiments will be... Arranging them in channel order yields a multi-spectral feature vector, where each element represents the response intensity of the corresponding channel at a specific frequency position.

[0194] After obtaining the multi-spectral feature vector, it can be input into the channel attention module to generate attention weights for each channel. In some embodiments, the channel attention module can be implemented using a Squeeze-and-Excitation (SE) module. Specifically, the multi-spectral feature vector can be input into a fully connected layer, where dimensionality reduction is performed first, followed by dimensionality increase, and a non-linear activation function is introduced between the dimensionality reduction and increase processes to obtain the channel attention weight vector. This process can be represented as:

[0195] (9)

[0196] in , For dimensionality reduction and dimensionality increase matrices, the former reduces the dimension from... Compress to This reduces computational cost; the latter reduces the dimension from Restore ; To reduce the dimensionality ratio; The Sigmoid function is used; ReLU is the modified linear unit activation function. For the channel attention weight vector, each element Indicates the first The importance of each channel. Through dimensionality reduction and expansion, the model can learn the complex dependencies between channels. If certain channels... The generally large values ​​indicate that these frequencies are important for identification, and the SE module will assign them higher weights. .

[0197] After generating the channel attention weights, the student features can be weighted using these weights to obtain the student frequency domain features. Specifically, the channel attention weights can first be expanded to the same spatial dimension as the student feature map, and then multiplied element-wise with the original student features to obtain the weighted student frequency domain features. This process can be represented as:

[0198] (10)

[0199] in yes Weights that are expanded to the same size as the feature map These are the original student characteristics.

[0200] In this way, effective frequency channels related to the overall outline and structure of the object, as well as the detailed texture of the object, can be strengthened, while unimportant channels are suppressed, so that the student model can focus more on the frequency domain information that is beneficial to the recognition task.

[0201] In this way, on the one hand, the response information of each channel at low and high frequencies can be organized into multi-spectral feature vectors; on the other hand, the channel attention module can adaptively generate the attention weights corresponding to each channel based on the multi-spectral feature vectors, thereby differentiating and strengthening the student features to obtain the student frequency domain features. These student frequency domain features can then be further used for comparison with teacher features and for calculating frequency domain distillation loss.

[0202] In some embodiments, after obtaining the student frequency domain features, the teacher features can be channel aligned to obtain aligned teacher features; the aligned teacher features are compared with the student frequency domain features, and the frequency domain distillation loss is calculated based on the comparison result; the student model is constrained based on the frequency domain distillation loss so that the student model learns the representation ability of the teacher model related to the overall outline, structure and texture frequency of the object.

[0203] Specifically, channel alignment can be performed on the teacher features first. Since the network structures of the teacher and student models are different, their output features may differ in the number of channels. Therefore, convolutional operations can be used to map the teacher features to channels, so that the teacher features correspond to the student frequency domain features in the channel dimension. Specifically, the teacher features are aligned using 1×1 convolutions to preserve the original semantic structure and match the student-side feature dimensions.

[0204] After channel alignment is completed, the aligned teacher features are compared with the student frequency domain features, and the frequency domain distillation loss is calculated based on the comparison result. In some embodiments, mean square error can be used as the frequency domain distillation loss. Let the aligned teacher features be denoted as... The student's frequency domain characteristics are denoted as The frequency domain distillation loss It can be represented as:

[0205] (11)

[0206] in, This represents the frequency domain distillation loss, and N represents the batch size. Record the index of the i-th sample in the batch. This represents the weighted frequency domain features of the student in the i-th sample. Let represent the teacher alignment feature of the i-th sample.

[0207] Here, the frequency domain distillation loss reflects the degree of difference between the student's frequency domain features and the teacher's aligned features. The smaller the difference, the more fully the student model has learned the teacher model's frequency domain knowledge.

[0208] Furthermore, the student model can be constrained based on frequency domain distillation loss. Specifically, during training, the frequency domain distillation loss can be used as part of the optimization objective, updating the student model parameters through backpropagation. This allows the student model to learn the representations related to the overall outline, structure, and texture frequencies of the object from the teacher model while maintaining its lightweight nature.

[0209] Through the aforementioned frequency domain branch distillation process, a frequency domain branch distillation loss characterizing the difference between student and teacher frequency domain features can be obtained. Furthermore, since different distillation weights have been assigned to each category based on the sample frequency of different item categories in the aforementioned implementation, the frequency domain branch distillation loss can be weighted using these corresponding distillation weights, thereby achieving differentiated frequency domain distillation constraints for samples of different categories. In this way, based on the student model learning the teacher model's frequency domain knowledge, the constraint strength for categories with fewer samples can be further enhanced, mitigating recognition bias caused by uneven category distribution.

[0210] Furthermore, the aforementioned frequency domain branch distillation is primarily used to enhance the student model's ability to learn information related to the overall outline, structure, and texture frequencies of an object. To further enhance the student model's ability to represent local details, spatial domain context, and semantic information of an object, in some embodiments, spatial branch distillation can be combined with further training of the student model. The specific implementation process of spatial branch distillation is described below.

[0211] Figure 5 This is a schematic diagram of the training process of spatial branch distillation of the student model in an embodiment of this disclosure.

[0212] like Figure 5 As shown, spatial branch distillation is performed on student features to transfer knowledge related to local details of objects and spatial domain semantics from teacher features, including:

[0213] Step S501: Perform multi-scale feature adaptation on the student features to obtain multi-scale student features that represent the local texture details and spatial domain context information of the object.

[0214] Multi-scale feature adaptation refers to the process of performing multi-scale convolution on student features to simultaneously capture local details, medium-scale patterns, and large-scale contextual information. Multi-scale student features are student-side feature representations obtained after multi-scale feature adaptation, which can simultaneously characterize local texture details and spatial domain contextual information of an object.

[0215] Here, parallel feature extraction can be performed on student features using convolution operations with different kernel sizes to obtain local details, medium-range patterns, and large-scale contextual information respectively. Then, the multi-channel features obtained in parallel extraction are concatenated. Finally, a 1×1 convolution is performed on the concatenated features to obtain multi-scale student features. In this way, student features can simultaneously possess local details and spatial domain contextual information, thereby enhancing the richness of student features.

[0216] Step S502: Apply a random channel mask to the multi-scale student features to obtain the masked features.

[0217] Random channel masking refers to a method of randomly setting some channels of multi-scale student features to zero in order to create information loss and simulate occlusion, noise, or incomplete features.

[0218] Masked features refer to the feature representation obtained after applying a random channel mask to multi-scale student features. These features are incomplete student features and are used to recover information from subsequent inputs into the generator network.

[0219] Here, a random channel mask matrix can be generated first, and some elements can be set to zero and the rest to one according to a preset mask ratio. Then, the random channel mask matrix is ​​multiplied element-wise with the multi-scale student features to obtain the mask features. In this way, feature loss can be randomly generated to simulate complex situations such as object occlusion, noise interference, or incomplete information.

[0220] Step S503: The mask features are recovered, and spatial distillation is performed based on the recovered features and the teacher features to enhance the student model's learning of local details of objects and spatial domain semantic features.

[0221] Spatial distillation refers to the process of distilling constraints on the student model based on a comparison of the recovered student spatial features and the teacher spatial features.

[0222] Here, the masked features are first input into the generator network, which outputs the recovered generated features, which are then used as the student's spatial features. Next, channel alignment is performed on the teacher's features to obtain the teacher's spatial features. Then, the teacher's spatial features are compared with the student's spatial features, and the spatial distillation loss is calculated based on the comparison result. Finally, the student model is constrained based on this spatial distillation loss. In this way, the student model can learn to recover effective spatial domain information from incomplete features, allowing it to more fully inherit the knowledge representation of the teacher model in the spatial domain.

[0223] By using the above methods, the student model's ability to represent local details and spatial semantics can be enhanced, and the stability and accuracy of subsequent object recognition can be improved.

[0224] In some alternative implementations, multi-scale feature adaptation may include multiple convolutional branches with different receptive fields, where smaller convolutional kernels are used to extract local texture details and larger convolutional kernels are used to extract contextual information over a wider range.

[0225] In some alternative implementations, the mask ratio of the random channel mask can be preset to control the proportion of features set to zero.

[0226] In some alternative implementations, the generator network can be a lightweight recovery network to reduce the additional computational overhead in the spatial branching distillation process.

[0227] In some alternative implementations, channel alignment of teacher features can be achieved through 1×1 convolution to make teacher features correspond to student spatial features in the channel dimension.

[0228] In some alternative implementations, spatial distillation loss can be used to measure the difference between the recovered student spatial features and the teacher spatial features, and together with other distillation losses, constitute the loss function during the student model training process.

[0229] Through the aforementioned spatial branching distillation process, on the one hand, multi-scale feature adaptation can enhance the student model's ability to represent local texture details and spatial domain contextual information of objects; on the other hand, random channel masking and generator recovery mechanisms can improve the robustness of the student model in the case of incomplete features; combined with teacher feature alignment and spatial distillation loss constraints, the student model can learn more fully the spatial domain semantic knowledge in the teacher model, thereby improving the stability and accuracy of subsequent object recognition.

[0230] Further, in step S501, multi-scale feature adaptation is performed on the student features to obtain multi-scale student features representing the local texture details and spatial domain context information of the object, including:

[0231] Convolutional operations with different kernel sizes are used to extract student features in parallel, in order to extract local texture details, medium-range patterns and spatial domain context information of the samples.

[0232] Channel splicing is performed on the multi-path features extracted in parallel.

[0233] Convolutional fusion is performed on the concatenated features to obtain multi-scale student features.

[0234] Specifically, multiple parallel convolutional branches can be set up, with a 3×3 convolutional branch for extracting local texture details, a 5×5 convolutional branch for extracting medium-range patterns, and a 7×7 convolutional branch for extracting large-scale contextual information. Through parallel convolutional processing with different receptive fields, student features can simultaneously contain fine-grained local information and large-scale contextual information.

[0235] Then, the feature maps output by each convolutional branch are concatenated along the channel dimension to form a pre-fusion feature representation containing information at different scales. In this way, local details, intermediate-range patterns, and large-scale contextual information extracted from different receptive fields can be preserved in the same feature representation.

[0236] Next, a 1×1 convolution is applied to the concatenated features to achieve cross-channel information fusion, and the information extracted from the multi-path parallel branches is integrated into a unified multi-scale student feature representation. In this way, the output dimension can be controlled while maintaining feature richness, resulting in multi-scale student features that represent both local texture details of the object and spatial domain context information.

[0237] For example, the outputs of each branch are concatenated along the channel dimension and then fused using a 1×1 convolution to obtain multi-scale student features. :

[0238] (12)

[0239] in Indicates the kernel size as Convolution operation, Indicates concatenation of channel dimensions; This represents 1×1 convolution fusion.

[0240] By employing the above methods, student features can simultaneously possess local details, medium-range patterns, and large-scale contextual information, enhancing the richness and robustness of student features. This provides a more sufficient feature foundation for subsequent random channel masking, generator recovery, and spatial distillation.

[0241] Further, in step S502, a random channel mask is applied to the multi-scale student features to obtain masked features, including:

[0242] Generate a random channel mask matrix;

[0243] According to the preset mask ratio, some elements in the random channel mask matrix are set to zero, and the rest are set to one.

[0244] The mask features are obtained by multiplying the random channel mask matrix element-wise with the multi-scale student features.

[0245] First, a mask matrix of the same size as the multi-scale student features is generated, ensuring that the mask matrix corresponds element-wise to each of the multi-scale student features. A random channel mask is used to randomly occlude some channel information to simulate incomplete features.

[0246] Then, a corresponding random number is generated for each element in the mask matrix; if the random number is less than the preset mask ratio, the element is set to 0, otherwise it is set to 1. Here, the preset mask ratio can be set to 0.15, that is, about 15% of the channel positions are randomly set to zero.

[0247] Next, the student features corresponding to positions with values ​​of 0 in the mask matrix are set to zero, while the original features remain unchanged at positions with values ​​of 1, thus forming mask features containing randomly missing information. This is achieved through element-wise multiplication, randomly creating missing information to simulate situations where features are incomplete due to occlusion, noise, etc., in real-world applications.

[0248] For example, the above masking process is implemented as follows:

[0249] Generate random channel mask matrix By randomly occluding some channel information, the student model is forced to recover useful features even with incomplete information, thereby learning more robust and redundant feature representations. Preset mask ratio. Set to 0.15, mask matrix Each element in The filling rules are as follows:

[0250] (13)

[0251] When the random number is less than At that time, element Set to 0, otherwise set to 1.

[0252] Applying masks to multi-scale student features Obtain the mask features :

[0253] (14)

[0254] in This represents an element-wise multiplication operation, where the mask is multiplied element-wise with the student's features. This randomly creates missing information, simulating incomplete features in real-world applications, and forces the model to learn the ability to recover complete features from incomplete information.

[0255] In this way, on the one hand, the student model can be exposed to incomplete input forms during the training process; on the other hand, it can force the student model to learn to recover effective information from incomplete features, thereby improving the robustness and redundancy of the student model under the conditions of object occlusion, noise interference or local information loss.

[0256] Further, in step S503, the mask features are recovered, including:

[0257] Input the mask features into the generator network, and output the restored generated features;

[0258] The restored generated features are used as student spatial features.

[0259] The above methods enable student models to learn the mapping relationship from incomplete features to complete features during training, thereby enhancing the feature recovery ability of student models under conditions of missing information, occlusion, or noise.

[0260] In some alternative implementations, the generator network can be a lightweight recovery network to reduce the additional computational overhead introduced during spatial branching distillation.

[0261] In some embodiments, the generator network may consist of two 3×3 convolutional layers, where the first 3×3 convolutional layer is used for preliminary recovery of the mask features, and the second 3×3 convolutional layer is used to generate the final recovered features. Specifically, the first 3×3 convolutional layer is first used to perform a convolutional transformation on the mask features, then batch normalization is performed on the convolutional result, and nonlinear activation is performed by correcting linear units; subsequently, the second 3×3 convolutional layer is used to further convolutionally transform the activated features to output the recovered generated features.

[0262] In some alternative implementations, batch normalization and nonlinear activation may also be included between the first 3×3 convolutional layer and the second 3×3 convolutional layer to enhance the feature modeling capability of the generator network.

[0263] Since the restored generated features are reconstructed based on the incomplete features, they can compensate for the information loss caused by the random channel mask while preserving the original spatial domain features of the student model. Using these features as student spatial features is beneficial for subsequent alignment and comparison with teacher features, and for calculating spatial distillation loss.

[0264] For example, the above recovery process is implemented in the following way:

[0265] Train a lightweight generator network to recover complete features similar to the teacher's features from masked, incomplete features, thereby enhancing the student's feature learning ability. The student-generated features are then recovered through the generator network. :

[0266] (15)

[0267] in The generator consists of two 3×3 convolutional layers, using the input mask features. ( This is the first 3×3 convolutional layer. ( This is a batch normalization operation. ( () is the modified linear unit activation function. ( ) is the second 3×3 convolutional layer.

[0268] In this way, through a simple two-layer convolutional network, students can learn the mapping from incomplete features to complete features, forcing the features extracted by students to have sufficient redundancy and robustness, so that useful representations can be reconstructed even if some information is missing.

[0269] Through the generator recovery process described above, the student model can still learn to recover effective spatial domain features from incomplete features even when information is lost due to random channel masks. This enhances the student model's adaptability to local information loss, occlusion, and noise interference. Furthermore, it provides more complete and stable student-side spatial features for subsequent spatial distillation with teacher features, thereby strengthening the student model's learning of local details and spatial domain semantic features of objects.

[0270] Further, in step S503, spatial distillation is performed based on the recovered features and teacher features, including:

[0271] Channel alignment is performed on teacher features to obtain teacher spatial features while preserving the original semantic structure of the teacher features.

[0272] Compare the spatial characteristics of teachers with those of students, and calculate the spatial distillation loss based on the comparison results;

[0273] The student model is constrained by spatial distillation loss so that it can learn the representations of the teacher model related to local details of objects and semantics of the spatial domain.

[0274] Here, because the network structures of the teacher model and the student model are different, their output features may differ in the number of channels. Therefore, channel alignment processing can be performed on the teacher features to make them correspond to the recovered student-side features in terms of channel dimension. In some embodiments, channel alignment processing can be implemented using 1×1 convolution. In this way, channel dimension matching between teacher features and student-side features can be achieved while preserving the original semantic structure of the teacher features as much as possible.

[0275] For example, channel alignment of teacher features is performed to obtain aligned teacher spatial features in the following manner. :

[0276] (16)

[0277] in The original number of channels for teacher characteristics. This represents the number of output channels for the multi-scale adapter.

[0278] The recovered generated features are used as student spatial features, and the teacher spatial features and student spatial features are input into the loss calculation module to measure the degree of difference between them. Since the student spatial features are recovered from the mask features by the generator, this comparison process is essentially an alignment comparison between the "incomplete feature recovery result" and the teacher-side spatial domain features.

[0279] In some embodiments, mean squared error loss can be used as spatial distillation loss to measure the difference between teacher spatial characteristics and student spatial characteristics.

[0280] For example, space distillation loss is calculated as follows: :

[0281] (17)

[0282] in Let be the restored student spatial features for the i-th sample. Let N be the aligned teacher spatial features of the i-th sample, and N represent the batch size. Record the index of the i-th sample in the batch.

[0283] This spatial distillation loss forces the student to recover features similar to the teacher's features. Even if the student only sees partial information, it can reconstruct a complete feature representation similar to the teacher's, improving the robustness and discriminativeness of the student's features. This spatial distillation loss can be used as one of the optimization objectives during the student model training process, and the student model parameters can be updated through backpropagation, enabling the student model to learn the representations related to local details and spatial semantics of objects from the teacher model. In this way, the student model's ability to express local texture details, contextual relationships, and spatial semantic features can be enhanced.

[0284] In some alternative implementations, when constraining the student model based on spatial distillation loss, the student model can be optimized in conjunction with classification loss to balance the distillation learning effect and the performance of the recognition task.

[0285] Through the aforementioned spatial distillation process, on the one hand, the recovered student spatial features and teacher features can be compared and contrasted in both channel and spatial dimensions; on the other hand, the spatial distillation loss can impose constraints on the student model, enabling it to learn the knowledge representations related to local details and spatial semantics of objects from the teacher model. This further enhances the student model's spatial domain representation ability under conditions of occlusion, noise interference, or missing local information, thereby improving the stability and accuracy of subsequent object recognition.

[0286] After obtaining the frequency domain branch distillation loss and spatial branch distillation loss after dynamic weight adjustment, the following describes how to construct the total training loss to train the student model, using specific examples.

[0287] Figure 6 This is a schematic diagram illustrating the process of constructing the total training loss to train the student model in an embodiment of this disclosure.

[0288] like Figure 6 As shown, the student model is trained based on the frequency domain branch distillation loss and spatial branch distillation loss after dynamic weight adjustment, including:

[0289] Step S601: Obtain the joint distillation classification loss, the frequency domain branch distillation loss after dynamic weight adjustment, and the spatial branch distillation loss;

[0290] Step S602: The distillation classification joint loss, the frequency domain branch distillation loss after dynamic weight adjustment, and the spatial branch distillation loss are weighted and fused to obtain the total training loss, and the student model is trained based on the total training loss.

[0291] We set corresponding weight coefficients for the distillation classification joint loss, the frequency domain branch distillation loss after dynamic weight adjustment, and the spatial branch distillation loss, and then summed the above loss terms by weight to obtain the total training loss.

[0292] Training the student model based on the total training loss allows backpropagation to be performed in each training batch based on the total training loss, and the student model parameters to be updated. This enables the student model to learn the knowledge of the teacher model in the output layer, frequency domain branch, and spatial branch at the same time, while also taking into account the standard classification task.

[0293] Further, in step S601, the acquisition of the joint loss of distillation and classification includes:

[0294] Obtain the output layer distillation loss and the student model's cross-entropy loss on the training samples.

[0295] The output layer distillation loss and cross-entropy loss are weighted and fused to obtain the joint distillation classification loss.

[0296] Here, the training samples in the current training batch can first be input into the teacher model and the student model respectively to obtain the output results of the teacher model and the student model; then, the output layer distillation loss can be calculated based on the output results of the teacher model and the student model. At the same time, the category prediction results output by the student model can be compared with the true category labels corresponding to the training samples to calculate the cross-entropy loss of the student model on the training samples.

[0297] This method allows for the simultaneous acquisition of distilled supervision information reflecting the knowledge transfer of the teacher's model output layer, as well as real-label supervision information reflecting the student's model classification ability.

[0298] Furthermore, corresponding weight coefficients can be set for the output layer distillation loss and cross-entropy loss respectively, and the output layer distillation loss and cross-entropy loss can be weighted and summed to obtain the joint loss of distillation and classification.

[0299] This approach allows student models to learn the soft label knowledge from the teacher model's output layer while maintaining their ability to classify and learn real category labels.

[0300] In some alternative implementations, the output layer distillation loss is obtained by including:

[0301] Obtain the output results of the teacher model and the student model;

[0302] Temperature scaling is applied to the outputs of the teacher model and the student model.

[0303] The output layer distillation loss is calculated based on the temperature-scaled output results of the teacher model and the student model.

[0304] Furthermore, the output layer distillation loss can be obtained in the following way:

[0305] First, in each training batch, the same batch of training samples is input into the teacher model and the student model respectively, and the output results of the teacher model and the student model are obtained. The output results can be the logits output by the teacher model and the logits output by the student model.

[0306] Then, temperature scaling is applied to the outputs of the teacher model and the student model. Specifically, the logits of the teacher model and the student model can be divided by the temperature parameter T to soften the corresponding output probability distributions. The temperature parameter T can be set to 4.0.

[0307] Next, the outputs of the teacher and student models after temperature scaling are subjected to probability normalization. For example, the scaled teacher logits and student logits can be transformed into corresponding probability distributions using the softmax function, where the probability distribution output by the teacher model is used as the soft label distribution, and the probability distribution output by the student model is used as the distribution to be fitted.

[0308] Finally, the output layer distillation loss is calculated based on the temperature-scaled outputs of the teacher and student models. In some embodiments, KL divergence can be used to measure the difference between the output probability distributions of the teacher and student models, thereby obtaining the output layer distillation loss. The output layer distillation loss can be expressed as:

[0309] (18)

[0310] Where T is the temperature parameter, T=4.0, KL( ) is the KL divergence function, softmax( ) represents the softmax normalization function, z t Z represents the logits output by the teacher model. s This represents the logits output by the student model.

[0311] In this way, the student model can learn the soft label knowledge provided by the teacher model at the output layer, thereby enhancing the student model's ability to express the relative relationships between categories while retaining the standard classification learning objectives.

[0312] In some alternative implementations, the cross-entropy loss can be calculated by the difference between the class prediction output by the student model and the true class labels of the training samples.

[0313] For example, the cross-entropy loss can be obtained as follows:

[0314] First, obtain the true class labels corresponding to each training sample in the current training batch. Specifically, the true class labels can be pre-labeled by the training set and correspond one-to-one with each training sample in the current training batch.

[0315] Next, the cross-entropy loss is calculated based on the category prediction results output by the student model and the true category labels.

[0316] In some embodiments, if the first i The true class label of each sample is denoted as . y i The student model's predicted probability for category k is denoted as... p i (k) Then the cross-entropy loss L ce It can be represented as:

[0317] (19)

[0318] Among them, L ce This represents the cross-entropy loss, where N represents the number of samples in the current training batch. Indicates that the i-th sample belongs to its true class. y i The predicted probability.

[0319] In the above manner, the standard classification loss of the student model on the training samples can be obtained as the cross-entropy loss, which is used to measure the difference between the student model's prediction results and the true class labels, and serves as a component of the subsequent construction of the distillation classification joint loss.

[0320] Further, in step S601, the combined loss of distillation and classification L KD It can be obtained in the following ways:

[0321] (20)

[0322] in, is a weighting coefficient used to balance the relative contributions of the output layer distillation loss and cross-entropy loss to the joint loss of distillation classification.

[0323] In some embodiments, It can be pre-configured according to the actual dataset and training requirements.

[0324] In this way, the output layer distillation loss and cross-entropy loss are fused to form a distillation classification joint loss that simultaneously incorporates the knowledge from the teacher model's output layer and the supervision information from the real labels.

[0325] Furthermore, in step S602, the total training loss It can be obtained in the following ways:

[0326] ;(twenty one)

[0327] in, For total training losses, For the combined loss of distillation classification, This refers to the frequency domain branch distillation loss after dynamic weight adjustment. For spatial branching distillation losses, These are the weighting coefficients for the distillation classification joint loss, the frequency domain branch distillation loss after dynamic weight adjustment, and the spatial branch distillation loss, respectively.

[0328] In some embodiments, .

[0329] In this way, the output layer distillation loss and cross-entropy loss can be fused into a distillation classification joint loss. Then, this joint loss can be fused together with the frequency domain branch distillation loss and spatial branch distillation loss after dynamic weight adjustment into a total distillation loss. This allows the student model to learn the knowledge of the teacher model in the output layer, frequency domain branch and spatial branch during the training process, while also taking into account the real label supervision information.

[0330] In some embodiments, training the student model based on the total training loss includes:

[0331] The gradient of the total training loss with respect to the student model parameters is calculated to obtain the gradient corresponding to each parameter of the student model.

[0332] The gradient is input into the optimizer to iteratively update the parameters of the student model based on the gradient;

[0333] During training, the teacher model parameters remain fixed, while the student model parameters are updated based on the total training loss.

[0334] Here, the gradient is used to characterize the trend of the total training loss relative to the student model parameters, indicating the direction and magnitude of the adjustment of the student model parameters. The optimizer is used to iteratively update the student model parameters based on the gradients corresponding to each parameter, in order to reduce the total training loss; the optimizer can be the AdamW optimizer, stochastic gradient descent optimizer, or other gradient optimization methods.

[0335] In this way, by taking the total training loss as a unified optimization objective and iteratively updating the parameters of the student model, the student model can learn output layer knowledge, frequency domain knowledge, and spatial domain knowledge simultaneously during the training process, while taking into account the supervision information of the real labels, thereby gradually improving the overall recognition performance of the student model.

[0336] By using the above methods, the student model can learn the knowledge of the teacher model in the output layer, frequency domain branch, and spatial branch during the training process, while taking into account the supervision information of the real label, thereby improving the overall recognition ability of the student model.

[0337] Compared with related technologies, this application has at least the following beneficial effects:

[0338] This application trains a student model by implementing differentiated distillation constraints based on the sample frequency of each item category. This allows for the application of constraints of varying strengths to the distillation training process to accommodate differences in the number of samples across different categories. This avoids the training bias problem caused by applying the same distillation intensity to all categories, thereby enhancing the student model's learning ability for categories with fewer samples and improving the overall accuracy of item recognition.

[0339] This application utilizes statistical analysis of the sample size for each item category in the training set, and then categorizes the item categories into long-tail levels based on the sample size, explicitly distinguishing between head, middle, and tail categories. This provides a basis for subsequently assigning different distillation weights based on different long-tail levels, thus giving the differentiated distillation constraints a clear categorical distribution foundation.

[0340] Furthermore, this application assigns different distillation weights to different item categories based on the long-tail ranking results, and makes the distillation weights for categories with fewer samples greater than those for categories with more samples. This allows tail categories to be more constrained during distillation training. Consequently, this enhances the learning ability of the student model to learn features related to tail categories, alleviates the class imbalance problem caused by long-tail distribution, and improves the recognition performance of categories with scarce samples.

[0341] Meanwhile, this application determines the corresponding long-tail level based on the category label of the current batch of samples during training and obtains the corresponding distillation weights, then uses the distillation weights to adjust the distillation loss. Therefore, dynamic and differentiated distillation enhancement can be implemented for samples of different categories during the training phase, enabling the student model to achieve a more balanced training effect under conditions of imbalanced sample distribution.

[0342] Furthermore, this application combines frequency domain branch distillation and spatial branch distillation during the distillation training process to transfer knowledge to the student model. Frequency domain branching helps the student model learn knowledge related to the overall contour, structure, and texture frequencies of the object, while spatial branching helps the student model learn knowledge related to the local details of the object and spatial semantics. Thus, while implementing differentiated distillation constraints, it also enhances the student model's comprehensive representation ability of object features, thereby improving the stability and accuracy of object recognition.

[0343] Finally, this application constructs a total training loss by fusing the output layer distillation loss, cross-entropy loss, distillation loss adjusted by differential weights, and other distillation losses. This allows the student model to simultaneously learn the output layer knowledge of the teacher model and the feature knowledge from different branches during training, while also taking into account real label supervision information. This is beneficial for further improving the overall recognition performance of the student model in scenarios with imbalanced class distribution.

[0344] By deploying the trained student model as an object recognition model, the recognition effect can be guaranteed while reducing the consumption of computing and storage resources when the model is deployed at the edge, thus meeting the application requirements of edge deployment and real-time recognition on the device.

[0345] In summary, this application improves the recognition performance of scarce categories under long-tail distribution conditions by implementing differentiated distillation constraints based on the sample frequency of different item categories and combining bi-branch knowledge transfer and multi-loss fusion training. This alleviates the problem of category imbalance and also takes into account the lightweight deployment requirements of the model, thereby enhancing the accuracy and practicality of intelligent item recognition in real-world application scenarios.

[0346] The following section uses specific usage examples to illustrate the intelligent item recognition method based on long-tail dynamic weight adjustment proposed in this application.

[0347] Figure 7 This is a flowchart illustrating the second intelligent item recognition method based on long-tail dynamic weight adjustment provided in this embodiment.

[0348] The technical solutions provided in this disclosure can be applied to smart object recognition scenarios. The device can be a smart device with image acquisition and edge recognition capabilities, and the object can be the object to be recognized in the corresponding application scenario.

[0349] For example, in some specific embodiments, the device can be a smart refrigerator, and the item can be food inside the refrigerator; correspondingly, the smart item recognition scenario can be a food inside the refrigerator recognition scenario. The following embodiments mainly describe the smart item recognition scenario of the device, and use the food inside the refrigerator recognition scenario as an example for illustration.

[0350] like Figure 7 As shown, the smart object recognition method includes:

[0351] Step S701: Collect image data of the item.

[0352] The device uses an image acquisition unit to collect image data of the object to be identified. This image acquisition unit can be an RGB camera or other image acquisition device. During data collection, different lighting conditions, different placement methods, and different object categories can be covered to obtain data distribution that more closely resembles real-world application scenarios.

[0353] For example, in the scenario of food recognition inside a refrigerator, a camera installed inside the refrigerator can be used to collect images of food in the refrigerator's storage area. The food images can cover situations such as when the door is open or closed, under different lighting conditions, with different placement densities, and with different food categories.

[0354] Step S702: Label the image data of the items and divide the dataset.

[0355] The collected object images are labeled to determine the object category label corresponding to each image; then, the labeled dataset is divided into training set, validation set and test set.

[0356] In some implementations, the labeled object images can be divided into a training set, a validation set, and a test set in an 8:1:1 ratio.

[0357] For example, in the scenario of recognizing food inside a refrigerator, each food image can be labeled with the corresponding food category, and divided into training set, validation set and test set according to the above proportion.

[0358] Step S703: Preprocess the image data of the item.

[0359] The acquired object images were uniformly adjusted to the preset input size, and the image pixel values ​​were normalized to ensure that each sample met the model input requirements. To enhance the model's generalization ability, data augmentation strategies such as random horizontal flipping, random rotation, and random occlusion were employed during training.

[0360] For example, in the scenario of recognizing food inside a refrigerator, the food image can be scaled to 224×224 and the above-mentioned normalization process can be performed as input for the subsequent teacher model and student model.

[0361] Step S704: Construct the teacher model and student model.

[0362] The teacher model is used to provide distillation guidance information, while the student model is used to form an item recognition model that can be deployed in edge devices after distillation training.

[0363] Here, a teacher model is constructed to provide distillation guidance information, and a student model is constructed to serve as the final deployment model.

[0364] Specifically, the teacher model can employ a deep recognition network pre-trained on a large-scale image dataset, with its classification head adjusted to a fully connected layer corresponding to the number of target item categories to adapt to the current item recognition task. During distillation training, the teacher model parameters can remain fixed and are used only for extracting teacher features and outputting the teacher model's classification results.

[0365] For example, in some specific implementations, the large-scale image dataset can be ImageNet, and the teacher model can be ResNet-101.

[0366] Furthermore, the student model can employ a lightweight recognition network suitable for device deployment, and its classification head can be adjusted to a fully connected layer corresponding to the number of target item categories. The student model parameters can be randomly initialized and updated during the distillation training process.

[0367] For example, in some implementations, the student model can be MobileNetV3-Small.

[0368] During model training, the input image can be simultaneously fed into both the teacher model and the student model to obtain their respective outputs, as well as the intermediate layer features of the teacher and student models. To facilitate subsequent frequency domain branch distillation and spatial branch distillation, the output of a preset layer from both the teacher and student models can be selected as the feature extraction layer.

[0369] Specifically, deeper output features can be selected from the teacher model as teacher features, and corresponding output features from the student model can be selected as student features. The selected teacher features and student features can be matched in spatial size to facilitate subsequent feature alignment and distillation loss calculation.

[0370] In some implementations, intermediate layer features can be obtained by capturing the output of specified layers in the teacher and student models through the registration of forward hooks.

[0371] For example, in some specific implementations, the 7×7 feature map output from a deeper layer of the teacher model can be selected as the teacher feature, and the 7×7 feature map output from the corresponding layer of the student model can be selected as the student feature.

[0372] The above methods can be used to construct teacher and student models suitable for smart object recognition scenarios, and provide a model and feature foundation for subsequent output layer distillation, frequency domain branch distillation, and spatial branch distillation.

[0373] For example, in the scenario of recognizing food inside a refrigerator, the target item category can be understood as the food category, and the teacher model and student model can be used for distillation training of the food recognition task inside a refrigerator.

[0374] Step S705: Initialize the training environment and training parameters.

[0375] Configure parameters such as batch size, optimizer, initial learning rate, weight decay, number of training epochs, temperature parameter, number of output channels for multi-scale feature adaptation, random channel mask ratio, and weight coefficients for each loss term.

[0376] For example, in some implementations, the AdamW optimizer can be used to update the student model parameters, with the initial learning rate set to 1×10⁻⁶. 3 The weight decay can be set to 1×10. 4 Furthermore, a cosine annealing learning rate scheduler can be used to dynamically adjust the learning rate, and the total number of training epochs can be set to 200. To improve training efficiency and reduce memory usage, automatic mixed-precision training can also be enabled during training.

[0377] In each training batch, the current batch samples can be input into the teacher model and student model respectively, and forward propagation is performed to obtain the output results of the teacher model, the output results of the student model, and the corresponding intermediate layer features. Then, based on the intermediate layer features and output results, the corresponding losses are calculated through frequency domain branch and spatial branch respectively, and the total training loss is constructed by combining the output layer distillation loss. Finally, backpropagation is performed based on the total training loss to update the student model parameters. During this process, the teacher model parameters can remain fixed and do not participate in training updates.

[0378] After each training round, the current student model can be evaluated using the validation set. Specifically, the classification accuracy of the current student model can be calculated based on the validation set, and the model checkpoint corresponding to the highest validation accuracy can be saved. The model checkpoint can include information such as student model parameters, optimizer state, and the current training round. In this way, the performance of the student model can be continuously monitored during training, and the best-performing student model can be selected at the end.

[0379] In some embodiments, the key hyperparameters during training can be set as follows: input image size is 224×224, batch size is 64, number of output channels for multi-scale feature adaptation is 256, random channel mask ratio is 0.15, long tail head ratio and tail ratio are both 0.3, temperature parameter is 4.0, weight coefficient corresponding to distillation classification joint loss is 0.7, weight coefficient corresponding to frequency domain branch distillation loss is 1.0, and weight coefficient corresponding to spatial branch distillation loss is 0.5.

[0380] Step S706: Read the current training batch samples and perform forward propagation.

[0381] Read the training samples and their corresponding true class labels from the current training batch in the training set; then, input the training samples from the current training batch into the teacher model and the student model respectively to obtain the output results and teacher features of the teacher model, and the output results and student features of the student model.

[0382] During training, the teacher model parameters can remain fixed, while the student model parameters are updated subsequently.

[0383] For example, in the scenario of recognizing food inside a refrigerator, the images of the current batch of food can be input into the teacher model and the student model respectively to obtain the corresponding output results and intermediate layer features.

[0384] Step S707: Perform frequency domain branch distillation and obtain the frequency domain branch distillation loss.

[0385] Frequency domain bifurcation distillation enables student models to learn knowledge related to the overall contour, structure, and texture frequencies of an object. A frequency domain transformation is performed on the student features to obtain a frequency domain coefficient matrix. Based on preset frequency positions, low-frequency coefficients corresponding to low-frequency locations and high-frequency coefficients corresponding to high-frequency locations are extracted from the frequency domain coefficient matrix. Finally, the object's frequency response is extracted based on these low-frequency and high-frequency coefficients.

[0386] Furthermore, the contour response and texture response of each channel can be combined into a multi-spectral feature vector, and the multi-spectral feature vector can be input into the channel attention module to generate the attention weights corresponding to each channel; the student features can be weighted using the attention weights to obtain the student frequency domain features.

[0387] Next, channel alignment processing can be performed on the teacher features, and the aligned teacher features can be compared with the student frequency domain features. The frequency domain branch distillation loss can be calculated based on the comparison results.

[0388] The above methods enable student models to learn knowledge related to the overall outline, structure, and texture frequency of objects.

[0389] For example, in the scenario of recognizing food inside a refrigerator, the student model can be enhanced to represent the overall shape, structural outline, and detailed texture of the food.

[0390] Step S708: Perform spatial branching distillation and obtain the spatial branching distillation loss.

[0391] Spatial branch distillation enables student models to learn knowledge related to local details of objects and spatial domain semantics, thereby improving the student models' ability to express object features.

[0392] Multi-scale feature adaptation is performed on student features to obtain multi-scale student features that represent local texture details and spatial domain context information of objects. Then, random channel masking is applied to the multi-scale student features to obtain masked features. The masked features are then restored, and spatial distillation is performed based on the restored features and teacher features.

[0393] Among them, multi-scale feature adaptation can extract local details, medium-range patterns and large-scale contextual information through parallel convolutional branches with different kernel sizes; random channel masking can zero out the multi-scale student features element by element through a random channel mask matrix; the recovery process can be completed by a generator network; spatial distillation can be achieved by comparing the recovered student spatial features with the teacher spatial features.

[0394] For example, student features are first input into a multi-scale feature adapter, which contains four parallel branches: a 3×3 convolution (focusing on local details), a 5×5 convolution (focusing on medium-range patterns), a 7×7 convolution (focusing on large-scale context), and a 3×3 max pooling followed by a 1×1 convolution (enhancing salient features). Each branch outputs 256 channels, which are then concatenated and fused using a 1×1 convolution to obtain the multi-scale student features. .

[0395] Teacher features are preserved in their original form, without DCT, spectral extraction, or other processing to avoid information loss introduced by complex teacher-side processing. Channel alignment is performed solely through 1×1 convolution. .

[0396] Then, spatial distillation is performed according to the aforementioned formula (17) to obtain the spatial branch distillation loss.

[0397] The above methods can enhance the student model's ability to learn local texture details and spatial domain context information of objects.

[0398] For example, in the scenario of food recognition inside a refrigerator, the student model can be enhanced to represent the local texture, edge details, and spatial context of adjacent food items.

[0399] For steps S707 and S708, the same batch of equipment and item training samples are input into the teacher model and student model respectively. Each model performs forward computation, outputting corresponding level teacher and student features respectively. Then, frequency domain feature processing is performed on the student features to extract information related to the overall outline, structure, and texture frequency of the item. This information is then distilled with the corresponding knowledge in the teacher features. Specifically, frequency domain transformation, frequency component extraction, frequency response construction, feature weighting, and loss calculation can be used to enable the student model to learn the frequency domain knowledge in the teacher model. Furthermore, spatial domain feature enhancement and restoration processing is performed on the student features to extract information related to the local details and spatial semantics of the item. This information is then distilled with the corresponding knowledge in the teacher features. Specifically, multi-scale feature adaptation, random masking, restoration processing, and spatial feature alignment can be used to enable the student model to learn the spatial domain knowledge in the teacher model. From an overall implementation perspective, this scheme is not a single-branch distillation, but rather performs knowledge transfer along the frequency domain branch and spatial branch for the teacher and student features corresponding to the same batch of training samples, respectively. In other words, the frequency domain branch and the spatial branch together constitute the complete two-branch distillation framework in this scheme.

[0400] Step S709: Count the number of samples for each item category.

[0401] Iterate through all training samples in the training set and count the number of samples corresponding to each item category; you can also sort the number of samples corresponding to each item category to obtain the category distribution in the training set.

[0402] For example, in the scenario of recognizing food inside a refrigerator, the number of images of each type of food in the training set can be counted.

[0403] Step S710: Divide the item categories into long-tail levels based on the number of samples.

[0404] Sort the sample counts for each item category; determine the head category based on the preset head ratio, determine the tail category based on the preset tail ratio, and classify the item categories other than the head and tail categories into the middle category.

[0405] Using the above method, the category distribution in the training set can be explicitly divided into head category, middle category, and tail category, providing a basis for subsequent differential weight allocation.

[0406] Step S711: Assign distillation weights to item categories of different long-tail levels.

[0407] Assign a first distillation weight to the head category, a second distillation weight to the middle category, and a third distillation weight to the tail category; wherein the third distillation weight is greater than the first distillation weight, and / or greater than the second distillation weight.

[0408] The above method allows tail categories with fewer samples to obtain stronger loss constraints during frequency domain branch distillation.

[0409] For example, in the scenario of recognizing food inside a refrigerator, scarce food categories can be given higher distillation attention during training.

[0410] Step S712: Dynamically adjust the weight of the frequency domain branch distillation loss.

[0411] Specifically, during training, the category labels of the current batch of samples can be read; the long-tail level corresponding to the item category to which the current batch of samples belongs can be determined based on the category labels; the distillation weight corresponding to the current batch of samples can be obtained based on the long-tail level; and the frequency domain branch distillation loss can be weighted using the distillation weight to obtain the frequency domain branch distillation loss after dynamic weight adjustment.

[0412] By using the above method, differentiated distillation constraints can be applied to samples of different long-tail levels during training, thereby enhancing the learning of frequency domain knowledge of tail categories.

[0413] Step S713: Obtain the output layer distillation loss and cross-entropy loss to construct the distillation classification joint loss.

[0414] The output layer distillation loss is calculated based on the output results of the teacher model and the student model. The output results of the teacher model and the student model can be temperature scaled first, and then the output layer distillation loss can be calculated based on the scaled output results. At the same time, the cross-entropy loss is calculated based on the student model output results and the true class labels.

[0415] The above methods can simultaneously obtain knowledge transfer information from the teacher model's output layer and real label supervision information.

[0416] The output layer distillation loss and cross-entropy loss are weighted and fused to obtain the joint distillation classification loss. Specifically, corresponding weight coefficients can be set for the output layer distillation loss and cross-entropy loss, and the two can be weighted and summed.

[0417] By using the above methods, the student model can learn the soft label knowledge of the teacher model's output layer while maintaining its ability to classify and learn real labels.

[0418] Step S714: Construct the total training loss.

[0419] The total training loss is obtained by weighted fusion of the distillation classification joint loss, the frequency domain branch distillation loss after dynamic weight adjustment, and the spatial branch distillation loss.

[0420] Specifically, corresponding weight coefficients can be set for the distillation classification joint loss, the frequency domain branch distillation loss after dynamic weight adjustment, and the spatial branch distillation loss, and then a weighted sum can be performed.

[0421] The above methods can be used to simultaneously integrate output layer knowledge transfer, frequency domain knowledge transfer, spatial domain knowledge transfer, and real label supervision information under a unified training objective.

[0422] Step S715: Update the student model parameters based on the total training loss.

[0423] Backpropagation is performed on the total training loss to calculate the gradients corresponding to each parameter of the student model; then, the optimizer updates the student model parameters based on the gradients.

[0424] During this process, only the student model parameters are updated, while the teacher model parameters remain fixed.

[0425] By using the above methods, the student model can continuously reduce the total training loss in each training batch, thereby gradually improving the overall recognition ability.

[0426] Step S716: Evaluate the student model on the validation set and save the best model.

[0427] After each training round, the recognition accuracy of the current student model is evaluated using the validation set; if the validation accuracy of the current student model is higher than the historical best validation accuracy, the parameters of the current student model are saved as the best model checkpoint.

[0428] By using the above method, the student model with the best performance can be selected.

[0429] Step S717: End training and output the trained student model.

[0430] When the preset number of training rounds is reached or the preset training termination condition is met, the training process can be terminated, and the student model with the best verification performance will be output as the trained object recognition model.

[0431] For example, in the scenario of recognizing food inside a refrigerator, the student model with the highest verification accuracy can be used as the output of the trained food recognition model.

[0432] Step S718: Deploy the trained student model on the edge device.

[0433] Specifically, the trained student model can be used as a trained object recognition model, exported into a model format suitable for edge inference, and deployed to edge devices on the device side.

[0434] For example, in the scenario of food recognition inside a refrigerator, the trained food recognition model can be deployed to the edge of the smart refrigerator.

[0435] Step S719: Obtain the image of the object captured by the image acquisition device.

[0436] The image acquisition device in the equipment acquires an image of the object to be identified and uses the image as input for the recognition stage.

[0437] For example, in a refrigerator food recognition scenario, images of the food collected by a refrigerator image acquisition device can be obtained.

[0438] Step S720: Preprocess the item image.

[0439] The object images are scaled to a preset input size and normalized to ensure that the distribution of input data during the deployment phase is consistent with that during the training phase.

[0440] For example, in the scenario of recognizing food inside a refrigerator, the food image can be scaled to 224×224 and normalized.

[0441] Step S721: Input the preprocessed object image into the trained student model for inference.

[0442] The preprocessed object images are input into the trained student model deployed on the edge device of the device. The student model performs forward inference and outputs the prediction results corresponding to each object category.

[0443] For example, in the scenario of food recognition inside a refrigerator, the food recognition model deployed on the refrigerator can output the prediction results corresponding to each food category.

[0444] Step S722: Obtain the item recognition result.

[0445] The category of the current item image is determined based on the category prediction results output by the student model, and this category is then used as the item recognition result.

[0446] For example, in the scenario of food recognition inside a refrigerator, the food category corresponding to the current food image can be determined based on the model output, and this can be used as the food recognition result.

[0447] Through the above deployment, the device can accurately identify various items in real-world scenarios based on the trained item recognition model, especially maintaining high robustness in identifying scarce items, thereby truly realizing functions such as intelligent item management.

[0448] Combination Figure 8 As shown, this disclosure provides an intelligent item recognition device based on long-tail dynamic weight adjustment, including a processor 800 and a memory 801. Optionally, the device may further include a communication interface 802 and a bus 803. The processor 800, communication interface 802, and memory 801 can communicate with each other via the bus 803. The communication interface 802 can be used for information transmission. The processor 800 can call logical instructions in the memory 801 to execute the intelligent item recognition method based on long-tail dynamic weight adjustment described in the above embodiment.

[0449] Furthermore, the logic instructions in the aforementioned memory 801 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0450] The memory 801, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 800 executes functional applications and data processing by running the program instructions / modules stored in the memory 801, thereby realizing the smart item recognition method based on long-tail dynamic weight adjustment in the above embodiments.

[0451] The memory 801 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 801 may include high-speed random access memory and may also include non-volatile memory.

[0452] This disclosure provides a device, including: a device body; an image acquisition device installed on the device body; and the aforementioned intelligent object recognition device based on long-tail dynamic weight adjustment, installed on the device body. The installation relationship described herein is not limited to placement within the device body, but also includes installation and connection with other components of the device, including but not limited to physical connections, electrical connections, or signal transmission connections. Those skilled in the art will understand that the intelligent object recognition device based on long-tail dynamic weight adjustment can be adapted to feasible devices, thereby realizing other feasible embodiments.

[0453] The equipment can be intelligent devices with image acquisition and intelligent recognition functions, including but not limited to intelligent refrigerators, intelligent freezers, intelligent biological cabinets, intelligent vending machines, intelligent retail shelves, intelligent storage cabinets, intelligent medicine cabinets, intelligent kitchen equipment, intelligent warehousing equipment, and intelligent sorting equipment.

[0454] For example, in some specific implementations, the device can be a smart refrigerator, with an image acquisition device for acquiring images of food inside the refrigerator, and a smart item recognition device based on long-tail dynamic weight adjustment for recognizing the food images.

[0455] It should be understood that any device with an image acquisition device and an object recognition requirement can be used with the technical solutions provided in the embodiments of this disclosure, and the embodiments of this disclosure do not limit this.

[0456] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to perform the above-described intelligent item recognition method based on long-tail dynamic weight adjustment.

[0457] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., and other media capable of storing program code.

[0458] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0459] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0460] The methods and products disclosed in the embodiments herein (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0461] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A smart object recognition method based on long-tail dynamic weight adjustment, characterized in that, include: Obtain an image sample of the item to be identified; The image samples are input into the trained item recognition model. The item recognition model is trained as follows: the joint distillation loss for classification, the frequency domain branch distillation loss after dynamic weight adjustment, and the spatial branch distillation loss are obtained; the joint distillation loss for classification, the frequency domain branch distillation loss after dynamic weight adjustment, and the spatial branch distillation loss are weighted and fused to obtain the total training loss; and the student model is trained based on the total training loss, and the trained student model is used as the item recognition model; wherein, the frequency domain branch distillation loss after dynamic weight adjustment is obtained by applying differentiated weight adjustment to the frequency domain branch distillation loss based on the sample frequencies of different item categories; Output the category identification result of the item to be identified; The frequency domain branch distillation loss after dynamic weight adjustment is obtained as follows: Count the number of samples corresponding to each item category in the training set; The item categories are divided into long-tail levels based on the number of samples corresponding to each item category; Different distillation weights are assigned to different item categories based on the long-tail classification results; The frequency domain branch distillation loss of the student model is adjusted using the distillation weights to obtain the dynamically weighted frequency domain branch distillation loss. The step of classifying the item categories into long-tail levels based on the number of samples corresponding to each item category includes: Sort the number of samples for each item category; The head category is determined based on the preset head proportion, the tail category is determined based on the preset tail proportion, and all other item categories, excluding the head and tail categories, are classified as the middle category.

2. The item identification method according to claim 1, characterized in that, The process of acquiring an image sample of the current object to be identified includes: The acquired image of the object to be identified is scaled to a preset size; The image of the item of the preset size is normalized to a preset numerical range in the following way to obtain the image sample of the item to be identified: ； in, These are the normalized pixel values. These are the original pixel values. and These are the maximum and minimum pixel values ​​in the image, respectively.

3. The item identification method according to claim 1, characterized in that, The process of assigning different distillation weights to different item categories based on the long-tail grading results includes: Assign the first distillation weight to the head category; Assign a second distillation weight to the central category; Assign a third distillation weight to the tail category; Wherein, the third distillation weight is greater than the first distillation weight, and / or the third distillation weight is greater than the second distillation weight.

4. The item identification method according to claim 1, characterized in that, The frequency domain branch distillation loss of the student model is adjusted using the distillation weights, including: During training, read the class labels of the current batch of samples; Determine the long-tail level corresponding to the item category of the current batch of samples based on the category label; The distillation weight corresponding to the current batch of samples is obtained based on the long-tail level. The frequency domain branch distillation loss is weighted using the distillation weights to obtain the weighted frequency domain branch distillation loss.

5. The item identification method according to claim 4, characterized in that, The weighted frequency domain branch distillation loss is calculated according to the following formula: ; in, This represents the weighted frequency domain branching distillation loss. Indicates the first yi The distillation weights corresponding to the categories of each sample. Let N represent the frequency domain branch distillation loss corresponding to the i-th sample, and N represent the batch size.

6. The item identification method according to claim 5, characterized in that, The total training loss is calculated according to the following formula: ; in, For total training losses, For the combined loss of distillation classification, This refers to the frequency domain branch distillation loss after dynamic weight adjustment. For spatial branching distillation losses, These are the weighting coefficients for the distillation classification joint loss, the frequency domain branch distillation loss after dynamic weight adjustment, and the spatial branch distillation loss, respectively.

7. A smart object recognition device based on long-tail dynamic weight adjustment, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to execute the smart item recognition method based on long-tail dynamic weight adjustment as described in any one of claims 1 to 6 when running the program instructions.

8. A device, characterized in that, include: Equipment body; The intelligent object recognition device based on long-tail dynamic weight adjustment as described in claim 7 is installed on the device body.

Images