Multi-task fine-grained classification method for intelligent electronic scale
Through a multi-task learning framework, combined with global and local feature extraction, picture segmentation and object detection, an end-to-end fine-grained classification model is built, which solves the problems of small in-class gaps and complex inter-class features in fresh food product identification, and achieves high-precision and robust fresh food classification.
Patent Information
- Application Number
- CN202510516440.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-23
- Publication Date
- 2025-08-08
AI Technical Summary
The existing fine-grained classification methods are difficult to achieve high accuracy in fresh product identification, especially in scenarios such as vending machines, intelligent warehousing management and food safety testing, which have overfitting and insufficient generalization capabilities.
Using a multi-task learning framework, combining global feature extraction, progressive local feature learning, local feature enhancement, picture segmentation and object detection, an end-to-end fine-grained classification model is built through the Transformer architecture and convolutional neural network, and the shared information between multiple tasks is used to improve feature extraction capabilities.
It significantly improves the classification accuracy and robustness of fresh products, reduces overfitting, enhances the generalization ability of the model, and adapts to the practical application of complex scenarios.
Smart Images
Figure CN120451640A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and pattern recognition, and in particular to a multi-task fine-grained classification method for intelligent electronic scales. Background Art
[0002] In today's rapidly developing retail and logistics sectors, accurate fresh produce identification technology is becoming crucial for improving efficiency and quality. Fresh produce is diverse, with high levels of similarity in appearance, and subtle variations between batches and origins. These characteristics make traditional image recognition technology a significant challenge in distinguishing fresh produce, especially in scenarios requiring high-precision identification, such as vending machines, intelligent warehouse management, and food safety testing.
[0003] With the continuous advancement of computer vision technology, fine-grained image recognition has gradually become a potential solution to this problem. Fine-grained image recognition aims to accurately classify similar objects by extracting subtle features from images. However, the image recognition task of fresh produce is unique: on the one hand, features such as texture, color, and shape of fresh produce vary significantly between individuals; on the other hand, within certain fresh produce categories, the differences are small, and learning between-category differences is difficult; on the other hand, fresh produce may deform, be damaged, or deteriorate during transportation and storage, further increasing the difficulty of recognition.
[0004] Most existing fine-grained classification methods rely on single-task learning, directly training the model on the target classification task. While this approach can achieve some success in certain situations, it often struggles to achieve ideal accuracy when dealing with complex and changing scenarios like fresh produce. Furthermore, single-task learning methods are prone to overfitting when handling fine-grained classification tasks, resulting in insufficient generalization capabilities in practical applications.
[0005] To address these issues, multi-task learning has gradually gained attention. Multi-task learning simultaneously trains multiple related tasks, leveraging shared information between tasks to improve the model's generalization and feature extraction capabilities. However, currently, there are still few multi-task learning methods for fine-grained classification of fresh produce, and most methods fail to fully utilize auxiliary tasks such as image segmentation and object detection to enhance feature extraction. In summary, developing a fine-grained classification method that can effectively combine multi-task learning is of great significance for improving the accuracy and robustness of fresh produce recognition.
[0006] In view of this, this application is filed. Summary of the Invention
[0007] The present invention provides a multi-task fine-grained classification method for an intelligent electronic scale, which can at least partially improve the above-mentioned problem.
[0008] To achieve the above object, the present invention adopts the following technical solutions:
[0009] A multi-task fine-grained classification method for smart electronic scales, comprising:
[0010] Acquire fresh produce images from the internet and from on-site fresh produce supermarkets to establish a fresh produce image dataset, and invoke the first global feature extraction module of the end-to-end fine-grained classification model to perform a first global extraction process on the fresh produce image dataset to obtain preliminary global features.
[0011] The fresh food image dataset is extracted using a progressive local feature learning module of an end-to-end fine-grained classification model to obtain discriminative local features, and the discriminative local features are enhanced. The enhanced local features are subjected to a second global feature extraction module of the end-to-end fine-grained classification model, and the extracted features are fused with the preliminary global features to obtain refined extracted features.
[0012] The image segmentation module, object detection and classification module, and fine-grained classification module of the end-to-end fine-grained classification model are respectively called to preprocess the refined extracted features to obtain segmentation results, object detection and classification results, and image classification results. The end-to-end fine-grained classification model is then trained based on the segmentation results, object detection and classification results, and image classification results.
[0013] Obtain the fresh produce images to be classified, use the trained end-to-end fine-grained classification model to classify the fresh produce images to obtain the final image classification results.
[0014] In summary, this multi-task fine-grained classification method for smart electronic scales focuses on improving the performance of smart electronic scales in fresh produce classification. It proposes an innovative multi-task learning framework. By combining multiple advanced deep learning techniques, it effectively addresses the classification challenges of fresh produce image recognition, which arise from the small intra-class variance and complex inter-class features.
[0015] The core of this method lies in achieving accurate classification of fresh produce images through the collaborative work of multiple modules: first, a global feature extraction module captures the overall image information, then a progressive local feature learning module gradually refines local features, and finally, a local feature enhancement module deeply fuses global and local features. Furthermore, by introducing image segmentation and object detection tasks as auxiliary tools, the feature extraction process is further optimized, enhancing the model's ability to learn texture and individual features in fresh produce images. Finally, the fine-grained classification module outputs highly accurate classification results based on the fused features. This method not only significantly improves classification accuracy but also enhances the model's generalization ability, providing an efficient and reliable solution for the application of smart electronic scales in the field of fresh produce classification. It aims to address the challenges of small intra-class differences and the difficulty of learning inter-class differential features in fresh produce image recognition, thereby improving the accuracy of fine-grained classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 1 is a flow chart of a multi-task fine-grained classification method for an intelligent electronic scale provided by an embodiment of the present invention;
[0017] Figure 2 This is a logic block diagram of a multi-task fine-grained classification method for an intelligent electronic scale provided by an embodiment of the present invention;
[0018] Figure 3 This is a block diagram of the implementation principle of the multi-task fine-grained classification method for smart electronic scales provided by an embodiment of the present invention;
[0019] Figure 4 This is a framework diagram of a global feature learning module provided by an embodiment of the present invention;
[0020] Figure 5 This is a framework diagram of a progressive local feature learning module provided by an embodiment of the present invention;
[0021] Figure 6 This is a framework diagram of the local feature enhancement module provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0023] refer to Figures 1 to 3 As shown, the first embodiment of the present invention discloses a multi-task fine-grained classification method for an intelligent electronic scale, which can be executed by a multi-task fine-grained classification device for an intelligent electronic scale (hereinafter referred to as a classification device), and in particular, is executed by one or more processors in the classification device to implement the following method:
[0024] See also Figure 4 S1, obtaining fresh food pictures on the Internet and fresh food pictures from on-site fresh food supermarkets, establishing a fresh food picture dataset, and calling the first global feature extraction module of the end-to-end fine-grained classification model to perform a first global extraction process on the fresh food picture dataset to obtain preliminary global features;
[0025] The fresh food image dataset includes fresh food images, label data and true value masks used for image segmentation, label information used for fine-grained detection and classification of images, and classification information of individual images.
[0026] Specifically, step S1 includes: flattening the training samples in the fresh image dataset to obtain a global sequence containing image blocks, and embedding position codes into the sequence;
[0027] The sequence of embedded position codes is transferred to the Transformer encoder for iterative processing. At the same time, the sequence is converted into three-dimensional features, and preliminary global features are extracted based on the self-attention mechanism.
[0028] In this example, a large number of fresh produce images were collected from the internet and combined with images taken in on-site fresh produce supermarkets to construct a comprehensive and diverse fresh produce image dataset. This dataset not only contains the fresh produce images themselves, but also includes label data and ground truth masks for image segmentation tasks, label information for fine-grained image detection and classification, and classification information for individual images. This rich data provides a solid foundation for model training, enabling the model to learn the characteristics of fresh produce images in different scenarios, thereby improving classification accuracy and robustness. The fresh produce images and ground truth masks are in JPG format, and the label information and classification information for individual images are in JSON format.
[0029] During the data preprocessing phase, the training samples in the dataset are flattened to obtain a global sequence of image patches. To preserve the global spatial information of the image, positional encodings are embedded in this sequence. This operation is crucial because it enables the model to better understand the spatial relationships between image patches during subsequent processing, thereby more accurately extracting global features. The sequence embedded with positional encodings is then passed to the Transformer encoder. Compared to traditional convolutional neural networks (CNNs), the Transformer encoder iteratively processes the sequence using a self-attention mechanism, which effectively captures associations across the entire feature space, providing a larger "receptive field" and thus better extracting global features. During processing, the sequence is converted into three-dimensional features, and ultimately, preliminary global features are extracted based on the self-attention mechanism.
[0030] See also Figures 5 and 6 S2, using the progressive local feature learning module of the end-to-end fine-grained classification model to extract and process the fresh image dataset to obtain discriminative local features, and then enhance the discriminative local features. The second global feature extraction module of the end-to-end fine-grained classification model is used to perform a second global extraction process on the enhanced local features, and the extracted features are fused with the preliminary global features to obtain refined extracted features.
[0031] Specifically, step S2 includes: using a convolutional neural network with a small receptive field as the optimization target of the first stage, gradually expanding the local area range, and introducing deep convolutional layers for training to obtain a progressive local feature learning module;
[0032] The fresh food image dataset is extracted and processed using a progressive local feature learning module to obtain fine discriminative local features.
[0033] Using a self-attention module to obtain the preliminary global features, obtain the relationship between local features of different regions, and enhance the discriminative local features based on the relationship to obtain enhanced local features, wherein the enhanced local features are three-dimensional features;
[0034] The enhanced local features are converted into sequence features, and the length of the sequence features is extended to the same length as the global sequence through the fully connected layer. The corresponding weight addition formula is: Output = F_s × F_g + F_g, where Output is the final output, F_s is the feature extracted by the self-attention module, and F_g is the preliminary global feature;
[0035] The final output is input into the second global feature extraction module for second global extraction processing, and the extracted features are fused and connected with the preliminary global features to obtain fused refined extracted features;
[0036] Among them, the mathematical expression of the global feature extraction module is: Q, K, V are all linear projections of self-attention, and d is a bias constant.
[0037] In this embodiment, the fresh image dataset is processed by the progressive local feature learning module of the end-to-end fine-grained classification model. The core of this module is to use a convolutional neural network (CNN) with a small receptive field as the optimization target in the initial stage. This design enables the model to focus on local details in the image, gradually expanding the scope of the local area while introducing deeper convolutional layers for training, thereby constructing a progressive local feature learning module. This process not only extracts fine discriminative local features, but also enables the model to capture a wider range of local information by gradually expanding the receptive field, providing a rich and detailed foundation for subsequent feature fusion.
[0038] After obtaining discriminative local features, the self-attention module processes the preliminary global features to identify relationships between local features in different regions. The self-attention mechanism dynamically focuses on more important parts of the image, thereby enhancing the discriminative local features. The enhanced local features are presented as three-dimensional features, further improving their expressiveness. Subsequently, the enhanced local features are converted into sequence features and extended to the length of the global sequence using a fully connected layer. This conversion process is achieved using a specific weight addition formula, with the final output being a fusion of the features extracted by the self-attention module and the preliminary global features. This fusion approach not only preserves the macroscopic information of the global features but also incorporates the enhanced local details, resulting in richer and more comprehensive features.
[0039] Finally, the fused features are fed into the second global feature extraction module for a second global extraction process. This module's mathematical expression is based on a self-attention mechanism, further refining and integrating features through linear projection and bias constant calculation. The extracted features are then fused with the preliminary global features to produce refined features. This refined feature extraction combines the advantages of both global and local features, providing a more accurate and richer feature representation for subsequent classification tasks.
[0040] Preferably, the first global extraction process and the second global extraction process share a weight, and the formula for the shared weight is: W second =α×W first ×β, where W second is the second global feature extraction module, α is the adaptive dynamic sharing ratio parameter, W first is the first global feature extraction module, β is the learnable shared sequence, and the length of the learnable shared sequence β is the same as the length of the global sequence;
[0041] Among them, the adaptive dynamic sharing ratio parameter α can be updated according to the total loss value, and the update formula is: α newis the updated adaptive dynamic sharing ratio parameter, and loss is the total loss value of the model training;
[0042] The learnability shared sequence β can be updated according to the number of training rounds epoch, where the learnability shared sequence β defaults to 1. When the epoch reaches one-third of the preset maximum training epoch, the learnability shared sequence β is updated.
[0043] In this embodiment, the multi-task fine-grained classification method for smart electronic scales performs two global feature extraction operations: the first for fresh produce images, and the second, after feature enhancement, followed by another global feature extraction operation. The two operations share weights. The weight sharing of the two global feature learning modules occurs in two phases. In the first phase, the two global feature learning modules fully share weights, forcing the model to learn a common feature representation, avoiding premature overfitting and accelerating convergence. In the second phase, the independent weights are gradually unfrozen, allowing the two global feature learning modules to learn finer-grained feature differences based on the input characteristics, thereby improving the model's expressiveness.
[0044] Specifically, in order to further optimize the training efficiency and feature extraction capabilities of the model, a weight sharing mechanism between global feature extraction modules is specially designed. The core of this mechanism is that the first global extraction process and the second global extraction process share weights. By dynamically adjusting the sharing ratio and sequence, the model can learn common feature representations more efficiently during training and gradually adapt to the specific needs of different tasks. The design of this dynamic sharing mechanism enables the model to learn common feature representations by fully sharing weights in the early stages of training, thereby avoiding premature overfitting and accelerating the convergence process. As training progresses, the model gradually unfreezes independent weights, allowing the two global feature extraction modules to learn finer-grained feature differences based on the input characteristics, thereby improving the expressiveness and adaptability of the model.
[0045] To achieve this dynamic adjustment, the adaptive dynamic sharing ratio parameter α can be updated based on the total loss value of model training. This loss-based update mechanism enables the model to dynamically adjust the sharing ratio based on the current training state, thereby better balancing generality and specificity during global feature extraction. In addition, the update strategy of the learnable sharing sequence β is also closely related to the number of training epochs. Initially, the learnable sharing sequence β defaults to all 1s, indicating that weights are fully shared at the beginning of training. As training progresses, when the epoch reaches one-third of the preset maximum training epoch, the learnable sharing sequence β begins to be updated. This phased update strategy allows the model to fully utilize shared weights for rapid convergence in the early stages of training, while gradually introducing independent weights in the later stages of training to learn more fine-grained feature differences.
[0046] S3: Call the image segmentation module, object detection and classification module, and fine-grained classification module of the end-to-end fine-grained classification model respectively to preprocess the refined extracted features, obtain segmentation results, object detection and classification results, and image classification results, and train the end-to-end fine-grained classification model based on the segmentation results, object detection and classification results, and image classification results;
[0047] Specifically, step S3 includes: using the Mamba module as the image segmentation module, wherein a splitting operation is performed based on the learnable upsampling layer of the Mamba module, and the image is divided into three upsampling modules, each of which includes two layers of transposed convolution and two layers of activation functions;
[0048] Convert the input three-dimensional features H×W×C into [H×W]×C serialized features, and input the serialized features into the upsampling modules in sequence. In each upsampling module, transposed convolution, activation function, transposed convolution, and activation function are performed in sequence.
[0049] The processed [H×W]×C serialized features are converted into H′×W′×C three-dimensional features, and a feature matrix of the same size as the original image is output to obtain the segmentation result.
[0050] The Mamba pyramid feature fusion module is used as the target detection and classification module. The target detection and classification module is divided into three parts, each of which consists of three Mamba modules.
[0051] The refined extracted features CNN_Featurne are serialized and the processed sequences are input into the first Mamba module for feature extraction to obtain the feature Output Mamba1 ,Output Mamba1 =Mamba(CNN_Feature);
[0052] Output the feature Mamba1 Channel fusion is performed with the refined extracted feature CNN_Featurne and input into the second Mamba module for feature extraction to obtain the feature Output Mamba2 , in, Indicates channel fusion;
[0053] Output the feature Mamba1 , Feature Output Mamba2 The channel is fused with the refined extracted feature CNN_Featurne and input into the third Mamba module for feature extraction to obtain the target detection classification result Output.
[0054] In this embodiment, a multi-task learning framework is designed, which includes an image segmentation module, an object detection and classification module, and a fine-grained classification module. These modules work together to process and refine extracted features, not only to obtain segmentation results, object detection and classification results, and image classification results, but also to use these results to train an end-to-end fine-grained classification model, thereby achieving model optimization and improvement.
[0055] Specifically, the image segmentation module utilizes the advanced Mamba module. The core of this module lies in its learnable upsampling layer, which is cleverly split into three upsampling modules, each of which contains two layers of transposed convolutions and two layers of activation functions. This design not only enhances the model's learning capabilities but also improves feature extraction efficiency. During processing, the input three-dimensional features are first converted into serialized features and then sequentially input into each upsampling module. Within each module, transposed convolution, activation function, transposed convolution, and activation function are sequentially performed. After this series of operations, the processed serialized features are reconverted into three-dimensional features, ultimately outputting a feature matrix of the same size as the original image, resulting in the segmentation result. This process not only preserves the spatial information of the image but also enhances the detailed representation of the features through upsampling, providing richer information for subsequent classification tasks.
[0056] The target detection and classification module utilizes the Mamba pyramid feature fusion module, which is divided into three parts, each consisting of three Mamba modules. This pyramid structure enables the model to effectively process features at different scales, thereby improving the accuracy and robustness of target detection. When processing refined features, the features are first serialized and then input into the first Mamba module for feature extraction to obtain a preliminary feature representation. Next, this preliminary feature is channel-wise fused with the refined features and input into the second Mamba module for further feature extraction. Finally, the resulting features are again channel-wise fused with the previous features and the refined features and input into the third Mamba module to obtain the target detection and classification results. This process, through multiple feature fusions and extractions, enables the model to better capture the target's characteristic information, thereby improving target detection accuracy.
[0057] Through this multi-task learning framework, this method can not only simultaneously handle image segmentation, object detection and classification, and fine-grained classification tasks, but also use the results of these tasks to train the model, thereby achieving model optimization and improvement. This design not only improves the model's generalization ability, but also makes the model more accurate and efficient when handling complex fresh food image classification tasks. Through the image segmentation task, the model can learn the texture information of the image; through the object detection and classification task, the model can learn the characteristics of individual fresh food items; and through the fine-grained classification task, the model can ultimately achieve accurate classification of fresh food images. This multi-task collaborative approach allows the model to fully utilize the feature information of each task during training, thereby achieving higher classification accuracy and robustness in practical applications.
[0058] Preferably, for each scale, the output of the target detection head of the target detection and classification module is a shape of [B, (5+C)*num abchors ,H,W] tensor, where B is the batch size, C is the number of categories, num abchors is the number of anchor points used for this scale, H is the height of the feature map, W is the width of the feature map, and the 5 in the formula represents the 5 parameters that need to be predicted for each bounding box. x , t y , t w , t h and confidence.
[0059] In this embodiment, the design of the target detection and classification module is particularly critical. The output structure of the target detection head is carefully designed to meet the requirements of accurate detection and classification of target objects in fresh images at different scales. Specifically, for each scale, the output of the target detection head is a [B, (5 + C) * num abchprs ,H,W] tensor. Each dimension of this tensor structure has a clear meaning and function, which together ensures efficient and accurate object detection. This design enables the model to not only determine the location and size of the target object, but also evaluate the reliability of the detection results, providing more accurate input for subsequent classification tasks.
[0060] Through this carefully designed object detection head output structure, the object detection and classification module can effectively process fresh produce images at different scales, achieving precise location and classification of target objects. This multi-scale detection capability is particularly important for fine-grained classification of fresh produce images, as the size and shape of fresh produce items in images can vary significantly depending on factors such as shooting angle and distance. By setting multiple anchor points and feature maps of different scales, the model can adapt to these variations, thereby improving detection accuracy and robustness.
[0061] Preferably, the overall loss function Loss of the end-to-end fine-grained classification model is: Loss = Loss U-Net +3×Loss cls +Loss obj_cls +Loss obj_IoU , which includes the loss function Loss for the image segmentation task U-Net , the loss function Loss of the target detection classification task obj_cls and Loss obj_IOU , and the loss function Loss for fine-grained classification tasks cls .
[0062] In this embodiment, the overall loss function of the end-to-end fine-grained classification model consists of three parts: the loss function for the image segmentation task, the loss function for the object detection and classification task, and the loss function for the fine-grained classification task. These weight coefficients can be adjusted based on the importance and difficulty of the task to ensure that the model can balance the optimization of each task during training.
[0063] The loss function for image segmentation tasks typically uses pixel-level cross-entropy loss, which measures the difference between the segmentation results output by the model and the ground-truth segmentation mask. By optimizing this loss function, the model can learn the texture characteristics of fresh images, thereby more accurately segmenting different regions. This segmentation capability not only provides clearer image region information for fine-grained classification, but also enhances the model's understanding of image details.
[0064] The loss function for object detection and classification tasks typically consists of two parts: a bounding box regression loss and a classification loss. The bounding box regression loss optimizes the prediction of the bounding box's position and size, while the classification loss optimizes the prediction of the class of the object within the bounding box. By optimizing this loss function, the model can learn the characteristics of individual objects in fresh images, enabling more accurate object detection and classification. This object detection capability not only provides more specific object features for fine-grained classification but also enhances the model's ability to recognize objects.
[0065] The loss function for fine-grained classification tasks typically uses categorical cross-entropy loss, which measures the difference between the model's output and the true category. By optimizing this loss function, the model can learn the fine-grained features of fresh images, enabling more accurate classification. This classification capability forms the basis of the model's final output and directly determines classification accuracy.
[0066] S4: Obtain the fresh food images to be classified, and use the trained end-to-end fine-grained classification model to classify the fresh food images to obtain the final image classification results.
[0067] Specifically, in this embodiment, a trained end-to-end fine-grained classification model is used to classify the fresh produce images. Through multi-task learning and joint optimization in the previous steps, this model already possesses powerful feature extraction and classification capabilities. The model first extracts global and local features from the image using a global feature extraction module and a progressive local feature learning module. The local feature enhancement module then further optimizes these features, making them more suitable for the classification task. Finally, the fine-grained classification module outputs the final classification results based on these optimized features.
[0068] In summary, the multi-task fine-grained classification method for smart electronic scales is specifically used for the fine-grained classification of fresh food images in smart electronic scales. By building an end-to-end fine-grained classification model and combining image segmentation, object detection and classification, and fine-grained classification tasks, the classification accuracy and robustness are significantly improved.
[0069] Specifically, a large number of fresh produce images were collected from the internet and in-person fresh produce supermarkets to create a dataset containing rich annotated information. This annotation information not only includes the image classification labels, but also includes ground truth masks for image segmentation and labels for object detection. This comprehensive dataset provides a solid foundation for the model's multi-task learning, enabling the model to simultaneously learn global image features, local texture features, and individual fresh produce item characteristics during training. The core of this multi-task learning framework is its collaborative work, which achieves efficient fresh produce image classification through the collaborative work of a global feature extraction module, a progressive local feature learning module, a local feature enhancement module, an image segmentation module, an object detection and classification module, and a fine-grained classification module. The global feature extraction module utilizes a Transformer architecture to extract global information from the image, while the progressive local feature learning module gradually refines local features using a convolutional neural network. The local feature enhancement module further optimizes the feature extraction process through a self-attention mechanism, enabling the model to better capture subtle differences in fresh produce images.
[0070] As auxiliary tasks, the image segmentation module and the object detection and classification module not only provide rich feature information for the fine-grained classification task, but also play a regularization role in the training process, effectively reducing overfitting. Through multi-task learning, the model can fully utilize the complementarity between tasks during training, thereby learning more comprehensive and fine-grained feature representations. This multi-task collaborative approach not only improves the model's classification accuracy, but also enhances the model's generalization ability, enabling it to better adapt to various complex practical application scenarios. Ultimately, by processing the fresh food images to be classified with the trained end-to-end fine-grained classification model, the model can quickly and accurately output classification results. This end-to-end classification approach not only improves work efficiency but also reduces the possibility of human error, enabling the model to meet the real-time requirements of practical application scenarios.
[0071] Compared to existing technologies, the proposed multi-task fine-grained classification method for smart electronic scales offers the following advantages: through a multi-task learning framework, the model can fully utilize auxiliary information from image segmentation and object detection and classification tasks, significantly improving the accuracy and robustness of fine-grained classification. Furthermore, by jointly training multiple tasks, the model can more effectively utilize training data, reduce overfitting, and thus improve the model's generalization ability. This innovative multi-task learning method provides an efficient, accurate, and practical solution for the fresh produce image classification task in smart electronic scales, with broad application prospects and important practical significance.
[0072] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A multi-task fine-grained classification method for smart electronic scales, characterized in that: include: Acquire fresh produce images from the internet and from on-site fresh produce supermarkets to establish a fresh produce image dataset, and invoke the first global feature extraction module of the end-to-end fine-grained classification model to perform a first global extraction process on the fresh produce image dataset to obtain preliminary global features. The fresh food image dataset is extracted using a progressive local feature learning module of an end-to-end fine-grained classification model to obtain discriminative local features, and the discriminative local features are enhanced. The enhanced local features are subjected to a second global feature extraction module of the end-to-end fine-grained classification model, and the extracted features are fused with the preliminary global features to obtain refined extracted features. The image segmentation module, object detection and classification module, and fine-grained classification module of the end-to-end fine-grained classification model are respectively called to preprocess the refined extracted features to obtain segmentation results, object detection and classification results, and image classification results. The end-to-end fine-grained classification model is then trained based on the segmentation results, object detection and classification results, and image classification results. Obtain the fresh produce images to be classified, use the trained end-to-end fine-grained classification model to classify the fresh produce images to obtain the final image classification results.
2. The multi-task fine-grained classification method for intelligent electronic scales according to claim 1 is characterized in that: The fresh food image dataset includes fresh food images, label data and true value masks used for image segmentation, label information used for fine-grained detection and classification of images, and classification information of individual images.
3. The multi-task fine-grained classification method for smart electronic scales according to claim 1, characterized in that: The first global feature extraction module of the end-to-end fine-grained classification model is called to perform a first global extraction process on the fresh food image dataset to obtain preliminary global features, specifically: Flattening the training samples in the fresh image dataset to obtain a global sequence containing image blocks, and embedding position encoding into the sequence; The sequence of embedded position codes is transferred to the Transformer encoder for iterative processing. At the same time, the sequence is converted into three-dimensional features, and preliminary global features are extracted based on the self-attention mechanism.
4. The multi-task fine-grained classification method for intelligent electronic scales according to claim 1, characterized in that: The progressive local feature learning module of the end-to-end fine-grained classification model is used to extract and process the fresh food image dataset to obtain discriminative local features, specifically: A convolutional neural network with a small receptive field is used as the optimization target in the first stage, and the local area range is gradually expanded. At the same time, deep convolutional layers are introduced for training to obtain a progressive local feature learning module. The fresh food image dataset is extracted and processed using a progressive local feature learning module to obtain fine discriminative local features.
5. The multi-task fine-grained classification method for intelligent electronic scales according to claim 3, characterized in that: The discriminative local features are increased, and the second global feature extraction module of the end-to-end fine-grained classification model is used to perform a second global extraction process on the enhanced local features. The extracted features are fused with the preliminary global features to obtain refined extracted features, specifically: Using a self-attention module to obtain the preliminary global features, obtain the relationship between local features of different regions, and enhance the discriminative local features based on the relationship to obtain enhanced local features, wherein the enhanced local features are three-dimensional features; The enhanced local features are converted into sequence features, and the length of the sequence features is extended to the same length as the global sequence through the fully connected layer. The corresponding weight addition formula is: Output = F_s × F_g + F_g, where Output is the final output, F_s is the feature extracted by the self-attention module, and F_g is the preliminary global feature; The final output is input into the second global feature extraction module for second global extraction processing, and the extracted features are fused and connected with the preliminary global features to obtain fused refined extracted features; Among them, the mathematical expression of the global feature extraction module is: Q, K, V are all linear projections of self-attention, and d is a bias constant.
6. The multi-task fine-grained classification method for intelligent electronic scales according to claim 3, characterized in that: The first global extraction process and the second global extraction process share a weight, and the formula for the shared weight is: W second =α×W first ×β, where W second is the second global feature extraction module, α is the adaptive dynamic sharing ratio parameter, W first is the first global feature extraction module, β is the learnable shared sequence, and the length of the learnable shared sequence β is the same as the length of the global sequence; Among them, the adaptive dynamic sharing ratio parameter α can be updated according to the total loss value, and the update formula is: α new is the updated adaptive dynamic sharing ratio parameter, and loss is the total loss value of the model training; The learnability shared sequence β can be updated according to the number of training rounds epoch, where the learnability shared sequence β defaults to 1. When the epoch reaches one-third of the preset maximum training epoch, the learnability shared sequence β is updated.
7. The multi-task fine-grained classification method for intelligent electronic scales according to claim 1, characterized in that: The image segmentation module of the end-to-end fine-grained classification model is called to preprocess the refined extracted features to obtain the segmentation results, specifically: The Mamba module is used as the image segmentation module, where The learnable upsampling layer based on the Mamba module is split into three upsampling modules, each of which contains two layers of transposed convolution and two layers of activation functions; Convert the input three-dimensional features H×W×C into [H×W]×C serialized features, and input the serialized features into the upsampling modules in sequence. In each upsampling module, transposed convolution, activation function, transposed convolution, and activation function are performed in sequence. The processed [H×W]×C serialized features are converted into H′×W′×C three-dimensional features, and a feature matrix of the same size as the original image is output to obtain the segmentation result.
8. The multi-task fine-grained classification method for smart electronic scales according to claim 1, characterized in that: The target detection and classification module of the end-to-end fine-grained classification model is called to preprocess the refined extracted features to obtain the target detection and classification results, specifically: The Mamba pyramid feature fusion module is used as the target detection and classification module. The target detection and classification module is divided into three parts, each of which consists of three Mamba modules. The refined extracted features CNN_Featurne are serialized and the processed sequences are input into the first Mamba module for feature extraction to obtain the feature Output Mamba1 ,Output Mamba1 =Mamba(CNN_Feature); Output the feature Mamba1 Channel fusion is performed with the refined extracted feature CNN_Featurne and input into the second Mamba module for feature extraction to obtain the feature Output Mamba2 , in, Indicates channel fusion; Output the feature Mamba1 , Feature Output Mamba2 The channel is fused with the refined extracted feature CNN_Featurne and input into the third Mamba module for feature extraction to obtain the target detection classification result Output.
9. The multi-task fine-grained classification method for intelligent electronic scales according to claim 1, characterized in that: For each scale, the output of the target detection head of the target detection and classification module is a shape of [B, (5 + C) * num abchors ,H,W] tensor, where B is the batch size, C is the number of categories, num abchors is the number of anchor points used for this scale, H is the height of the feature map, W is the width of the feature map, and the 5 in the formula represents the 5 parameters that need to be predicted for each bounding box. x , t y , t w , t h and confidence.
10. The multi-task fine-grained classification method for smart electronic scales according to claim 1, characterized in that: The overall loss function Loss of the end-to-end fine-grained classification model is: Loss = Loss U-Net +3×Loss cls +Loss obj_cls +Loss obj_IOU , which includes the loss function Loss for the image segmentation task U-Net , the loss function Loss of the target detection classification task obj_cls and Loss obj_IOU , and the loss function Loss for fine-grained classification tasks cls .