Image classification model training method, image classification method, device and equipment
By improving the ShuffleNetv2 model and flexibly adjusting model parameters, the problems of high complexity and poor stability of image classification models on edge devices were solved, thereby improving classification performance and efficiency and enhancing the accuracy of image classification results.
Patent Information
- Application Number
- CN202411103471.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-13
AI Technical Summary
Existing deep learning image classification methods suffer from problems such as high model complexity, poor classification performance, long model training time, poor stability, and low reliability when classifying images on very small datasets on edge devices with extremely limited computing power.
We adopted the improved ShuffleNetv2 model, adjusted the network structure, replaced the fully connected layers with multilayer perceptrons, and flexibly adjusted the timing of the optimal model parameters during training. We also combined data augmentation and early stopping algorithms to optimize model training, reduce the number of parameters and model complexity, and improve the model's ability to adapt to dynamic changes in training data.
It improves the classification performance, stability, and reliability of image classification models, enhances the accuracy and efficiency of image classification results on edge devices, and reduces model complexity.
Smart Images

Figure CN118644740B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to an image classification model training method, an image classification method, a device and equipment. Background Art
[0002] Although the existing deep learning image classification methods have made significant progress, they still have some shortcomings when performing image classification processing on extremely small data sets (only 5-10 images per category) on edge devices with extremely limited computing power, or completing model training and inference in a very short training time. The specific manifestations include high model complexity, poor model classification performance, long model training time, poor model stability and low model reliability. Summary of the invention
[0003] The present invention provides an image classification model training method, an image classification method, an apparatus and equipment to improve the model classification performance, model stability and model reliability of the image classification model, thereby improving the classification efficiency of image classification completed on edge devices and the accuracy of image classification results.
[0004] According to one aspect of the present invention, a method for training an image classification model is provided, the method comprising:
[0005] Based on the initial image classification model, determine the sample image classification result of the sample image data in the current training round;
[0006] Determine a current loss value of a current training round according to the sample image classification result and the label image data of the sample image data;
[0007] Get the reference data determined in the previous training round;
[0008] The optimal model parameters are determined according to the current loss value and the reference data, and a target image classification model using the optimal model parameters is obtained.
[0009] According to one aspect of the present invention, there is provided an image classification method, the method comprising:
[0010] Obtain image data to be classified;
[0011] Detect whether the target chip has a neural network processor NPU and obtain the detection result;
[0012] According to the detection results, the target image classification model is subjected to file conversion processing to obtain a model execution file on the candidate processor in the target chip; wherein the target image classification model is trained based on the image classification model training method provided by any embodiment of the present invention;
[0013] determining a target processor from the candidate processors according to the working status of the candidate processors;
[0014] Run the model execution file on the target processor, perform image classification on the image data to be classified, and obtain the target image classification result.
[0015] According to another aspect of the present invention, there is provided an image classification model training device, the device comprising:
[0016] A sample image classification result determination module is used to determine the sample image classification result of the sample image data in the current training round based on the initial image classification model;
[0017] A current loss value determination module, used to determine a current loss value of a current training round according to a sample image classification result and label image data of the sample image data;
[0018] A reference data determination module, used to obtain reference data determined in the previous training round;
[0019] The target image classification model determination module is used to determine the optimal model parameters according to the current loss value and reference data, and obtain the target image classification model using the optimal model parameters.
[0020] According to another aspect of the present invention, there is provided an image classification device, the device comprising:
[0021] The module for acquiring image data to be classified is used to acquire image data to be classified;
[0022] A detection result determination module is used to detect whether the target chip has a neural network processor NPU and obtain a detection result;
[0023] A model execution file determination module is used to perform file conversion processing on the target image classification model according to the detection result to obtain a model execution file on the candidate processor in the target chip; wherein the target image classification model is trained based on the image classification model training method provided by any embodiment of the present invention;
[0024] A target processor determination module, used to determine a target processor from the candidate processors according to the working status of the candidate processors;
[0025] The target image classification result determination module is used to run the model execution file on the target processor, perform image classification on the image data to be classified, and obtain the target image classification result.
[0026] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0027] at least one processor; and
[0028] a memory communicatively connected to at least one processor; wherein,
[0029] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the image classification model training method or image classification method of any embodiment of the present invention.
[0030] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image classification model training method or image classification method of any embodiment of the present invention when executed.
[0031] According to another aspect of the present invention, a computer program product is provided, including a computer program, which, when executed by a processor, implements the image classification model training method or the image classification method of any embodiment of the present invention.
[0032] The technical solution of the embodiment of the present invention determines the sample image classification result of the current training round for image classification of sample image data based on the initial image classification model; determines the current loss value of the current training round according to the sample image classification result and the label image data of the sample image data; obtains the reference data determined in the previous training round; determines the optimal model parameters according to the current loss value and the reference data, and obtains the target image classification model using the optimal model parameters. Compared with the method of simply comparing the training loss in each training round and determining the model parameters associated with the minimum training loss as the optimal model parameters, the above technical solution can more flexibly adjust the timing of determining the optimal model parameters during the training process of the model, so that the image classification model can better adapt to the dynamic changes of the training data, thereby improving the model classification performance, model stability and model reliability of the image classification model, and further improving the accuracy of the image classification results on the edge device. At the same time, the improved shufflenetv2 model is used as the initial image classification model, which reduces the number of parameters and model complexity of the image classification model, thereby improving the classification efficiency of completing image classification on the edge device.
[0033] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0035] Figure 1A is a flowchart of an image classification model training method provided according to Embodiment 1 of the present invention;
[0036] Figure 1B is a network structure diagram of the native shufflenetv2 model involved in an embodiment of the present invention;
[0037] Figure 2 is a flowchart of an image classification method provided according to Embodiment 2 of the present invention;
[0038] Figure 3 is a structural schematic diagram of an image classification model training device provided according to Embodiment 3 of the present invention;
[0039] Figure 4 is a structural schematic diagram of an image classification device provided according to a fourth embodiment of the present invention;
[0040] Figure 5 It is a structural schematic diagram of an electronic device for implementing the image classification model training method or the image classification method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0041] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0042] It should be noted that the terms "target", "first" and "second" in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0043] In addition, it should be noted that the collection, storage, use, processing, transmission, provision and disclosure of sample image data involved in the technical solution of the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0044] Embodiment 1
[0045] Figure 1A This is a flowchart of an image classification model training method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case where an image classification model is trained on an edge device. The method can be executed by an image classification model training device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 1A As shown, the method includes:
[0046] S101. Determine, based on an initial image classification model, a sample image classification result of image classification of sample image data in a current training round.
[0047] The initial image classification model refers to the improved shufflenetv2 model. Compared with the network structure of the original shufflenetv2 model (see Figure 1B), the present invention reduces the number of convolution modules of Stage2 in the native shufflenetv2 model from 4 to 3, thereby reducing the number of feature maps output by Stage2 from 48 to 36, reducing the number of convolution modules of Stage3 from 8 to 7, thereby reducing the number of feature maps output by Stage3 from 96 to 72, reducing the number of convolution modules of Stage4 from 4 to 3, thereby reducing the number of feature maps output by Stage4 from 192 to 144, and further reducing the number of feature maps output by Conv5 from 1024 to 576; at the same time, a multilayer perceptron (MLP) is used to replace the fully connected layer (FC) in the native shufflenetv2 model.
[0048] It is understandable that improving the native shufflenetv2 model to obtain the initial image classification model can not only ensure that the classification performance of the image classification model does not decrease significantly on edge devices with extremely limited computing power, but also reduce the number of parameters and model complexity of the image classification model.
[0049] It should be noted that Figure 1B In the above figure, BN (Batch Normalization) represents batch normalization processing, GlobalPool represents the global pooling layer, and Repeat 8 represents 8 Stage3s with the same structure. Figure 1B The other two Repeat 4s represent the meanings. In addition, for the internal structure of each module in the native ShuffleNetV2 model, the last ReLU activation function of each module is deleted to reduce feature loss and improve model accuracy; and the remaining ReLU activation function of each module is replaced by the LeakyReLU activation function to alleviate the gradient vanishing problem, especially when training deep networks, which helps to maintain the flow of gradients and improve the training efficiency and stability of image classification models.
[0050] The sample image data can be obtained from the database of the corresponding industrial enterprise according to the actual industrial business needs. For example, the sample image in the sample image data can be a picture of fruit taken during the fruit sorting process. Optionally, the sample image data can be amplified by performing geometric transformation (e.g., translation, rotation, stretching, and scaling), flipping (e.g., flipping up and down and / or flipping left and right), adding Gaussian noise, changing the illumination and contrast of the sample image, etc., to increase the data volume of the sample image data and increase the diversity of the sample image data. In the case of very little sample image data, the risk of overfitting of the model can be reduced.
[0051] Specifically, before training the initial image classification model, a size transformation operation is performed on the sample images in the sample image data. For example, the image size of each sample image in the sample image data is transformed into format to obtain the preprocessed sample image data; in the current training round, the preprocessed sample image data is subjected to image classification processing by the initial image classification model to obtain the sample image classification result.
[0052] It can be understood that before training the initial image classification model, all sample images in the sample image data are converted into sample images of the same image size, which can avoid the influence of the sample image size on the model training, thereby effectively improving the training speed of the model training and reducing the memory occupancy during model training.
[0053] It should also be noted that during the training of the initial image classification model, the parameters of all convolutional layers from Conv1 to Stage 4 of the model are frozen, and only Conv5 and MLP are trained to balance the model accuracy and model training efficiency of the image classification model.
[0054] S102: Determine the current loss value of the current training round according to the sample image classification result and the label image data of the sample image data.
[0055] Among them, the labeled image data refers to the data consisting of sample images with real classification labels. The current loss value refers to the training loss value of the model in the current training round.
[0056] Specifically, based on a preset loss function, a current loss value of a current training round is determined according to the sample image classification result and the label image data of the sample image data.
[0057] S103: Obtain reference data determined in the previous training round.
[0058] The reference data refers to the benchmark data used to determine the best model parameters; optionally, the parameter data includes the minimum loss value and the reference accuracy associated with the minimum loss value. The minimum loss value refers to the minimum training loss value during the model training process.
[0059] It should be noted that if the current training round is the first one, the minimum loss value determined in the previous training round is a preset value. For example, the minimum loss value may be positive infinity, and the reference accuracy may be 0.
[0060] S104. Determine optimal model parameters according to the current loss value and reference data, and obtain a target image classification model using the optimal model parameters.
[0061] The optimal model parameters refer to the parameters within the model that enable the classification performance of the initial image classification model to be optimal during the training of the initial image classification model.
[0062] Specifically, the current loss value is counted and retained to obtain the target loss value; based on the target loss value and the minimum loss value, the optimal model parameters in the model training process are determined.
[0063] Among them, the current loss value is counted and retained to obtain the target loss value, which can be specifically: the current loss value is rounded to three decimal places to obtain the target loss value. Compared with directly comparing the current loss value with the minimum loss value to determine the optimal model parameters, the target loss value obtained after the current loss value is counted and retained is compared with the minimum loss value to determine the optimal model parameters. The training loss obtained in the later training rounds can be retained as much as possible, so that the optimal model parameters finally determined at the end of the model training are as global optimal as possible, rather than local optimal.
[0064] Among them, the optimal model parameters in the model training process are determined according to the target loss value and the minimum loss value. Specifically, if the target loss value is less than or equal to the minimum loss value, the model parameters associated with the current loss value are used as the optimal model parameters; otherwise, the first absolute loss difference between the target loss value and the minimum loss value is calculated; if the first absolute loss difference is less than or equal to the first threshold, and the current accuracy associated with the current loss value is greater than or equal to the reference accuracy, the model parameters associated with the current loss value are used as the optimal model parameters.
[0065] The first absolute loss difference refers to the absolute value of the loss difference between the current loss value and the minimum loss value. The first threshold refers to the threshold used to compare the absolute loss difference between the current loss value and the minimum loss value; the first threshold can be preset based on the experience of those skilled in the art, for example, the first threshold can be 0.03, and the embodiment of the present invention does not specifically limit it.
[0066] More specifically, if the target loss value is less than or equal to the minimum loss value, the model parameters associated with the current loss value are used as the optimal model parameters; at the same time, the minimum loss value is updated to the current loss value, and the reference accuracy is updated to the current accuracy associated with the current loss value. If the target loss value is greater than the minimum loss value, the loss difference between the target loss value and the minimum loss value is calculated; the loss difference is processed using the absolute value function to obtain the first absolute loss difference; if the first absolute loss difference is less than or equal to the first threshold, and the current accuracy associated with the current loss value is greater than or equal to the reference accuracy, the model parameters associated with the current loss value are used as the optimal model parameters. At the same time, the average value between the current loss value and the minimum loss value is used as the new minimum loss value, and the average value between the current accuracy and the reference accuracy is used as the new reference accuracy.
[0067] It should be noted that regardless of whether the optimal model parameters have been determined in the current training round, if the training round has not reached the preset total training round threshold and the count value of the counter in the current training round has not reached the preset early stopping number threshold, the next model training will continue until the count value of the counter reaches the preset early stopping number threshold or the training round reaches the preset total training round threshold, then the model training will be stopped.
[0068] The technical solution of the embodiment of the present invention determines the sample image classification result of the current training round for image classification of sample image data based on the initial image classification model; determines the current loss value of the current training round according to the sample image classification result and the label image data of the sample image data; obtains the reference data determined in the previous training round; determines the optimal model parameters according to the current loss value and the reference data, and obtains the target image classification model using the optimal model parameters. Compared with the method of simply comparing the training loss in each training round and determining the model parameters associated with the minimum training loss as the optimal model parameters, the above technical solution can more flexibly adjust the timing of determining the optimal model parameters during the training process of the model, so that the image classification model can better adapt to the dynamic changes of the training data, thereby improving the model classification performance, model stability and model reliability of the image classification model, and further improving the accuracy of the image classification results on the edge device. At the same time, the improved shufflenetv2 model is used as the initial image classification model, which reduces the number of parameters and model complexity of the image classification model, thereby improving the classification efficiency of completing image classification on the edge device.
[0069] On the basis of the above embodiment, as an optional method of the embodiment of the present invention, the previous loss value of the previous training round can also be obtained; the second absolute loss difference between the current loss value and the previous loss value is calculated; if the second absolute loss difference is greater than the second threshold, the count value of the counter is reset to re-count the rounds that make the model training tend to stabilize; otherwise, the count value is updated; the count value is compared with the third threshold, and if the count value is greater than or equal to the third threshold, the model training is stopped.
[0070] Among them, the second absolute loss difference refers to the absolute value of the difference between the current loss value and the previous loss value. The second threshold refers to the threshold used to determine whether the current loss value changes significantly; optionally, the second threshold can be pre-set based on the experience of technical personnel in this field. For example, the second threshold can be 0.01, and the embodiment of the present invention does not specifically limit it. The third threshold refers to the maximum number of training rounds that allow the training loss to change continuously within a preset range without significant change. It should be noted that, initially, the count value of the counter is 0.
[0071] Specifically, if the current training round is the first one, there is no previous loss value of the previous training round, and the previous loss value is directly set to positive infinity; if the current training round is the nth time, n is a positive integer, n is greater than 1 and n is less than or equal to the total training round threshold, then the previous loss value of the previous training round is obtained; the second absolute loss difference between the current loss value and the previous loss value is calculated; if the second absolute loss difference is greater than the second threshold, the count value of the counter is reset to 0 to re-count the rounds that make the model training tend to be stable; otherwise, the count value of the counter is increased by 1; the count value is compared with the third threshold, if the count value is greater than or equal to the third threshold, the model training is stopped; if the count value is less than the third threshold, the previous loss value is updated to the current loss value, and the next round of model training continues.
[0072] Among them, the total training round threshold refers to the threshold that limits the maximum value of theoretically possible training rounds; optionally, the total training round threshold can be pre-set based on the experience of technical personnel in this field, for example, the total training round threshold can be 100, and the embodiment of the present invention does not specifically limit it.
[0073] It can be understood that compared with the traditional model training early stopping algorithm, the training process of the image classification model is adaptively adjusted through a counter according to the dynamic changes of the loss value during the training of the image classification model, which reduces the risk of overfitting and improves the model training efficiency of the image classification model. While speeding up the model training speed of the image classification model, the training loss of the image classification model is always kept in a relatively stable state of change, thereby improving the stability of the model training.
[0074] Embodiment 2
[0075] Figure 2 This is a flowchart of an image classification method provided in the second embodiment of the present invention. This embodiment is applicable to the case of fast image classification of extremely small-scale image data on edge devices. The method can be executed by an image classification device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 2 As shown, the method includes:
[0076] S201, obtaining image data to be classified.
[0077] The image data to be classified refers to image data that needs to be classified.
[0078] Specifically, in response to a received image classification requirement, the image data to be classified may be acquired through an image acquisition device.
[0079] Exemplarily, in response to the received fruit sorting demand, the fruit sorting console may acquire the image data to be classified corresponding to the fruit to be sorted through an image acquisition device installed at a fixed position.
[0080] S202, detect whether the target chip has a neural network processor NPU and obtain a detection result.
[0081] The target chip refers to the chip in the edge device to be used for image classification. For example, the target chip can be a Rockchip RK series chip. Optionally, the target chip includes at least one CPU. It should be noted that the neural network processing unit (NPU) is an embedded neural network processor that can simulate the structure and working principle of human brain neurons. It should be noted that the number of NPUs in the target chip is determined according to the specific model of the target chip.
[0082] Specifically, it is detected whether the target chip has an NPU; if so, the detection result is yes; if not, the detection result is no.
[0083] S203. According to the detection result, the target image classification model is converted into a file to obtain a model execution file on the candidate processor in the target chip.
[0084] The target image classification model is trained based on the image classification model training method provided by any embodiment of the present invention. The candidate processor refers to a processor that can be selected on the target chip. For example, if the detection result is yes, the candidate processor includes at least one NPU and a CPU; if the detection result is no, the candidate processor is a CPU. The model execution file refers to a file that stores data related to the target image classification model.
[0085] Specifically, the target image classification model can be converted into an ONNX (Open Neural Network Exchange) model file and stored in a preset storage unit; then, if the detection result is no, the ONNX model file is loaded from the preset storage unit through a preset reading program, and when it is detected that the file is loaded successfully, the NCNN Convolutional Neural Network (NCNN) framework is used to convert the ONNX model file into an NCNN model execution file on the CPU in the target chip.
[0086] Among them, the ONNX model file saves the structure of the target image classification model and the corresponding model parameters, which can be used for subsequent model deployment. The NCNN framework is a lightweight, high-performance deep learning framework that provides a flexible model conversion module that can convert models of mainstream deep learning frameworks (such as Caffe, TensorFlow, and ONNX) into the NCNN model format.
[0087] Among them, the model conversion module specifically includes a graph fusion submodule and an operator mapping submodule; the graph fusion module can simplify the ONNX model file and fuse the operator nodes in the ONNX model file; the operator mapping module can map the simplified ONNX model operator nodes one by one to the NCNN model operator nodes, and multiple NCNN model operator nodes are connected according to the connection order between the ONNX model operator nodes to obtain a complete NCNN model execution file. The NCNN model execution file can include a model structure file and a model parameter file, which is used to run the target image classification model on the CPU in the target chip.
[0088] If the detection result is yes, the ONNX model file is loaded from the preset storage unit through the preset reading program. When the file is successfully loaded, on the one hand, the NCNN framework is used to convert the ONNX model file into an NCNN model execution file on the CPU in the target chip; on the other hand, a number of sample images (for example, 5-10) are randomly extracted from each image category from the sample images used in the training process of the target image classification model to obtain calibration image data; according to the corporate logo of the target chip manufacturer, the target model conversion module is determined; for example, if the target chip is a Rockchip RK series chip, the target model conversion module is an RKNN (Rockchip Neural Network Processing Unit) model conversion module; the target model conversion module is used to process the ONNX model file and the calibration image data to obtain the model execution file on the NPU in the target chip, for example, the RKNN model conversion module is used to process the ONNX model file and the calibration image data to obtain the RKNN model execution file on the NPU in the target chip. Among them, the RKNN model execution file may include a model structure file and a model parameter file, which is used to run the target image classification model on the NPU in the target chip.
[0089] Among them, the RKNN model conversion module is used to process the ONNX model file and the calibration image data to obtain the RKNN model execution file, which can be specifically: the ONNX model operator nodes in the ONNX model file are mapped one by one to the RKNN model operator nodes, and multiple RKNN model operator nodes are connected according to the connection order between the ONNX model operator nodes to obtain a complete intermediate RKNN model file. Afterwards, the intermediate RKNN model file is calibrated by the calibration image data to obtain a low-precision RKNN model execution file.
[0090] S204: Determine a target processor from the candidate processors according to the working status of the candidate processors.
[0091] The target processor refers to a processor used to process image data to be classified.
[0092] Specifically, if the detection result is no, the target processor is the CPU; if the detection result is yes, the candidate processors in the target chip include at least one NPU and a CPU; according to the working status of each candidate processor, the processor at the head of the queue is obtained from the preset processor idle queue as the target processor.
[0093] S205 , running the model execution file on the target processor, performing image classification on the image data to be classified, and obtaining a target image classification result.
[0094] Specifically, if the target processor is a CPU, the NCNN model execution file on the CPU is run to perform image classification on the image data to be classified, and obtain the target image classification result; if the target processor is a certain NPU, the model execution file on the NPU is run to perform image classification on the image data to be classified, and obtain the target image classification result.
[0095] The technical solution of the embodiment of the present invention obtains image data to be classified; detects whether the target chip has a neural network processor NPU to obtain a detection result; according to the detection result, performs file conversion processing on the target image classification model to obtain a model execution file on the candidate processor in the target chip; wherein the target image classification model is trained based on the image classification model training method provided by any embodiment of the present invention; determines the target processor from the candidate processor according to the working state of the candidate processor; runs the model execution file on the target processor, performs image classification on the image data to be classified, and obtains the target image classification result. The above technical solution decouples the input of the image data to be classified, the model reasoning of the image classification model, and the acquisition of the model classification result, so that there is no strict execution order between them, but by judging whether the target chip has an NPU and judging the working state of the candidate processor in the target chip, a suitable processor is selected to process the image data to be classified, which greatly improves the utilization rate of the device hardware when the edge device classifies the image, and improves the classification efficiency of the image classification on the edge device and the accuracy of the image classification result. At the same time, image classification of extremely small-scale image data can be quickly completed on different types of edge devices, which improves the flexibility and adaptability of completing image classification on edge devices.
[0096] Embodiment 3
[0097] Figure 3 This is a schematic diagram of the structure of an image classification model training device provided in Embodiment 3 of the present invention. This embodiment is applicable to the case of training an image classification model on an edge device. The device can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 3 As shown, the device comprises:
[0098] The sample image classification result determination module 301 is used to determine the sample image classification result of the sample image data in the current training round based on the initial image classification model;
[0099] A current loss value determination module 302, used to determine a current loss value of a current training round according to the sample image classification result and the label image data of the sample image data;
[0100] A reference data acquisition module 303 is used to acquire reference data determined in the previous training round;
[0101] The target image classification model determination module 304 is used to determine the optimal model parameters according to the current loss value and the reference data, and obtain the target image classification model using the optimal model parameters.
[0102] The technical solution of the embodiment of the present invention determines the sample image classification result of the current training round for image classification of sample image data based on the initial image classification model; determines the current loss value of the current training round according to the sample image classification result and the label image data of the sample image data; obtains the reference data determined in the previous training round; determines the optimal model parameters according to the current loss value and the reference data, and obtains the target image classification model using the optimal model parameters. Compared with the method of simply comparing the training loss in each training round and determining the model parameters associated with the minimum training loss as the optimal model parameters, the above technical solution can more flexibly adjust the timing of determining the optimal model parameters during the training process of the model, so that the image classification model can better adapt to the dynamic changes of the training data, thereby improving the model classification performance, model stability and model reliability of the image classification model, and further improving the accuracy of the image classification results on the edge device. At the same time, the improved shufflenetv2 model is used as the initial image classification model, which reduces the number of parameters and model complexity of the image classification model, thereby improving the classification efficiency of completing image classification on the edge device.
[0103] Optionally, the reference data includes a minimum loss value and a reference accuracy associated with the minimum loss value.
[0104] Optionally, the target image classification model determination module 304 includes:
[0105] A target loss value determination unit, used for performing a counting and retention process on the current loss value to obtain a target loss value;
[0106] The optimal model parameter determination unit is used to determine the optimal model parameters according to the target loss value and the minimum loss value.
[0107] Optionally, an optimal model parameter determination unit is specifically used to:
[0108] If the target loss value is less than or equal to the minimum loss value, the model parameters associated with the current loss value are taken as the optimal model parameters;
[0109] Otherwise, the first absolute loss difference between the target loss value and the minimum loss value is calculated;
[0110] If the first absolute loss difference is less than or equal to the first threshold, and the current accuracy associated with the current loss value is greater than or equal to the reference accuracy, the model parameters associated with the current loss value are used as the optimal model parameters.
[0111] Optionally, the device further includes a model training control module, wherein the model training control module is specifically used to:
[0112] Get the last loss value of the last training round;
[0113] Calculate the second absolute loss difference between the current loss value and the previous loss value;
[0114] If the second absolute loss difference is greater than the second threshold, the count value of the counter is reset to re-count the number of rounds that make the model training tend to be stable;
[0115] Otherwise, update the count value;
[0116] The count value is compared with the third threshold value, and if the count value is greater than or equal to the third threshold value, the model training is stopped.
[0117] The image classification model training device provided in the embodiment of the present invention can execute the image classification model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing each image classification model training method.
[0118] Embodiment 4
[0119] Figure 4 This is a schematic diagram of the structure of an image classification device provided in the fourth embodiment of the present invention. This embodiment can be applied to the situation where extremely small-scale image data is quickly classified on an edge device. The device can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 4 As shown, the device comprises:
[0120] The image data to be classified acquisition module 401 is used to acquire the image data to be classified;
[0121] The detection result determination module 402 is used to detect whether the target chip has a neural network processor NPU and obtain a detection result;
[0122] The model execution file determination module 403 is used to perform file conversion processing on the target image classification model according to the detection result to obtain the model execution file on the candidate processor in the target chip; wherein the target image classification model is trained based on the image classification model training method provided by any embodiment of the present invention;
[0123] The target processor determination module 404 is used to determine the target processor from the candidate processors according to the working status of the candidate processors;
[0124] The target image classification result determination module 405 is used to run the model execution file on the target processor, perform image classification on the image data to be classified, and obtain the target image classification result.
[0125] The technical solution of the embodiment of the present invention obtains image data to be classified; detects whether the target chip has a neural network processor NPU to obtain a detection result; according to the detection result, performs file conversion processing on the target image classification model to obtain a model execution file on the candidate processor in the target chip; wherein the target image classification model is trained based on the image classification model training method provided by any embodiment of the present invention; determines the target processor from the candidate processor according to the working state of the candidate processor; runs the model execution file on the target processor, performs image classification on the image data to be classified, and obtains the target image classification result. The above technical solution decouples the input of the image data to be classified, the model reasoning of the image classification model, and the acquisition of the model classification result, so that there is no strict execution order between them, but by judging whether the target chip has an NPU and judging the working state of the candidate processor in the target chip, a suitable processor is selected to process the image data to be classified, which greatly improves the utilization rate of the device hardware when the edge device classifies the image, and improves the classification efficiency of the image classification on the edge device and the accuracy of the image classification result. At the same time, image classification of extremely small-scale image data can be quickly completed on different types of edge devices, which improves the flexibility and adaptability of completing image classification on edge devices.
[0126] The image classification device provided in the embodiment of the present invention can execute the image classification method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing each image classification method.
[0127] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium and a computer program product.
[0128] Embodiment 5
[0129] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0130] like Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11 in communication, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, ROM 12 and RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0131] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0132] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as an image classification model training method or an image classification method.
[0133] In some embodiments, the image classification model training method or the image classification method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image classification model training method or the image classification method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the image classification model training method or the image classification method in any other appropriate manner (e.g., by means of firmware).
[0134] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0135] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0136] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0137] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0138] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0139] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0140] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0141] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for training an image classification model, characterized in that: include: Based on the initial image classification model, determine the sample image classification results of the current training round for the sample image data; wherein the sample image data is a small-scale image data with only 5 to 10 sample images per category; the initial image classification model is obtained by improving the native shufflenetv2 model as follows: The number of convolutional modules of Stage 2 in the native shufflenetv2 model is reduced from 4 to 3, the number of convolutional modules of Stage 3 is reduced from 8 to 7, and the number of convolutional modules of Stage 4 is reduced from 4 to 3; A multi-layer perceptron MLP is used to replace the fully connected layer FC in the native shufflenetv2 model; Delete the last ReLU activation function in the internal structure of each module in the native shufflenetv2 model, and replace the remaining ReLU activation functions in the internal structure of each module with the LeakyReLU activation function; Determining a current loss value of the current training round according to the sample image classification result and the label image data of the sample image data; Obtain reference data determined in the previous training round; the reference data includes a minimum loss value and a reference accuracy rate associated with the minimum loss value; the minimum loss value refers to the minimum training loss value during the model training process; Determine optimal model parameters according to the current loss value and the reference data, and obtain a target image classification model using the optimal model parameters; Obtain the last loss value of the last training round; Calculating a second absolute loss difference between the current loss value and the previous loss value; If the second absolute loss difference is greater than a second threshold, resetting the count value of the counter to re-count the number of rounds that make the model training tend to be stable; Otherwise, updating the count value; Comparing the count value with a third threshold, and stopping model training if the count value is greater than or equal to the third threshold; Wherein, determining the optimal model parameters according to the current loss value and the reference data includes: Performing counting and retention processing on the current loss value to obtain a target loss value; If the target loss value is less than or equal to the minimum loss value, the model parameters associated with the current loss value are used as the optimal model parameters, the minimum loss value is updated to the current loss value, and the reference accuracy is updated to the current accuracy associated with the current loss value; Otherwise, calculating a first absolute loss difference between the target loss value and the minimum loss value; If the first absolute loss difference is less than or equal to the first threshold, and the current accuracy associated with the current loss value is greater than or equal to the reference accuracy, the model parameters associated with the current loss value are used as the optimal model parameters, and the average value between the current loss value and the minimum loss value is used as the new minimum loss value, and the average value between the current accuracy and the reference accuracy is used as the new reference accuracy.
2. An image classification method, characterized in that: include: Obtain image data to be classified; Detect whether the target chip has a neural network processor NPU and obtain the detection result; According to the detection result, the target image classification model is subjected to file conversion processing to obtain a model execution file on the candidate processor in the target chip, including: for each candidate processor in the target chip, converting the target image classification model into a model execution file on the candidate processor; wherein the target image classification model is trained based on the image classification model training method according to claim 1; Determining a target processor from the candidate processors according to the working status of the candidate processors; The model execution file on the target processor is run to perform image classification on the image data to be classified to obtain a target image classification result.
3. An image classification model training device, characterized in that: include: A sample image classification result determination module is used to determine the sample image classification result of the sample image data for image classification in the current training round based on the initial image classification model; wherein the sample image data is small-scale image data with only 5 to 10 sample images per category; the initial image classification model is obtained by improving the native shufflenetv2 model as follows: reducing the number of convolution modules of Stage2 in the native shufflenetv2 model from 4 to 3, reducing the number of convolution modules of Stage3 from 8 to 7, and reducing the number of convolution modules of Stage4 from 4 to 3; using a multi-layer perceptron MLP to replace the fully connected layer FC in the native shufflenetv2 model; deleting the last ReLU activation function in the internal structure of each module in the native shufflenetv2 model, and replacing the remaining ReLU activation functions in the internal structure of each module with a LeakyReLU activation function; A current loss value determination module, used to determine the current loss value of the current training round according to the sample image classification result and the label image data of the sample image data; A reference data acquisition module, used to acquire reference data determined in the previous training round; the reference data includes a minimum loss value and a reference accuracy rate associated with the minimum loss value; the minimum loss value refers to the minimum training loss value during the model training process; A target image classification model determination module, used to determine optimal model parameters according to the current loss value and the reference data, and obtain a target image classification model using the optimal model parameters; A model training control module, used to obtain the last loss value of the last training round; calculate the second absolute loss difference between the current loss value and the last loss value; if the second absolute loss difference is greater than a second threshold, reset the count value of the counter to re-count the rounds that make the model training tend to be stable; otherwise, update the count value; compare the count value with a third threshold, and if the count value is greater than or equal to the third threshold, stop model training; Wherein, the target image classification model determination module includes: a target loss value determination unit, configured to perform a counting and retention process on the current loss value to obtain a target loss value; The optimal model parameter determination unit is used to, if the target loss value is less than or equal to the minimum loss value, use the model parameters associated with the current loss value as the optimal model parameters, and update the minimum loss value to the current loss value, and update the reference accuracy to the current accuracy associated with the current loss value; otherwise, calculate the first absolute loss difference between the target loss value and the minimum loss value; if the first absolute loss difference is less than or equal to a first threshold, and the current accuracy associated with the current loss value is greater than or equal to the reference accuracy, use the model parameters associated with the current loss value as the optimal model parameters, and use the average value between the current loss value and the minimum loss value as the new minimum loss value, and use the average value between the current accuracy and the reference accuracy as the new reference accuracy.
4. An image classification device, characterized in that: include: The module for acquiring image data to be classified is used to acquire image data to be classified; A detection result determination module is used to detect whether the target chip has a neural network processor NPU and obtain a detection result; A model execution file determination module, used to perform file conversion processing on the target image classification model according to the detection result to obtain a model execution file on the candidate processor in the target chip, including: for each candidate processor in the target chip, converting the target image classification model into a model execution file on the candidate processor; wherein the target image classification model is trained based on the image classification model training method according to claim 1; A target processor determination module, used to determine a target processor from the candidate processors according to the working status of the candidate processors; The target image classification result determination module is used to run the model execution file on the target processor, perform image classification on the image data to be classified, and obtain the target image classification result.
5. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image classification model training method described in claim 1 or the image classification method described in claim 2.
6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image classification model training method described in claim 1 or the image classification method described in claim 2 when executed.
Citation Information
Patent Citations
Classification model training and image classification methods and related devices
CN113052246A
Aging detection method, device and equipment and storage medium
CN117854129A
