Image recognition network training method, image recognition method and device

Abstract

The present disclosure relates to a training method, an image recognition method, and an image recognition device for an image recognition network. The training method includes: configuring an importance weight for each convolution kernel of each convolution layer of the image recognition network; weighting the image feature vectors of image samples extracted by each convolution kernel of each convolution layer according to the configured importance weights to obtain a weighted image feature map; using the weighted image feature map to adjust network parameters and importance weights of the image recognition network; cropping the convolution kernels of the image recognition network according to the adjusted importance weights, and removing the importance weights of the convolution kernels of the image recognition network after the cropping, to obtain a cropped image recognition network; and training the cropped image recognition network using image samples.

Description

Technical Field

[0001] The present disclosure relates to the field of image recognition technology, and in particular to an image recognition network training method, an image recognition method, and an image recognition device. Background Art

[0002] To solve increasingly complex and diverse tasks, deep neural networks used for image recognition are becoming increasingly broad and deep. With the widespread adoption of smartphones, more and more image recognition networks need to be deployed on mobile devices. However, mobile phones have limited computing power and storage space, making it difficult to meet the computational demands of network models with increasingly large parameters. This results in slower response times and increased power consumption during inference.

[0003] To address this issue, model compression techniques have been introduced to reduce model size while maintaining reasonable accuracy. Deep neural networks are often over-parameterized, making network pruning, as a model compression technique, promising a way to remove redundant weights while minimizing model accuracy, thereby reducing on-device memory and computational overhead. Traditional pruning methods involve three stages: pre-training a large model, pruning "unimportant" weights from the large model according to pre-defined criteria, and fine-tuning the pruned model. Therefore, defining the pruning criteria is crucial. However, most current methods use heuristic criteria designed based on human experience. These criteria fail to accurately reflect the importance of convolutional kernels, resulting in the pruning of important weights, which impacts model performance, while retaining unimportant weights, failing to fully reduce the model size and resulting in poor pruning results. Summary of the Invention

[0004] The present disclosure provides an image recognition network training method, an image recognition method, and an apparatus to at least solve the problem of poor pruning effect of image recognition networks in related technologies, but may not solve any of the above problems.

[0005] According to a first aspect of the present disclosure, a training method for an image recognition network is provided, the training method comprising: configuring an importance weight for each convolution kernel of each convolution layer of the image recognition network; weighting the image feature vectors of the image samples extracted by each convolution kernel of each convolution layer according to the configured importance weights to obtain a weighted image feature map; using the weighted image feature map to adjust the network parameters and the importance weights of the image recognition network; according to the adjusted importance weights, cropping the convolution kernels of the image recognition network, and removing the importance weights of the convolution kernels of the image recognition network after the cropping, to obtain the cropped image recognition network; and training the cropped image recognition network using the image samples.

[0006] Optionally, the image sample includes a first image sample and a second image sample, and the use of the weighted image feature map to adjust the network parameters and the importance weight of the image recognition network includes: keeping the importance weight unchanged, and using the weighted image feature map corresponding to the first image sample to adjust the network parameters to perform a first stage training on the image recognition network; keeping the network parameters unchanged, and using the weighted image feature map corresponding to the second image sample to adjust the importance weight to perform a second stage training on the image recognition network; and looping through the first stage training and the second stage training until a stop condition is met.

[0007] Optionally, using the weighted image feature map to adjust the network parameters and the importance weights of the image recognition network includes: inputting the weighted image feature map into the output layer of the image recognition network to obtain an estimated category probability; determining a training loss value of the image recognition network based on the estimated category probability and the category label of the image sample; and adjusting the network parameters and the importance weights based on the training loss value.

[0008] Optionally, the cropping of the convolution kernels of the image recognition network according to the adjusted importance weight includes: for each convolution layer, removing a set proportion of convolution kernels, wherein the importance weight of the removed convolution kernels is less than the importance weight of the retained convolution kernels; or for each convolution layer, removing the convolution kernels whose importance weight is less than a set threshold.

[0009] Optionally, after configuring importance weights for each convolution kernel of each convolution layer of the image recognition network, the training method further includes: normalizing the importance weights corresponding to the convolution kernels of each convolution layer to obtain normalized importance weights; weighting the image feature vectors of the image samples extracted by each convolution kernel of each convolution layer according to the configured importance weights to obtain a weighted image feature map, which includes: weighting the image feature vectors of the image samples extracted by each convolution kernel of each convolution layer according to the normalized importance weights to obtain a weighted image feature map.

[0010] Optionally, the importance weights corresponding to each convolution kernel of each convolution layer are normalized to obtain the normalized importance weights, including: determining the exponential value of each importance weight according to the importance weights corresponding to each convolution kernel of the convolution layer; summing the exponential values of each importance weight to obtain an exponential sum value; determining the ratio of the exponential value of the importance weight to the exponential sum value to obtain the normalized importance weight.

[0011] According to a second aspect of the present disclosure, an image recognition method is provided, comprising: acquiring image data to be recognized; inputting the image data into an image recognition network to obtain an estimated recognition probability, wherein the image recognition network is trained using the above-mentioned training method.

[0012] According to a third aspect of the present disclosure, a training device for an image recognition network is provided, the training device comprising: a configuration unit, configured to: configure an importance weight for each convolution kernel of each convolution layer of the image recognition network; an extraction unit, configured to: weight the image feature vectors of the image samples extracted by each convolution kernel of each convolution layer according to the configured importance weights to obtain a weighted image feature map; an adjustment unit, configured to: use the weighted image feature map to adjust the network parameters and the importance weights of the image recognition network; a cropping unit, configured to: crop the convolution kernels of the image recognition network according to the adjusted importance weights, and remove the importance weights of the convolution kernels of the image recognition network after the cropping, to obtain the cropped image recognition network; and a training unit, configured to: train the cropped image recognition network using the image samples.

[0013] Optionally, the image sample includes a first image sample and a second image sample, and the adjustment unit is further configured to: keep the importance weight unchanged, and use the weighted image feature map corresponding to the first image sample to adjust the network parameters to perform a first stage training on the image recognition network; keep the network parameters unchanged, and use the weighted image feature map corresponding to the second image sample to adjust the importance weight to perform a second stage training on the image recognition network; and loop through the first stage training and the second stage training until a stop condition is met.

[0014] Optionally, the adjustment unit is further configured to: input the weighted image feature map into the output layer of the image recognition network to obtain an estimated category probability; determine a training loss value of the image recognition network based on the estimated category probability and the category label of the image sample; and adjust the network parameters and the importance weights based on the training loss value.

[0015] Optionally, the cropping unit is further configured to: for each convolution layer, remove a set proportion of convolution kernels, wherein the importance weight of the removed convolution kernels is less than the importance weight of the retained convolution kernels; or for each convolution layer, remove the convolution kernels whose importance weight is less than a set threshold.

[0016] Optionally, the training device also includes a preprocessing unit, which is configured to: normalize the importance weights corresponding to each convolution kernel of each convolution layer to obtain normalized importance weights; the extraction unit is also configured to: weight the image feature vectors of the image samples extracted by each convolution kernel of each convolution layer according to the normalized importance weights to obtain a weighted image feature map.

[0017] Optionally, the preprocessing unit is further configured to: determine the exponential value of each importance weight according to the importance weight corresponding to each convolution kernel of the convolution layer; sum the exponential values of each importance weight to obtain an exponential sum value; determine the ratio of the exponential value of the importance weight to the exponential sum value to obtain the normalized importance weight.

[0018] According to a fourth aspect of the present disclosure, an image recognition device is provided, comprising: an acquisition unit configured to acquire image data to be recognized; and an estimation unit configured to input the image data into an image recognition network to obtain an estimated recognition probability, wherein the image recognition network is trained using the above-mentioned training method.

[0019] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the image recognition network training method or image recognition method according to the present disclosure.

[0020] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the image recognition network training method or image recognition method according to the present disclosure.

[0021] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, implement the image recognition network training method or image recognition method according to the present disclosure.

[0022] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0023] According to the image recognition network training method and apparatus and image recognition method and apparatus of the embodiments of the present disclosure, by configuring an importance weight for each convolution kernel, the probability of the convolution kernel being retained can be represented by a continuous random variable. At the same time, by associating the output of each convolution layer with the importance weight, the dependencies between the convolution kernels of adjacent layers can be fully considered. Therefore, the image recognition network training method and apparatus of the embodiments of the present disclosure can accurately characterize the importance of the convolution kernel, improve pruning efficiency, and obtain an image recognition network with reduced size and reliable performance, thereby ensuring the adaptability of the trained image recognition network to mobile terminals, helping to maintain or even shorten the response time during the inference process and maintain or even reduce power consumption.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0026] Figure 1 is a flowchart illustrating a training method of an image recognition network according to an exemplary embodiment of the present disclosure.

[0027] Figure 2 4 is a flowchart illustrating a method for training an image recognition network according to a specific embodiment of the present disclosure.

[0028] Figure 3 is a flowchart illustrating an image recognition method according to an exemplary embodiment of the present disclosure.

[0029] Figure 4 is a block diagram illustrating a training apparatus for an image recognition network according to an exemplary embodiment of the present disclosure.

[0030] Figure 5 is a block diagram illustrating an image recognition apparatus according to an exemplary embodiment of the present disclosure.

[0031] Figure 6 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0033] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0034] It should be noted that the phrase "at least one of the items" in this disclosure includes three types of parallel situations: "any one of the items", "a combination of any multiple items of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step 1 and step 2" includes the following three parallel situations: (1) performing step 1; (2) performing step 2; and (3) performing steps 1 and 2.

[0035] Image recognition networks often employ deep neural networks, particularly convolutional neural networks. A convolutional neural network consists of an input layer that performs preliminary image processing, a convolutional layer that extracts image features, and an output layer that generates recognition results based on the extracted image features. These layers are connected in a cascade, with the convolutional layer hidden in the middle. This often involves multiple layers (i.e., multiple convolutional layers). Each convolutional layer contains multiple convolution kernels, each with multiple channels, each configured with weights and biases required for the convolution operation. The convolution kernels in the first convolutional layer use the configured weights and biases to convolve with the image vector output by the input layer, generating a preliminary one-dimensional or two-dimensional image feature vector. The image feature vectors extracted by these convolution kernels are combined to form a two-dimensional or three-dimensional image feature map (essentially still a multidimensional vector). The length of the vector's first dimension represents the number of convolution kernels in that convolutional layer, and therefore the number of channels, reflecting the depth of the convolutional layer. The image feature map output by the first convolutional layer is input to the next convolutional layer, and a new image feature map is obtained after similar convolution and aggregation. This is passed layer by layer, and the output of the last convolutional layer is the final extracted image feature map, which is input into the output layer as the image feature. In simple terms, taking the image classification task as an example, the input is image X, and the output is the probability p of each category. The input image X is extracted through the neural network into a hierarchical feature map {X1, X2, ..., X l}, where X i is the feature map of the i-th layer.i ∈R C×H×W , which is a three-dimensional vector, where C is the number of channels of the feature map, H and W represent the height and width of the feature map. Through N convolution kernels F[i]∈R C×k×k , i∈{1, 2, ...N}, output feature map X i+1 ∈R N×H×W , where k is the width and height of the convolution kernel.

[0036] It can be seen that to extract more complex and diverse image features, the number and depth of convolutional layers need to be increased, making the image recognition network increasingly wider and deeper, and the computational workload also increases significantly. With the popularity of smartphones, more and more image recognition networks need to be deployed on mobile devices. However, the computing power and storage space of mobile phones are limited, making it difficult to meet the computational requirements of network models with increasingly large parameter counts. This leads to slower response times and increased power consumption during inference.

[0037] Deep neural networks are often over-parameterized, making network pruning possible. This technique promises to remove redundant weights while minimizing model accuracy, thereby reducing on-device memory and computational overhead. Traditional pruning methods involve three stages: pre-training a large model, pruning "unimportant" weights from the large model according to pre-defined criteria, and fine-tuning the pruned model. Therefore, defining the pruning criteria is crucial. However, most current methods use heuristic criteria designed based on human experience. These criteria fail to accurately reflect the importance of convolution kernels, resulting in the pruning of important weights, impacting model performance, while retaining unimportant weights, failing to fully reduce the model size and leading to poor pruning results.

[0038] In order to improve the pruning effect and obtain an image recognition network with reduced size and reliable performance, the present disclosure proposes a training method for an image recognition network and an image recognition network obtained by applying the method. By configuring an importance weight for each convolution kernel, a continuous random variable can be used to represent the probability of the convolution kernel being retained. At the same time, by associating the output of each convolution layer with the importance weight and automatically calculating a reasonable importance weight with the help of image samples, the dependency of the convolution kernels of adjacent layers can be fully considered without relying on artificial priors. Therefore, the training method and device of the image recognition network disclosed in the present disclosure can accurately characterize the importance of the convolution kernel and improve the pruning efficiency, thereby ensuring the adaptability of the trained image recognition network to the mobile terminal, helping to maintain or even shorten the response time during the inference process and maintain or even reduce power consumption.

[0039] Below, we will refer to Figures 1 to 5 A training method and a training device for an image recognition network, and an image recognition method and an image recognition device according to exemplary embodiments of the present disclosure are described in detail.

[0040] Figure 1 is a flowchart illustrating a method for training an image recognition network according to an exemplary embodiment of the present disclosure. It should be understood that the method for training an image recognition network according to an exemplary embodiment of the present disclosure can be implemented in a terminal device such as a smartphone, a tablet computer, or a personal computer (PC), or can also be implemented in a device such as a server.

[0041] Reference Figure 1 In step 101, an importance weight is assigned to each convolution kernel of each convolution layer of the image recognition network. Since each convolution kernel in the image recognition network is used to extract different features, and in an over-parameterized network, different convolution kernels have different effects on the final recognition accuracy. By assigning an importance weight to each convolution kernel, the importance of each convolution kernel can be evaluated accordingly. The larger the importance weight, the more important the corresponding convolution kernel is, and the greater its impact on the final recognition effect. This also provides the possibility of directly pruning unimportant convolution kernels to achieve network pruning.

[0042] Optionally, after configuring the importance weights, the importance weights corresponding to the convolution kernels of each convolution layer can be normalized to obtain normalized importance weights, so as to facilitate intuitive comparison of the relative importance of different convolution kernels in the same convolution layer. Specifically, the normalization process includes: determining the exponential value of each importance weight according to the importance weights corresponding to the convolution kernels of the convolution layer; summing the exponential values of each importance weight to obtain an exponential sum value; determining the ratio of the exponential value of the importance weight to the exponential sum value to obtain the normalized importance weight. That is, selecting a base number, using each importance weight as an exponent, and calculating the corresponding exponential value can help to dig out more detailed differences between different importance weights. For each convolution layer, determining the ratio of each exponential value to the exponential sum value of the corresponding convolution layer can ensure that the sum of all ratios of the same convolution layer is 1, thereby achieving normalization. Specifically, the natural constant e can be taken as the base, that is, the softmax function is used to map the importance weight to the (0,1) interval, thereby realizing the normalization of the importance weight. At this time, the existing function can be directly called, which can simplify the program and reduce development costs. The softmax function is:

[0043]

[0044] Among them, a i is the original importance weight, b i is the importance weight after normalization, j∈{1, 2, ...N}, N is the number of convolution kernels in the corresponding convolution layer.

[0045] In step 102, the image feature vectors of the image samples extracted by each convolution kernel of each convolution layer are weighted according to the configured importance weights to obtain a weighted image feature map. Specifically, the image feature vector output by the i-th convolution kernel of a convolution layer is Y[i]∈R H×W , if the importance weight is not configured, the image feature map output by the convolution layer is Y∈R N×H×W . Combined with the normalized importance weight b i The weighted image feature map obtained by weighting is Y′∈R N×H×W , where the i-th image feature vector is Y′[i]=b i By associating the importance weights with the image feature maps output by the corresponding convolutional layer, the importance weights can influence the outputs of subsequent convolutional layers, thereby affecting the final image recognition results, fully considering the dependencies between convolution kernels in adjacent layers.

[0046] In step 103, the weighted image feature map is used to adjust the network parameters and importance weights of the image recognition network. Since the weighted image feature map is obtained by inputting image samples into an image recognition network configured with importance weights, it directly affects the recognition results. However, the characteristics of the image samples themselves that need to be recognized are known. For example, for image classification tasks, the probability of an image belonging to a certain category needs to be recognized, and the category of the image samples is already known. Therefore, after comparative adjustment, convolution kernels that have a greater impact on the recognition results will receive a larger importance weight, while convolution kernels that have a smaller impact on the recognition results will receive a smaller importance weight, allowing the adjusted importance weights to more accurately represent the importance of the convolution kernels. Furthermore, network parameters have a significant impact on the recognition results of the image recognition network. By adjusting the network parameters of the image recognition network simultaneously with the importance weights, the entire network can be pre-trained, ensuring that the adjusted importance weights more accurately reflect the importance of the corresponding convolution kernels. Specifically, the adjusted network parameters can be solely the weights within each convolution kernel, or they can also include bias terms, which are not limited in this disclosure.

[0047] Still taking the image classification task as an example, step 103 can be specifically executed as follows: inputting the weighted image feature map into the output layer of the image recognition network to obtain the estimated category probability; determining the training loss value of the image recognition network based on the estimated category probability and the category label of the image sample; adjusting the network parameters and importance weights according to the training loss value. By combining the training loss value to adjust the network parameters and importance weights, the importance weights and network parameters of each convolution kernel can be automatically learned based on data-driven, which not only reduces the workload of manual calculations, but also does not need to rely on manual priors. Among them, the training loss value is, for example, the value of the loss function. The loss function can adopt the cross-entropy loss function, and the network parameters and importance weights are updated by the gradient descent method, which helps to improve learning efficiency. The expression of the cross-entropy loss function can be specifically:

[0048]

[0049] Among them, T is the number of image samples, M is the number of categories, and y ic The value is 1 or 0, indicating the category label of image sample i. 1 means that image sample i is of category c, and 0 means that image sample i is not of category c. ic represents the probability that image sample i belongs to category c.

[0050] Optionally, step 103 can divide the image samples into a first image sample and a second image sample, that is, split the original training set into two training sets, each used to adjust the network parameters and importance weights. Specifically, step 103 includes: maintaining the importance weights unchanged, using the weighted image feature map corresponding to the first image sample to adjust the network parameters to perform the first stage training of the image recognition network; maintaining the network parameters unchanged, using the weighted image feature map corresponding to the second image sample to adjust the importance weights to perform the second stage training of the image recognition network; and looping through the first and second stages of training until a stopping condition is met. The stopping condition can be training convergence or the number of iterations reaching a set number. By using two different training sets to train and adjust the network parameters and importance weights separately, overfitting of the importance weights can be prevented, the generalization performance of the adjustment results can be improved, and misjudgment can be reduced. Furthermore, by adjusting the network parameters first, the image recognition network can be initially determined, and the importance weights can be subsequently adjusted based on this to evaluate the importance of the convolution kernel, which helps to improve adjustment efficiency. It is understandable that the training of the two stages can be carried out as described above, combining the estimated category probabilities, category labels of image samples, and loss functions. The only difference is that the specific objects adjusted in each stage are slightly different, which will not be repeated here.

[0051] In step 104, the convolution kernels of the image recognition network are cropped based on the adjusted importance weights, and the importance weights of the convolution kernels of the image recognition network after the cropping are removed, resulting in a cropped image recognition network. Specifically, the cropping operation can be performed by removing a set proportion of convolution kernels from each convolution layer, where the importance weights of the removed convolution kernels are less than the importance weights of the retained convolution kernels. In other words, a corresponding proportion of low-importance convolution kernels from a convolution layer are cropped according to a set proportion, facilitating pruning according to a clear size target. For example, the convolution kernels can be sorted in ascending order according to their corresponding importance weights, and a set proportion of convolution kernels can be removed from lowest to highest importance. For example, if the user wishes to crop 25% of the convolution kernels, the convolution kernels with the lowest importance weights in each convolution layer will be removed. The cropping operation can also be performed by removing convolution kernels with importance weights less than a set threshold from each convolution layer, facilitating flexible pruning without a clear size target, balancing network size and performance. It is understandable that, in the case of cropping convolution kernels according to a set threshold, the importance weights can be normalized, in particular, to facilitate setting a universally applicable threshold. Since the importance weights are only used to characterize the importance of each convolution kernel in the image recognition network before cropping, after cropping the convolution kernels with low importance, by removing the importance weights from the network, it can be ensured that the resulting image recognition network can be used normally. In other words, the importance weights are a temporary parameter configured for cropping convolution kernels and no longer appear in the image recognition network during actual application.

[0052] In step 105, the cropped image recognition network is trained using the image samples. By retraining the cropped image recognition network with the complete image samples, the network parameters of the image recognition network can be fine-tuned, reducing the impact of the cropped convolution kernel on the network and optimizing the network's recognition performance. Specifically, as described above, training can be performed by combining the estimated class probabilities, the class labels of the image samples, and the loss function. However, the specific adjustments are limited to the network parameters, which will not be detailed here.

[0053] Figure 2 1 is a flowchart illustrating a method for training an image recognition network according to a specific embodiment of the present disclosure, wherein the image recognition network is used to perform image classification tasks and adopts a convolutional neural network.

[0054] Reference Figure 2This training method first assigns importance weights a to the convolution kernels in the convolutional neural network, randomly divides the training image set into a first training image set and a second training image set, and then begins training on importance weight a. During training, importance weight a is first fixed, and the network weight parameter F is updated on the first training image set. Then, network weight parameter F is fixed again, and importance weight a is updated on the second training image set. After updating importance weight a, it is necessary to determine whether the current cumulative number of iterations has reached the set number. If not, the training process is repeated. If it has, training is stopped and importance weight a is fixed. Then, based on the cropping ratio entered by the user, convolution kernels with cropped kernels are removed from low to high importance weights a. The importance weights of the convolution kernels of the convolutional neural network after the cropping kernels are removed to obtain the cropped convolutional neural network. Finally, the cropped convolutional neural network is trained on the complete training image set for fine-tuning, completing the training of the network and outputting the trained convolutional neural network.

[0055] Figure 3 is a flowchart illustrating an image recognition method according to an exemplary embodiment of the present disclosure.

[0056] Reference Figure 3 , in step 301, image data to be identified is obtained.

[0057] In step 302, the image data is input into the image recognition network to obtain the estimated recognition probability. Figure 1 The training method described above is used for training, so the image recognition method of the exemplary embodiment of the present disclosure has all the beneficial technical effects of the above training method, which will not be described in detail here. Specifically, taking the image classification task as an example, the estimated recognition probability obtained in step 302 is the estimated category probability.

[0058] Figure 4 is a block diagram illustrating an apparatus for training an image recognition network according to an exemplary embodiment of the present disclosure. It should be understood that the apparatus for training an image recognition network according to an exemplary embodiment of the present disclosure can be implemented in a terminal device such as a smartphone, tablet computer, or personal computer (PC) using software, hardware, or a combination of software and hardware, or can also be implemented in a device such as a server.

[0059] Reference Figure 4 The image recognition network training device 400 includes a configuration unit 401, an extraction unit 402, an adjustment unit 403, a cropping unit 404, and a training unit 405.

[0060] The configuration unit 401 can configure an importance weight for each convolution kernel of each convolution layer of the image recognition network. Since each convolution kernel in the image recognition network is used to extract different features, and in an over-parameterized network, different convolution kernels have different effects on the final recognition accuracy. By configuring an importance weight for each convolution kernel, the importance of each convolution kernel can be evaluated accordingly. The larger the importance weight, the more important the corresponding convolution kernel is, and the greater the impact on the final recognition effect. This also provides the possibility of directly pruning unimportant convolution kernels to achieve network pruning.

[0061] Optionally, after configuring the importance weights, the configuration unit 401 may also normalize the importance weights corresponding to the convolution kernels of each convolution layer to obtain normalized importance weights, so as to intuitively compare the relative importance of different convolution kernels in the same convolution layer. Specifically, the normalization process includes: determining the exponential value of each importance weight according to the importance weights corresponding to the convolution kernels of the convolution layer; summing the exponential values of each importance weight to obtain an exponential sum value; determining the ratio of the exponential value of the importance weight to the exponential sum value to obtain the normalized importance weight. That is, selecting a base number, using each importance weight as an exponent, and calculating the corresponding exponential value can help to dig out more detailed differences between different importance weights. For each convolution layer, the ratio of each exponential value to the exponential sum value of the corresponding convolution layer is determined to ensure that the sum of all ratios of the same convolution layer is 1, thereby achieving normalization. Specifically, the natural constant e can be taken as the base, that is, the softmax function is used to map the importance weight to the (0,1) interval, so as to realize the normalization of the importance weight. At this time, the existing function can be called directly, which can simplify the program and reduce development costs.

[0062] Extraction unit 402 can weight the image feature vectors of the image samples extracted by each convolution kernel of each convolution layer according to the configured importance weights to obtain a weighted image feature map. By associating the importance weights with the image feature map output by the corresponding convolution layer, the importance weights can influence the output of subsequent convolution layers, thereby affecting the final image recognition results, fully considering the dependencies between convolution kernels in adjacent layers.

[0063] Adjustment unit 403 can use a weighted image feature map to adjust the network parameters and importance weights of the image recognition network. Since the weighted image feature map is obtained by inputting image samples into an image recognition network configured with importance weights, the weighted image feature map directly affects the recognition results. However, the characteristics of the image samples themselves that need to be recognized are known. For example, for image classification tasks, what needs to be recognized is the probability of an image belonging to a certain category, and the category of the image sample is already known. Therefore, after comparative adjustment, convolution kernels that have a greater impact on the recognition results can be given a larger importance weight, and convolution kernels that have a smaller impact on the recognition results can be given a smaller importance weight, so that the adjusted importance weights can more accurately represent the importance of the convolution kernel. At the same time, network parameters have a significant impact on the recognition results of the image recognition network. By adjusting the network parameters of the image recognition network when adjusting the importance weights, the entire network can be pre-trained at the same time, ensuring that the adjusted importance weights can more accurately reflect the importance of the corresponding convolution kernel. Specifically, the adjusted network parameters can be only the weights within each convolution kernel, or they can also include bias terms, which are not limited in this disclosure.

[0064] Still taking the image classification task as an example, the adjustment operation of the adjustment unit 403 can be specifically performed as follows: inputting the weighted image feature map into the output layer of the image recognition network to obtain the estimated category probability; determining the training loss value of the image recognition network based on the estimated category probability and the category label of the image sample; and adjusting the network parameters and importance weights based on the training loss value. By combining the training loss value to adjust the network parameters and importance weights, the importance weights and network parameters of each convolution kernel can be automatically learned based on data-driven learning, which not only reduces the workload of manual calculations but also does not require reliance on manual priors. Among them, the loss function can adopt the cross entropy loss function, and the network parameters and importance weights are updated by the gradient descent method, which helps to improve learning efficiency.

[0065] Optionally, adjustment unit 403 may further divide the image samples into first and second image samples, that is, splitting the original training set into two training sets, each used to adjust network parameters and importance weights. Specifically, adjustment unit 403 separately adjusts network parameters and importance weights by: maintaining the importance weights unchanged and adjusting network parameters using the weighted image feature map corresponding to the first image sample to perform first-stage training of the image recognition network; maintaining the network parameters unchanged and adjusting the importance weights using the weighted image feature map corresponding to the second image sample to perform second-stage training of the image recognition network; and looping through the first and second stages of training until a stopping condition is met. The stopping condition may be training convergence or a set number of iterations. By using two different training sets to separately train and adjust network parameters and importance weights, overfitting of the importance weights can be prevented, improving the generalization performance of the adjusted results and reducing misclassification. Furthermore, by first adjusting network parameters, the image recognition network can be initially determined, and then the importance weights can be adjusted based on this initial determination to assess the importance of the convolution kernel, thereby improving adjustment efficiency. It is understandable that the training of the two stages can be carried out as described above, combining the estimated category probabilities, category labels of image samples, and loss functions. The only difference is that the specific objects adjusted in each stage are slightly different, which will not be repeated here.

[0066] The cropping unit 404 can crop the convolution kernels of the image recognition network based on the adjusted importance weights, and remove the importance weights of the convolution kernels of the image recognition network after the cropping, thereby obtaining a cropped image recognition network. Specifically, the operation of cropping the convolution kernels can be performed as follows: for each convolution layer, remove a set proportion of convolution kernels, wherein the importance weights of the removed convolution kernels are less than the importance weights of the retained convolution kernels. In other words, according to a set proportion, a corresponding proportion of low-importance convolution kernels in a convolution layer are cropped, facilitating pruning according to a clear size target. For example, the convolution kernels can be sorted in ascending order according to their corresponding importance weights, and a set proportion of convolution kernels can be removed from low to high. The operation of cropping the convolution kernels can also be performed as follows: for each convolution layer, remove convolution kernels with importance weights less than a set threshold, facilitating flexible pruning without a clear size target, balancing network size and network performance. It is understood that when cropping convolution kernels according to a set threshold, the importance weights can be normalized to facilitate setting a universally applicable set threshold. Because importance weights are only used to represent the importance of each convolution kernel in the image recognition network before pruning, removing the importance weights from the network after pruning the less important kernels ensures that the resulting image recognition network will function properly. In other words, importance weights are a temporary parameter configured for kernel pruning and no longer appear in the actual image recognition network.

[0067] Training unit 405 can use the image samples to train the cropped image recognition network. By retraining the cropped image recognition network with the complete image samples, the network parameters of the image recognition network can be fine-tuned, reducing the impact of the cropped convolution kernel on the network and optimizing the network's recognition performance. Specifically, as described above, training can be performed by combining the estimated class probabilities, the class labels of the image samples, and the loss function. However, the specific adjustments are limited to the network parameters, which will not be detailed here.

[0068] Figure 5 is a block diagram showing an image recognition device according to an exemplary embodiment of the present disclosure. It should be understood that the image recognition device according to the exemplary embodiment of the present disclosure can be implemented in software, hardware, or a combination of software and hardware. For example, it can be implemented in an APP of a smartphone. Figure 5 The image recognition device.

[0069] Reference Figure 5 , the image recognition device 500 includes an acquisition unit 501 and an estimation unit 502 .

[0070] The acquisition unit 501 can acquire image data to be recognized.

[0071] The estimation unit 502 can input the image data into the image recognition network to obtain the estimated recognition probability. Figure 1 The training method described above is used for training, so the image recognition device 500 of the exemplary embodiment of the present disclosure has all the beneficial technical effects of the above training method, which will not be described in detail here. Specifically, taking the image classification task as an example, the estimated recognition probability obtained by the estimation unit 502 is the estimated category probability.

[0072] Figure 6 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

[0073] Reference Figure 6 The electronic device 600 includes at least one memory 601 and at least one processor 602, wherein the at least one memory 601 stores a set of computer-executable instructions. When the computer-executable instruction set is executed by the at least one processor 602, the image recognition network training method or the image recognition method according to the exemplary embodiment of the present disclosure is executed.

[0074] As an example, the electronic device 600 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 600 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction set) individually or in combination. The electronic device 600 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0075] In electronic device 600, processor 602 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0076] The processor 602 can execute instructions or codes stored in the memory 601, wherein the memory 601 can also store data. Instructions and data can also be sent and received over the network via the network interface device, wherein the network interface device can use any known transmission protocol.

[0077] The memory 601 may be integrated with the processor 602, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. Furthermore, the memory 601 may comprise a separate device, such as an external disk drive, a storage array, or any other storage device usable by a database system. The memory 601 and the processor 602 may be operatively coupled or may communicate with each other, for example, via an I / O port, a network connection, or the like, such that the processor 602 can access files stored in the memory.

[0078] In addition, the electronic device 600 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 600 may be connected to each other via a bus and / or a network.

[0079] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided, which, when the instructions in the computer-readable storage medium are executed by at least one processor, prompts the at least one processor to execute the training method of the image recognition network or the image recognition method according to the exemplary embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or ultra-fast digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device configured to store the computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0080] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, which includes computer instructions. When the computer instructions are executed by at least one processor, the at least one processor is prompted to perform the image recognition network training method or image recognition method according to the exemplary embodiment of the present disclosure.

[0081] According to the training method and device of the image recognition network and the image recognition method and device of the exemplary embodiments of the present disclosure, by configuring an importance weight for each convolution kernel, a continuous random variable can be used to represent the probability of the convolution kernel being retained. At the same time, the output of each convolution layer is associated with the importance weight, which can fully consider the dependency relationship between the convolution kernels of adjacent layers, and thus accurately characterize the importance of the convolution kernel, improve pruning efficiency, and obtain an image recognition network with reduced size and reliable performance, thereby ensuring the adaptability of the trained image recognition network to mobile terminals, helping to maintain or even shorten the response time during the inference process, and maintain or even reduce power consumption.

[0082] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0083] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for an image recognition network, characterized in that: The training method comprises: Configuring importance weights for each convolution kernel of each convolution layer of the image recognition network; Weighting the image feature vectors of the image samples extracted by each convolution kernel of each convolution layer according to the configured importance weights to obtain a weighted image feature map; Adjusting network parameters and the importance weights of the image recognition network using the weighted image feature map with the goal of reducing image recognition error, wherein the image recognition error is an error of an image recognition result relative to a label of the image sample, the image recognition result being obtained by inputting the weighted image feature map into an output layer of the image recognition network; According to the adjusted importance weight, cropping the convolution kernel of the image recognition network, and removing the importance weight of the convolution kernel of the image recognition network after the cropping of the convolution kernel, to obtain the cropped image recognition network; The cropped image recognition network is trained using the image samples.

2. The training method according to claim 1, wherein: The image sample includes a first image sample and a second image sample, and the using the weighted image feature map to adjust the network parameters and the importance weight of the image recognition network includes: Keeping the importance weight unchanged, adjusting the network parameters using the weighted image feature map corresponding to the first image sample to perform a first stage of training on the image recognition network; Keeping the network parameters unchanged, adjusting the importance weight using the weighted image feature map corresponding to the second image sample to perform a second stage of training on the image recognition network; The first-stage training and the second-stage training are executed cyclically until a stopping condition is met.

3. The training method according to claim 1 or 2, characterized in that: The using the weighted image feature map to adjust the network parameters and the importance weight of the image recognition network includes: Inputting the weighted image feature map into the output layer of the image recognition network to obtain estimated class probabilities; Determining a training loss value of the image recognition network based on the estimated category probabilities and the category labels of the image samples; According to the training loss value, the network parameters and the importance weights are adjusted.

4. The training method according to claim 1, wherein: The step of clipping the convolution kernel of the image recognition network according to the adjusted importance weight comprises: For each convolutional layer, remove a set proportion of convolution kernels, wherein the importance weights of the removed convolution kernels are less than the importance weights of the retained convolution kernels; or For each convolution layer, the convolution kernels whose importance weight is less than the set threshold are removed.

5. The training method according to claim 1, wherein: After configuring an importance weight for each convolution kernel of each convolution layer of the image recognition network, the training method further includes: Normalizing the importance weights corresponding to the convolution kernels of each convolution layer to obtain normalized importance weights; The step of weighting the image feature vectors of the image samples extracted by each convolution kernel of each convolution layer according to the configured importance weights to obtain a weighted image feature map includes: The image feature vectors of the image samples extracted by each convolution kernel of each convolution layer are weighted according to the normalized importance weights to obtain a weighted image feature map.

6. The training method according to claim 5, wherein: Normalizing the importance weights corresponding to the convolution kernels of each convolution layer to obtain the normalized importance weights includes: Determining an exponential value of each importance weight according to the importance weight corresponding to each convolution kernel of the convolution layer; summing the indexed values of the respective importance weights to obtain an indexed sum value; A ratio of the indexed value of the importance weight to the indexed sum value is determined to obtain a normalized importance weight.

7. An image recognition method, characterized in that: include: Acquire image data to be recognized; Input the image data into the image recognition network to obtain the estimated recognition probability, The image recognition network is trained using the training method according to any one of claims 1 to 6.

8. A training device for an image recognition network, characterized in that: The training device comprises: A configuration unit, configured to: configure an importance weight for each convolution kernel of each convolution layer of the image recognition network; an extraction unit configured to: weight the image feature vectors of the image samples extracted by each convolution kernel of each convolution layer according to the configured importance weights to obtain a weighted image feature map; an adjustment unit configured to adjust network parameters and the importance weights of the image recognition network using the weighted image feature map with the goal of reducing an image recognition error, wherein the image recognition error is an error of an image recognition result relative to a label of the image sample, and the image recognition result is obtained by inputting the weighted image feature map into an output layer of the image recognition network; a cropping unit configured to: crop the convolution kernel of the image recognition network according to the adjusted importance weight, and remove the importance weight of the convolution kernel of the image recognition network after the cropping of the convolution kernel, to obtain the cropped image recognition network; The training unit is configured to: use the image samples to train the cropped image recognition network.

9. The training device according to claim 8, characterized in that The image samples include a first image sample and a second image sample, and the adjustment unit is further configured to: Keeping the importance weight unchanged, adjusting the network parameters using the weighted image feature map corresponding to the first image sample to perform a first stage of training on the image recognition network; Keeping the network parameters unchanged, adjusting the importance weight using the weighted image feature map corresponding to the second image sample to perform a second stage of training on the image recognition network; The first-stage training and the second-stage training are executed cyclically until a stopping condition is met.

10. The training device according to claim 8 or 9, characterized in that The adjustment unit is further configured to: Inputting the weighted image feature map into the output layer of the image recognition network to obtain estimated class probabilities; Determining a training loss value of the image recognition network based on the estimated category probabilities and the category labels of the image samples; According to the training loss value, the network parameters and the importance weights are adjusted.

11. The training device according to claim 8, wherein The cropping unit is further configured to: For each convolutional layer, remove a set proportion of convolution kernels, wherein the importance weights of the removed convolution kernels are less than the importance weights of the retained convolution kernels; or For each convolution layer, the convolution kernels whose importance weight is less than the set threshold are removed.

12. The training device according to claim 8, wherein The training device further includes a preprocessing unit configured to: perform normalization processing on the importance weights corresponding to each convolution kernel of each convolution layer to obtain normalized importance weights; The extraction unit is further configured to: weight the image feature vectors of the image samples extracted by each convolution kernel of each convolution layer according to the normalized importance weights to obtain a weighted image feature map.

13. The training device according to claim 12, wherein: The pre-processing unit is further configured to: Determining an exponential value of each importance weight according to the importance weight corresponding to each convolution kernel of the convolution layer; summing the indexed values of the respective importance weights to obtain an indexed sum value; A ratio of the indexed value of the importance weight to the indexed sum value is determined to obtain a normalized importance weight.

14. An image recognition device, characterized in that: include: An acquisition unit is configured to: acquire image data to be identified; The estimation unit is configured to: input the image data into the image recognition network to obtain an estimated recognition probability, The image recognition network is trained using the training method according to any one of claims 1 to 6.

15. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer-executable instructions, When the computer-executable instructions are executed by the at least one processor, the computer-executable instructions cause the at least one processor to execute the image recognition network training method according to any one of claims 1 to 6 or the image recognition method according to claim 7.

16. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the image recognition network training method according to any one of claims 1 to 6 or the image recognition method according to claim 7.

17. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by at least one processor, the image recognition network training method according to any one of claims 1 to 6 or the image recognition method according to claim 7 is implemented.

Images