Image recognition method, device, equipment, storage medium and product thereof

Abstract

The present application discloses an image recognition method, apparatus, device, machine-readable storage medium, and computer program product, relating to the field of image recognition technology. The method comprises: identifying the object type of an object to be recognized using an image recognition neural network model; the image recognition neural network model is obtained by the following steps: determining an expected sparsity metric interval for the model based on the resources of the image recognition device; determining the current sparsity metric for model training; if the current sparsity metric is greater than the upper bound of the expected sparsity metric, increasing the weight of the first regularization term and the weight of the second regularization term in the loss function in the next round of training; if the current sparsity metric is less than the lower bound of the expected sparsity metric, decreasing the weight of the first regularization term and the weight of the second regularization term in the loss function in the next round of training; and finally completing the training to obtain the image recognition neural network model. This method can streamline the model and achieve the expected image recognition accuracy.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and specifically to an image recognition method, apparatus, device, machine-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of artificial intelligence (AI), its applications are becoming increasingly widespread, and it has been applied to a wide range of fields, including image analysis, speech recognition, text processing, intelligent recommendations, and security monitoring. In most cases, the more parameters a model has, the better its performance. However, in applications where hardware computing and storage resources are limited, such as in facial recognition equipment, intelligent driving vision systems, and electronic surveillance equipment, the hardware required to run these models may not be sufficient to support these large numbers of parameters. For example, if a model has a large number of non-zero parameters, and its complexity exceeds the hardware's computational capacity, the hardware's image recognition speed will be significantly reduced. This overloaded hardware may also lead to errors in the computation and final image recognition results. In applications such as intelligent driving and electronic surveillance, where accurate image recognition results must be delivered within a limited timeframe, how to streamline the model without significantly compromising its prediction accuracy, so that the image recognition model can achieve the expected image recognition accuracy when applied to these scenarios, remains a pressing technical challenge. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide an image recognition method, apparatus, device, machine-readable storage medium, and computer program product to solve the technical problem in the prior art of how to streamline the model without significantly reducing the prediction accuracy of the model, so that the image recognition model can provide the expected recognition accuracy when applied to the above-mentioned scenarios.

[0004] To achieve the above objectives, the present application provides, in a first aspect, an image recognition method, applied to an image recognition device, the method comprising:

[0005] Acquire an image of a target area, where the image includes an object to be identified;

[0006] Identify the object type of the object to be identified by using an image recognition neural network model;

[0007] Output image recognition results according to object type;

[0008] Among them, the image recognition neural network model is obtained through the following steps:

[0009] Obtain available computing resources and available storage resources of the image recognition device;

[0010] Based on available computing resources and available storage resources, determine the expected sparsity metric range of the image recognition neural network model. The sparsity metric is the ratio of the number of non-zero parameters to the total number of parameters in the image recognition neural network model.

[0011] Based on the training image set, determine the current sparsity metric of the pre-trained image recognition neural network model after the current round of training;

[0012] When the current sparsity metric is greater than the upper bound of the expected sparsity metric, increase the weight of the first regularization term and the second regularization term of the loss function in the next round of training;

[0013] When the current sparsity metric is less than the lower bound of the expected sparsity metric, the weight of the first regularization term and the weight of the second regularization term of the loss function in the next round of training are reduced;

[0014] When the loss function converges or the number of training rounds reaches a preset number, the training of the pre-trained image recognition neural network model is terminated to obtain an image recognition neural network model.

[0015] In an embodiment of the present application, the first regularization term includes: the logarithm of the L1 norm plus a constant term, where the constant term is greater than or equal to 1.

[0016] In this embodiment of the present application, the loss function includes:

[0017] ;

[0018] in, is the classification loss function, is the weight of the first regularization term, is the weight of the second regularization term, is the parameter vector of the pre-trained image recognition neural network model, is the L1 norm of the parameter vector, is the square of the L2 norm of the parameter vector, and the square of the L2 norm is the second regularization term.

[0019] In an embodiment of the present application, when the current sparse metric and the previous round sparse metric of the pre-trained image recognition neural network model in the previous round of training are respectively located on both sides of the sparse metric interval, the change in the first regularization term weight and the second regularization term weight is reduced; when the current sparse metric and the previous round sparse metric are located on the same side of the sparse metric interval, the change in the first regularization term weight and the second regularization term weight is increased.

[0020] In an embodiment of the present application, the step of obtaining an image recognition neural network model also includes: after completing the training of the pre-trained image recognition neural network model, obtaining a trained image recognition neural network model; and trimming the zero-value parameters in the trained image recognition neural network model to obtain an image recognition neural network model.

[0021] In an embodiment of the present application, the step of obtaining an image recognition neural network model further includes: after trimming the zero-value parameters in the trained image recognition neural network model, determining the image classification accuracy of the trimmed trained image recognition neural network model according to the first test image set; determining a parameter sorting table for each neural network layer according to the neural network layer structure of the trimmed trained image recognition neural network model; wherein, in each parameter sorting table, the parameters are sorted according to the absolute value; when the image classification accuracy is greater than the upper limit of the accuracy threshold interval, trimming the parameters with smaller absolute values in each parameter sorting table in equal proportion to obtain the second trimmed image recognition neural network model. An image recognition neural network model; wherein the cropping of zero-value parameters is included in the proportional calculation; the secondary cropping image classification accuracy of the secondary cropping image recognition neural network model is determined based on the second test image set; when the secondary cropping image classification accuracy is greater than the upper limit of the accuracy threshold interval, the parameters with smaller absolute values in each parameter sorting table are cyclically cropped in proportion, and the secondary cropping image classification accuracy is determined until the secondary cropping image classification accuracy falls within the accuracy threshold interval to obtain the secondary cropping image recognition neural network model; based on the secondary cropping image recognition neural network model, the image recognition neural network model is determined.

[0022] In an embodiment of the present application, the step of obtaining an image recognition neural network model also includes: after obtaining the secondary cropped image recognition neural network model, training the secondary cropped image recognition neural network model according to a third test image set until the secondary cropped image recognition neural network model meets the convergence conditions to obtain the image recognition neural network model; wherein the learning rate of training the secondary cropped image recognition neural network model is lower than the learning rate of training the pre-trained image recognition neural network model.

[0023] In an embodiment of the present application, the pruned image recognition neural network model includes: a convolutional layer, a BN layer and a fully connected layer; according to the neural network layer structure, the parameter sorting table of each neural network layer is determined, including: determining the bias parameter sorting table of each convolutional layer; determining the scaling parameter sorting table of each BN layer; determining the fully connected layer parameter sorting table of the fully connected layer; when the image classification accuracy is greater than the upper limit of the accuracy threshold interval, the parameters with smaller absolute values in each parameter sorting table are pruned in proportion to obtain the image recognition neural network model after secondary pruning, including: pruned in proportion to each bias parameter sorting table, each scaling parameter sorting table and the fully connected layer parameter sorting table to obtain the image recognition neural network model after secondary pruning.

[0024] In an embodiment of the present application, each scaling parameter of the BN layer corresponds to the input data of a channel and several front-layer neurons that generate the channel data; in the case of trimming the scaling parameters in the scaling parameter sorting table, the offset parameters corresponding to the scaling parameters and the parameters of the front-layer neurons corresponding to the scaling parameters are trimmed.

[0025] A second aspect of the present application provides an image recognition device, comprising: a memory configured to store instructions; and a processor configured to call instructions from the memory and implement the image recognition method provided according to any of the above embodiments when executing the instructions.

[0026] A third aspect of the present application provides a machine-readable storage medium having stored thereon instructions for enabling a machine to execute the image recognition method provided according to any one of the above embodiments.

[0027] A fourth aspect of the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the image recognition method provided according to any one of the above embodiments.

[0028] Through the above technical solution, the total number of parameters of the image recognition neural network model can be determined according to the storage resources of the image device and the needs of model establishment, and then the expected sparsity measurement range of the image recognition neural network can be determined in combination with the computing resources of the image recognition device; then the pre-trained image recognition neural network model is trained, and the weights of the first regularization term and the second regularization term in the next round of training are adjusted according to the size relationship between the current sparsity measurement of the model after each round of training and the expected sparsity measurement, so as to adjust the size of the current sparsity after the next round of training, so that the sparsity measurement of the image recognition neural network model finally obtained meets the expected sparsity measurement value within the expected sparsity measurement range; therefore, the image recognition neural network model finally obtained can run smoothly in the image recognition device without causing overload of the hardware of the image recognition device, and can ensure that the accuracy of the image recognition results meets the needs of the application scenario.

[0029] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present application but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0031] Figure 1 The following schematically shows a flow chart of an image recognition method according to an embodiment of the present application;

[0032] Figure 2 Schematically shows Figure 1 A flowchart of a training method for an image recognition neural network model;

[0033] Figure 3 A flow chart of a method for trimming a trained image recognition neural network model according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0034] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0035] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. In the embodiments of this application, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use such solutions.

[0036] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), such directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0037] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0038] [First embodiment]

[0039] Figure 1 The following schematically shows a flow chart of an image recognition method according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides an image recognition method, which is applied to an image recognition device. The method may include the following steps:

[0040] S102, acquiring an image of the target area, where the image includes the object to be identified;

[0041] S104, identifying the object type of the object to be identified by using an image recognition neural network model;

[0042] S106. Output the image recognition result according to the object type;

[0043] Among them, such as Figure 2 As shown in Figure 2, the image recognition neural network model is obtained through the following steps:

[0044] S202: Obtain available computing resources and available storage resources of the image recognition device;

[0045] S204. Determine an expected sparsity metric interval of the image recognition neural network model based on available computing resources and available storage resources, where the sparsity metric is the ratio of non-zero parameters to total parameters in the image recognition neural network model.

[0046] S206. Determine the current sparsity metric of the pre-trained image recognition neural network model after the current round of training based on the training image set;

[0047] S208. When the current sparse metric is greater than the upper bound of the expected sparse metric, increase the weight of the first regularization term and the weight of the second regularization term of the loss function in the next round of training; when the current sparse metric is less than the lower bound of the expected sparse metric, reduce the weight of the first regularization term and the weight of the second regularization term of the loss function in the next round of training;

[0048] S210. When the loss function converges or the number of training rounds reaches a preset number, the training of the pre-trained image recognition neural network model is terminated to obtain an image recognition neural network model.

[0049] The image recognition method provided in the embodiment of the present application can apply the trained image recognition neural network model to the image recognition device to perform image recognition. The image recognition device may, for example, include an on-board processor, a built-in processor for electronic monitoring, etc. The computing resources of the above-mentioned processor are limited, and during the execution of intelligent driving and electronic monitoring functions, they will rely on the computing resources of the local processor. In the image recognition method provided in the embodiment of the present application, the expected sparse measurement interval of the image recognition neural network model is determined based on the computing resources and storage resources of the image recognition device, wherein, for example, the total number of parameters of the image recognition neural network model can be first determined based on the storage resources of the image device and the needs of model establishment, and then the expected proportion of non-zero parameters to the total number of parameters is determined in combination with the computing resources of the image recognition device, that is, the expected sparse measurement interval of the image recognition neural network is determined. Then, the pre-trained image recognition neural network model is trained, and the loss function of the model training has a first regularization term and a second regularization term. By adjusting the weights of the first regularization term and the second regularization term in the next round of training according to the size relationship between the current sparsity metric of the model after each round of training and the expected sparsity metric, the size of the current sparsity after the next round of training is adjusted, so that the sparsity metric of the image recognition neural network model finally obtained meets the expected sparsity metric value within the expected sparsity metric range. Therefore, the image recognition neural network model finally obtained can run smoothly in the image recognition device without causing the hardware of the image recognition device to overload, thereby ensuring that the image recognition results will not be erroneous. Therefore, in application scenarios such as intelligent driving or electronic monitoring that require accurate image recognition results within a limited time, detection and recognition results can also be given in a timely manner while ensuring the expected recognition accuracy.

[0050] On the other hand, the image recognition method provided in the embodiment of the present application can also reduce the cost of image recognition equipment. For example, the total number of parameters and the expected sparse measurement interval of the image recognition neural network model are determined according to the expected model accuracy. Therefore, the image recognition neural network model obtained after training can reduce the proportion of non-zero parameters in the total number of parameters while meeting its model accuracy requirements, thereby reducing the amount of calculation required for model operation, and thus reducing the processing performance requirements for the image recognition equipment, thereby providing a possibility for reducing the cost of the image recognition equipment.

[0051] In the above training process, the first regularization term is used to perform sparse optimization on small-value parameters, and the second regularization term focuses on sparse optimization on large-value parameters. The combined use of the first and second regularization terms can improve the overall sparsification effect of the model. When the current sparsity metric is greater than the upper bound of the expected sparsity metric, it means that the current non-zero parameter amount of the model is large. By increasing the weight of the first regularization term and the weight of the second regularization term, the sparsification effect of the first and second regularization terms can be increased, thereby tending to reduce the amount of non-zero parameters in the next round of training; conversely, when the current sparsity metric is less than the lower bound of the expected sparsity metric, it is necessary to reduce the sparsification effect of the first and second regularization terms, so that the training of the model gradually approaches a sparse metric value within the expected sparsity metric range as the number of iterations increases.

[0052] It is understood that the training of the image recognition neural network model can be performed on a computer with sufficient computing and storage resources, or on a distributed computer network architecture. After the training is completed, the image recognition neural network model obtained can be imported into the image recognition device for application.

[0053] In some embodiments of the present application, in the loss function during training of the pre-trained image recognition neural network model, the first regularization term includes: the logarithm of the L1 norm plus a constant term, where the constant term is greater than or equal to 1.

[0054] The first regularization term in the embodiment of the present application takes the form of the logarithm of the L1 norm plus the constant term. Therefore, a larger sparsification force is applied when the parameter value is close to 0, and the sparsification force on the parameter is gradually reduced when the parameter value is far from 0, thereby focusing the sparsification effect of the first regularization term near the value of 0, and leaving the sparsification of parameters with larger absolute values to the second regularization term. The second regularization term can be, for example, the L2 norm, and its sparsification force for parameters with values close to 0 is almost 0, while the sparsification force for large-value parameters far exceeds that of the first regularization term.

[0055] In some embodiments of the present application, the loss function during training of the pre-trained image recognition neural network model includes:

[0056] ; (1)

[0057] in, is the classification loss function, is the weight of the first regularization term, is the weight of the second regularization term, is the parameter vector of the pre-trained image recognition neural network model, is the L1 norm of the parameter vector, is the square of the L2 norm of the parameter vector, and the square of the L2 norm is the second regularization term.

[0058] In some embodiments of the present application, when the current sparse metric and the previous round sparse metric of the pre-trained image recognition neural network model in the previous round of training are respectively located on both sides of the sparse metric interval, the change in the first regularization term weight and the second regularization term weight is reduced; when the current sparse metric and the previous round sparse metric are located on the same side of the sparse metric interval, the change in the first regularization term weight and the second regularization term weight is increased.

[0059] During the training of a pre-trained image recognition neural network model, the magnitude of the first and second regularization term weights is directly related to the magnitude of the change in the model's sparsity metric during training. During the weight adjustment process, excessive weight adjustments may cause the model's sparsity metric obtained in the previous training round and the model's sparsity metric in the current training round to fall on opposite sides of the expected sparsity metric range. This indicates that the weight adjustment, or the change, is too large, causing the model's sparsity metric to directly cross the expected sparsity metric range between training cycles. Therefore, reducing the change in the first and second regularization term weights allows the model to better approximate values within the expected sparsity metric range during training. On the other hand, if the model's sparsity metric obtained in the previous training round and the model's sparsity metric in the current training round fall on the same side of the expected sparsity metric range, this indicates that the change in the weights after the previous training round was small. Therefore, the change can be appropriately increased to bring the model's sparsity metric back into the expected sparsity metric range as quickly as possible.

[0060] In some embodiments of the present application, the step of obtaining the image recognition neural network model further includes:

[0061] S302, after completing the training of the pre-trained image recognition neural network model, obtaining a trained image recognition neural network model;

[0062] S304: trimming the zero-value parameters in the trained image recognition neural network model to obtain an image recognition neural network model.

[0063] By pruning zero-valued parameters in a trained image recognition neural network model, the storage resources required to store the model can be reduced.

[0064] Specifically, in some embodiments of the present application, the step of obtaining the image recognition neural network model further includes:

[0065] S306. After trimming the zero-value parameters in the trained image recognition neural network model, determine the image classification accuracy of the trimmed trained image recognition neural network model based on the first test image set;

[0066] S308. Determine a parameter sorting table for each neural network layer based on the trimmed neural network layer structure of the trained image recognition neural network model; wherein, in each parameter sorting table, the parameters are sorted by absolute value;

[0067] S310: When the image classification accuracy is greater than the upper limit of the accuracy threshold, parameters with smaller absolute values in the parameter sorting table are trimmed in equal proportion to obtain a secondary trimmed image recognition neural network model; wherein the trimming of zero-valued parameters is included in the calculation of the equal proportion;

[0068] S312, determining the classification accuracy of the secondary cropped image of the secondary cropped image recognition neural network model based on the second test image set;

[0069] S314. When the image classification accuracy after secondary cropping is greater than the upper limit of the accuracy threshold interval, proportionally cropping the parameters with smaller absolute values in each parameter sorting table in a loop, and determining the image classification accuracy after secondary cropping until the image classification accuracy after secondary cropping falls within the accuracy threshold interval, thereby obtaining a neural network model for image recognition after secondary cropping.

[0070] S316. Determine the image recognition neural network model based on the image recognition neural network model after secondary cropping.

[0071] Based on the above-mentioned scheme of trimming model parameters according to the parameter sorting table, the number of model parameters can be compressed as much as possible while ensuring that the accuracy of the model meets expectations, thereby reducing the storage space required to store the obtained image recognition neural network model.

[0072] It is understandable that the accuracy threshold interval in step S314 can be set according to the expected accuracy of the model. For example, if the expected accuracy of the model is 0.9, the accuracy threshold interval can be [0.9, 0.91].

[0073] Specifically, step S314 may include:

[0074] When the image classification accuracy after secondary cropping is lower than the accuracy threshold range, the current round of secondary cropping is canceled, and secondary cropping is performed again based on the secondary cropped image recognition neural network model obtained from the previous round of secondary cropping; wherein the number of parameters for the re-secondary cropping is reduced.

[0075] In some embodiments of the present application, the step of obtaining the image recognition neural network model further includes:

[0076] After obtaining the secondary cropped image recognition neural network model, the secondary cropped image recognition neural network model is trained according to the third test image set until the secondary cropped image recognition neural network model meets the convergence conditions to obtain the image recognition neural network model; wherein the learning rate of training the secondary cropped image recognition neural network model is lower than the learning rate of training the pre-trained image recognition neural network model.

[0077] The secondary cropped image recognition neural network model is trained based on the third test image set. The model can be further fine-tuned based on the existing accuracy of the secondary cropped image recognition neural network model, so that the accuracy of the model can be further improved.

[0078] As an example, the pruned image recognition neural network model includes: convolutional layer, BN layer and fully connected layer;

[0079] Step S308 may include:

[0080] Determine the bias parameter sorting table for each convolutional layer;

[0081] Determine the scaling parameter sorting table for each BN layer;

[0082] Determine the fully connected layer parameter sorting table of the fully connected layer;

[0083] When the image classification accuracy is greater than the upper limit of the accuracy threshold interval, the parameters with smaller absolute values in the parameter sorting table are pruned in equal proportion to obtain the image recognition neural network model after secondary pruning, including:

[0084] Each bias parameter sorting table, each scaling parameter sorting table, and the fully connected layer parameter sorting table are pruned in equal proportion to obtain a secondary pruned image recognition neural network model.

[0085] As mentioned above, for the BN layer, each scaling parameter corresponds to the input data of a channel and a number of front-layer neurons that generate the channel data. Based on this, in some embodiments of the present application, when the scaling parameters in the scaling parameter sorting table are trimmed, the offset parameters corresponding to the scaling parameters and the parameters of the front-layer neurons corresponding to the scaling parameters are trimmed.

[0086] The following is an illustrative description of the image recognition method provided in the embodiments of the present application in combination with the technical solutions provided in the above embodiments.

[0087] The image recognition device provided in the embodiment of the present application is, for example, a face recognition device.

[0088] First, based on the computing and storage resources of the face recognition device and the total number of parameters of the pre-trained convolutional neural network model (for example, 30 million), the expected parameter sparsity measure S=0.7 of the model adapted to the face recognition device is determined, and the expected parameter sparsity measure range is [0.699, 0.701].

[0089] Then, the facial images of the people to be identified are used as the training data set, and the Adam algorithm is used to iteratively train the parameters of the convolutional neural network model until the target classification loss function converges or the training rounds exceed the preset number (for example, 200 rounds). The target classification loss function used in the training process is shown in formula (1): If the sparsity of the model parameters at the end of the current training round is greater than , then in the next round of training 、 The value of increases by 1%; if the sparsity of the model parameters is less than , then in the next round of training 、 The value of is reduced by 1%.

[0090] For each convolutional layer of the trained convolutional neural network model, all bias parameters of the convolutional layer are sorted by absolute value, and a sorted list is generated for each convolutional layer. For the batch normalization layer parameters of the trained convolutional neural network model, the BN layer parameters include scaling parameters and offset parameters. All scaling parameters of each BN layer are sorted by absolute value, and a sorted list is generated for each BN layer. For the fully connected layer parameters of the trained convolutional neural network model, they are sorted by absolute value, and a sorted list is generated for the fully connected layers.

[0091] Prune all zero-valued parameters of the trained convolutional neural network model.

[0092] The target classification accuracy of the trimmed model is tested on the test dataset. If the accuracy is higher than the preset classification accuracy threshold , then the parameters in each of the aforementioned sorted lists are trimmed appropriately according to their absolute values from small to large, and the number of trimmed parameters in different sorted lists (including the aforementioned trimmed zero-value parameters) accounts for the same proportion of the total number of parameters in the sorted lists. Repeat this step until the target classification accuracy of the model is equal to the preset classification accuracy threshold. .

[0093] The pruned model is fine-tuned on the training dataset with a smaller learning rate. The fine-tuned model is deployed as the final face recognition model on the face recognition device to perform specific face recognition tasks.

[0094] [Second embodiment]

[0095] A second embodiment of the present application provides an image recognition method, which is applied to an image recognition device. The method may include the following steps:

[0096] Acquire an image of a target area, where the image includes an object to be identified;

[0097] Identify the object type of the object to be identified by using an image recognition neural network model;

[0098] Output image recognition results according to object type;

[0099] The image recognition neural network model is obtained through a sparse and robust federated learning method. The sparse and robust federated learning method is applied to a server, and the server communicates with a client. The sparse and robust federated learning method may include the following steps:

[0100] Obtain available computing resources and available storage resources of the image recognition device;

[0101] Based on available computing resources and available storage resources, determine the expected sparsity metric range of the image recognition neural network model. The sparsity metric is the ratio of non-zero parameters to the total parameters in the image recognition neural network model.

[0102] Initializing the parameters of the pre-trained image recognition neural network model according to the expected sparsity measurement interval to obtain initialization parameters;

[0103] Sending the initialization parameters to the plurality of target client nodes respectively, so that the client updates the local model according to the initialization parameters, the first regularization term and the second regularization term, and the local training image set of the client, thereby obtaining a plurality of local model update parameters;

[0104] Receive multiple local model update parameters returned by the client. Each local model update parameter can be obtained through the following steps:

[0105] Based on the client's local training image set, determine the current sparsity metric of the local model after this round of training;

[0106] When the current sparsity metric is greater than the upper bound of the expected sparsity metric, the weight of the first regularization term and the weight of the second regularization term of the loss function in the next round of training are increased; when the current sparsity metric is less than the lower bound of the expected sparsity metric, the weight of the first regularization term and the weight of the second regularization term of the loss function in the next round of training are reduced;

[0107] When the local training rounds reach the preset number, the local model training is terminated to obtain the local model update parameters;

[0108] determining aggregate parameters of a pre-trained image recognition neural network model based on multiple local model update parameters;

[0109] Determine whether the aggregation parameters of the pre-trained image recognition neural network model meet the preset termination conditions;

[0110] When the aggregation parameters of the pre-trained image recognition neural network model meet the preset termination conditions, it is determined that the training of the pre-trained image recognition neural network model is completed to obtain the image recognition neural network model.

[0111] The image recognition method provided in the embodiment of the present application can apply the trained image recognition neural network model to the image recognition device to perform image recognition. The image recognition device may, for example, include an on-board processor, a built-in processor for electronic monitoring, etc. The computing resources of the above-mentioned processor are limited, and during the execution of intelligent driving and electronic monitoring functions, they will rely on the computing resources of the local processor. In the image recognition method provided in the embodiment of the present application, the expected sparse measurement interval of the image recognition neural network model is determined based on the computing resources and storage resources of the image recognition device, wherein, for example, the total number of parameters of the image recognition neural network model can be first determined based on the storage resources of the image device and the needs of model establishment, and then the expected proportion of non-zero parameters to the total number of parameters is determined in combination with the computing resources of the image recognition device, that is, the expected sparse measurement interval of the image recognition neural network is determined. Then, the pre-trained image recognition neural network model is trained, and the loss function of the model training has a first regularization term and a second regularization term. By adjusting the weights of the first regularization term and the second regularization term in the next round of training according to the size relationship between the current sparsity metric of the model after each round of training and the expected sparsity metric, the size of the current sparsity after the next round of training is adjusted, so that the sparsity metric of the image recognition neural network model finally obtained meets the expected sparsity metric value within the expected sparsity metric range. Therefore, the image recognition neural network model finally obtained can run smoothly in the image recognition device without causing the hardware of the image recognition device to overload, thereby ensuring that the image recognition results will not be erroneous. Therefore, in application scenarios such as intelligent driving or electronic monitoring that require accurate image recognition results within a limited time, detection and recognition results can also be given in a timely manner while ensuring the expected recognition accuracy.

[0112] On the other hand, the image recognition method provided in the embodiment of the present application can also reduce the cost of image recognition equipment. For example, the total number of parameters and the expected sparse measurement interval of the image recognition neural network model are determined according to the expected model accuracy. Therefore, the image recognition neural network model obtained after training can reduce the proportion of non-zero parameters in the total number of parameters while meeting its model accuracy requirements, thereby reducing the amount of calculation required for model operation, and thus reducing the processing performance requirements for the image recognition equipment, thereby providing a possibility for reducing the cost of the image recognition equipment.

[0113] In the embodiment of the present application, before model training, the server first initializes the parameters of the pre-trained image recognition neural network model to obtain the initialization parameters as For example, the pre-trained image recognition neural network model can be a deep neural network model. After obtaining the initialization parameters, the initialization parameters are sent to multiple target client nodes respectively. In one example, the client has nodes, Randomly select from client nodes client nodes, the selected federated learning client nodes are ,in is the client node serial number, . And the federated learning client nodes are Send initialization parameters After receiving the initialization parameters, the client updates the local model according to the initialization parameters, the first regularization term, and the second regularization term, thereby obtaining multiple local model update parameters. The first regularization term can be, for example, a Log-modified regularization term of the L1 norm. For example, when updating, the pre-trained image recognition neural network model can be trained using a stochastic gradient descent algorithm using a local training image set of the client. In one example, in the first When the global model parameters are issued, the server node S Randomly select from client nodes client nodes, and pre-trained the parameters of the image recognition neural network model Sent to the selected federated learning client node to start Round model training and global model parameter update.

[0114] In an embodiment of the present application, the model on the client node is a local model, and the model on the server is a global model. The client node submits updated parameters for multiple local models to the server. The server node aggregates the updated parameters using the multiple parameters to generate a global model. The aggregated updated global model parameters are then sent to the client node. The client node then updates its local model using the newly received global model parameters, repeating this cycle. After the client updates the local model and obtains multiple local model update parameters, the client node returns the multiple local model update parameters to the server. After receiving the multiple local model update parameters returned by the client, the server determines the aggregate parameters of the global model based on the multiple local model update parameters. For example, when determining the aggregate parameters, the number of samples contained in the locally stored sample datasets of the multiple client nodes and the multiple local model update parameters can be obtained. The number of samples contained in the locally stored sample datasets of each client node and the local model update parameters are then multiplied to obtain multiple products. Finally, the multiple products are summed and divided by the sum of the number of samples from the multiple client nodes to obtain the aggregate parameters of the global model. The aggregate parameters are then determined to determine whether they meet a preset termination condition. The preset termination condition refers to a pre-set condition for terminating global model training. For example, the preset termination condition may be that the aggregate parameters of the global model converge, or that the total number of training rounds of the global model is greater than or equal to a threshold constant. When the aggregate parameters of the global model meet the preset termination condition, the global model training is determined to be complete.

[0115] Through the above technical solution, the server initializes the parameters of the global model to obtain initialization parameters. The initialization parameters are then sent to multiple target client nodes, allowing the clients to update local models based on the initialization parameters, the first regularization term, and the second regularization term, thereby obtaining multiple local model update parameters. The server then receives the multiple local model update parameters returned by the clients. The server then determines the aggregation parameters of the global model based on the multiple local model update parameters. Finally, it determines whether the aggregation parameters of the global model meet the preset termination conditions. If the aggregation parameters of the global model meet the preset termination conditions, the global model training is determined to be complete. This can improve the sparsity of the model while improving its robustness. The regularization term weights can be automatically and dynamically adjusted with the increase in training rounds, enhancing the model training effect. Since the training process of the image recognition neural network model involves reducing the non-zero parameters of the model, to better ensure the accuracy of the image recognition neural network model, federated learning can be used to enable the obtained image recognition neural network model to integrate the data knowledge of the training image sets of multiple clients, thereby improving the generalization ability of the image recognition neural network model as a global model, thereby helping to improve the accuracy of the image recognition neural network model in practical applications.

[0116] In the above training process, the first regularization term is used to perform sparse optimization on small-value parameters, and the second regularization term focuses on sparse optimization on large-value parameters. The combined use of the first and second regularization terms can improve the overall sparsification effect of the model. When the current sparsity metric is greater than the upper bound of the expected sparsity metric, it means that the current non-zero parameter amount of the model is large. By increasing the weight of the first regularization term and the weight of the second regularization term, the sparsification effect of the first and second regularization terms can be increased, thereby tending to reduce the amount of non-zero parameters in the next round of training; conversely, when the current sparsity metric is less than the lower bound of the expected sparsity metric, it is necessary to reduce the sparsification effect of the first and second regularization terms, so that the training of the model gradually approaches a sparse metric value within the expected sparsity metric range as the number of iterations increases.

[0117] It is understood that the training of the image recognition neural network model can be performed on a computer with sufficient computing and storage resources, or on a distributed computer network architecture. After the training is completed, the image recognition neural network model obtained can be imported into the image recognition device for application.

[0118] In the embodiment of the present application, the update parameters may satisfy formula (2):

[0119] ; (2)

[0120] in, is the client node serial number, is the number of model training rounds, For the Round, first The updated parameters of the global model of the client nodes, For the Parameters of the wheel global model, is the local model parameter vector, is the learning rate constant, represents the novelty loss function exist Place gradient.

[0121] Specifically, the update parameters can satisfy the formula In the During the round of model training, each selected federated learning client node Using slave server nodes Received global model parameters Update the local model. For the Round, first Update parameters of the global model for each client node. is the local model parameter vector, is the learning rate constant, represents the novelty loss function exist Place The new loss function is the function of the original loss function plus the regularization term. By determining the update parameters of the global model, the aggregation parameters of the global model can be determined.

[0122] In the embodiment of the present application, the new loss function may satisfy the formula:

[0123] ; (3)

[0124] in, The new loss function after adding the regularization term to the original loss function of the global model, is the classification loss function, is the weight of the first regularization term, is the weight of the second regularization term, is the local model parameter vector, is the L1 norm of the parameter vector, is the square of the L2 norm of the parameter vector, and the square of the L2 norm is the second regularization term.

[0125] In some embodiments of the present application, when the current sparse metric and the previous round sparse metric of the pre-trained image recognition neural network model in the previous round of training are respectively located on both sides of the sparse metric interval, the change in the first regularization term weight and the second regularization term weight is reduced; when the current sparse metric and the previous round sparse metric are located on the same side of the sparse metric interval, the change in the first regularization term weight and the second regularization term weight is increased.

[0126] During the training of a pre-trained image recognition neural network model, the magnitude of the first and second regularization term weights is directly related to the magnitude of the change in the model's sparsity metric during training. During the weight adjustment process, excessive weight adjustments may cause the model's sparsity metric obtained in the previous training round and the model's sparsity metric in the current training round to fall on opposite sides of the expected sparsity metric range. This indicates that the weight adjustment, or the change, is too large, causing the model's sparsity metric to directly cross the expected sparsity metric range between training cycles. Therefore, reducing the change in the first and second regularization term weights allows the model to better approximate values within the expected sparsity metric range during training. On the other hand, if the model's sparsity metric obtained in the previous training round and the model's sparsity metric in the current training round fall on the same side of the expected sparsity metric range, this indicates that the change in the weights after the previous training round was small. Therefore, the change can be appropriately increased to bring the model's sparsity metric back into the expected sparsity metric range as quickly as possible.

[0127] In an embodiment of the present application, determining whether the aggregation parameters of the global model meet a preset termination condition may include:

[0128] Determine whether the aggregation parameters of the global model have converged;

[0129] Alternatively, determine whether the total number of training rounds of the global model is greater than or equal to a threshold constant.

[0130] Specifically, the preset termination condition refers to a pre-set condition for terminating global model training. The preset termination condition can be convergence of the global model's aggregate parameters, or the total number of global model training rounds being greater than or equal to a threshold constant. The threshold constant is a positive integer greater than 1. When the global model's aggregate parameters meet the preset termination condition, global model training can be determined to be complete, thereby obtaining an image recognition neural network model.

[0131] In an embodiment of the present application, determining the aggregation parameter of the global model based on multiple local model update parameters may include:

[0132] Respectively obtain the number of samples contained in the locally stored sample data sets of the multiple client nodes and multiple local model update parameters;

[0133] Multiplying the number of samples contained in the locally stored sample data set of each client node and the local model update parameter to obtain multiple products;

[0134] The multiple products are summed and divided by the sum of the number of samples of multiple client nodes to obtain the aggregate parameters of the global model.

[0135] Specifically, after the client updates the global model and obtains multiple local model update parameters, the multiple local model update parameters are returned to the server. After receiving the multiple local model update parameters returned by the client, the server determines the aggregation parameters of the global model based on the multiple local model update parameters. First, the server obtains the number of samples contained in the locally stored sample data sets of multiple client nodes and the multiple local model update parameters returned by the client. Then, the number of samples contained in the locally stored sample data sets of each client node and the local model update parameters are multiplied to obtain multiple products. Finally, the multiple products are summed and divided by the sum of the number of samples of the multiple client nodes to obtain the aggregation parameters of the global model.

[0136] Specifically, the aggregation parameters can satisfy the formula . is the aggregation parameter of the global model. For The number of client nodes randomly selected from the client nodes. For client nodes Locally stored sample dataset, For collection Contains the number of elements. Client nodes The number of elements in the locally stored sample dataset is multiplied by the update parameter to obtain Then multiply The products are summed and divided by the sum of the number of samples of all selected client nodes to obtain the aggregation parameters of the global model.

[0137] In an embodiment of the present application, the federated learning method may further include:

[0138] When the aggregation parameters of the global model do not meet the preset termination conditions, the global model is retrained.

[0139] Specifically, when the aggregation parameters of the global model do not meet the preset termination conditions, multiple client nodes are randomly selected again, and initialization parameters are sent to the client nodes to perform model training until the aggregation parameters of the global model meet the preset termination conditions.

[0140] Through the above technical solution, the server initializes the parameters of the global model to obtain initialization parameters. The server then sends the initialization parameters to multiple target client nodes, which then update the local models based on the initialization parameters, the first regularization term, and the second regularization term, thereby obtaining multiple local model update parameters. The server then receives the multiple local model update parameters returned by the client. The server then determines the aggregate parameters of the global model based on the multiple local model update parameters. Finally, it determines whether the aggregate parameters of the global model meet a preset termination condition. If the aggregate parameters of the global model meet the preset termination condition, the global model training is considered complete.

[0141] After obtaining the trained global model, the trained image recognition neural network model provided in the first embodiment of the present application is obtained, and the global model can be tailored according to the first embodiment of the present application. This allows the number of model parameters to be compressed as much as possible while ensuring that the model's accuracy meets expectations, thereby reducing the storage space required to store the obtained image recognition neural network model.

[0142] [Third embodiment]

[0143] The present application also provides an image recognition device, which may include: a memory configured to store instructions; and a processor configured to call the instructions from the memory and, when executing the instructions, implement the image recognition method provided in any of the above embodiments.

[0144] An embodiment of the present application further provides a machine-readable storage medium having stored thereon instructions for enabling a machine to execute the image recognition method provided in any one of the above embodiments.

[0145] An embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the image recognition method provided according to any of the above embodiments.

[0146] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0150] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0151] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0152] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0154] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. An image recognition method, characterized in that: Applied to an image recognition device, the method includes: Acquire an image of a target area, wherein the image includes an object to be identified; Identifying the object type of the object to be identified by using an image recognition neural network model; Outputting an image recognition result according to the object type; The image recognition neural network model is obtained by the following steps: Obtaining available computing resources and available storage resources of the image recognition device; Determining an expected sparsity metric interval of the image recognition neural network model based on the available computing resources and the available storage resources, wherein the sparsity metric is a ratio of the number of non-zero parameters to the total number of parameters in the image recognition neural network model; Based on the training image set, determine the current sparsity metric of the pre-trained image recognition neural network model after the current round of training; When the current sparsity metric is greater than the upper bound of the expected sparsity metric, increasing the first regularization term weight and the second regularization term weight of the loss function in the next round of training; When the current sparsity metric is less than the lower bound of the expected sparsity metric, reducing the first regularization term weight and the second regularization term weight of the loss function in the next round of training; When the loss function converges or the number of training rounds reaches a preset number, the training of the pre-trained image recognition neural network model is terminated to obtain the image recognition neural network model; The step of obtaining the image recognition neural network model further includes: After completing the training of the pre-trained image recognition neural network model, obtaining a trained image recognition neural network model; trimming zero-valued parameters in the trained image recognition neural network model; After trimming zero-value parameters in the trained image recognition neural network model, determining an image classification accuracy rate of the trimmed trained image recognition neural network model based on a first test image set; Determine a parameter sorting table for each neural network layer according to the trimmed neural network layer structure of the trained image recognition neural network model; wherein, in each parameter sorting table, the parameters are sorted according to the absolute value; When the image classification accuracy is greater than the upper limit of the accuracy threshold interval, parameters with smaller absolute values in each parameter sorting table are trimmed in equal proportion to obtain a secondary trimmed image recognition neural network model; wherein the trimming of the zero-value parameters is included in the equal proportion calculation; Determining the secondary cropped image classification accuracy of the secondary cropped image recognition neural network model based on the second test image set; When the classification accuracy of the image after secondary cropping is greater than the upper limit of the accuracy threshold interval, cyclically performing the proportional cropping on the parameters with smaller absolute values in each parameter sorting table, and determining the classification accuracy of the image after secondary cropping until the classification accuracy of the image after secondary cropping falls within the accuracy threshold interval, so as to obtain a neural network model for image recognition after secondary cropping; The image recognition neural network model is determined according to the image recognition neural network model after secondary cropping.

2. The method according to claim 1, characterized in that The first regularization term includes: the logarithm of the L1 norm plus a constant term, where the constant term is greater than or equal to 1.

3. The method according to claim 2, characterized in that The loss function includes: Wherein, F(w) is the classification loss function, α1 is the weight of the first regularization term, α2 is the weight of the second regularization term, w is the parameter vector of the pre-trained image recognition neural network model, ||w||1 is the L1 norm of the parameter vector, is the square of the L2 norm of the parameter vector, and the square of the L2 norm is the second regularization term.

4. The method according to claim 1, wherein: When the current sparse metric and the previous round sparse metric of the pre-trained image recognition neural network model in the previous round of training are respectively located on both sides of the sparse metric interval, reducing the change amount of the first regularization term weight and the second regularization term weight; In a case where the current sparse metric and the previous round sparse metric are located on the same side of the sparse metric interval, the change amounts of the first regularization term weight and the second regularization term weight are increased.

5. The method according to claim 1, wherein The step of obtaining the image recognition neural network model also includes: After obtaining the secondary cropped image recognition neural network model, the secondary cropped image recognition neural network model is trained according to a third test image set until the secondary cropped image recognition neural network model meets a convergence condition to obtain the image recognition neural network model; wherein the learning rate of training the secondary cropped image recognition neural network model is lower than the learning rate of training the pre-trained image recognition neural network model.

6. The method according to claim 1, characterized in that The cropped image recognition neural network model includes: a convolutional layer, a BN layer and a fully connected layer; Determining a parameter sorting table for each neural network layer according to the neural network layer structure includes: Determine a bias parameter sorting table for each of the convolutional layers; Determine a scaling parameter sorting table for each of the BN layers; Determine a fully connected layer parameter sorting table of the fully connected layer; When the image classification accuracy is greater than the upper limit of the accuracy threshold interval, parameters with smaller absolute values in each parameter sorting table are trimmed in equal proportion to obtain a secondary trimmed image recognition neural network model, including: Each of the bias parameter sorting tables, each of the scaling parameter sorting tables, and the fully connected layer parameter sorting table is pruned in equal proportion to obtain the secondary pruned image recognition neural network model.

7. The method according to claim 6, characterized in that Each scaling parameter of the BN layer corresponds to the input data of a channel and a number of front-layer neurons that generate the channel data; In the case of trimming the scaling parameters in the scaling parameter sorting table, the offset parameters corresponding to the scaling parameters and the parameters of the front layer neurons corresponding to the scaling parameters are trimmed.

8. An image recognition device, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instructions from the memory and implement the image recognition method according to any one of claims 1 to 7 when executing the instructions.

9. A machine-readable storage medium, characterized in that The machine-readable storage medium stores instructions, which are used to enable a machine to execute the image recognition method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the image recognition method according to any one of claims 1 to 7.

Images