Image recognition method, device and equipment, storage medium and product thereof

Abstract

The invention discloses an image recognition method, device and equipment, a machine readable storage medium and a computer program product, and relates to the technical field of image recognition. The method comprises the following steps: identifying an object type of a to-be-identified object through an image identification neural network model; the image recognition neural network model is obtained through the following steps: determining an expected sparse measurement interval of the model based on resources of image recognition equipment; determining a current sparse measure of model training; under the condition that the current sparse metric is greater than the upper bound of the expected sparse metric, increasing a first regular term weight and a second regular term weight of a loss function in the next round of training; under the condition that the current sparse metric is smaller than the lower bound of the expected sparse metric, reducing the first regular term weight and the second regular term weight of the loss function in the next round of training; and the training is ended, so that the image recognition neural network model is obtained. According to the method, the model can be simplified and the expected image recognition precision can be realized.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and specifically to an image recognition method, apparatus, device, machine-readable storage medium, and computer program product. Background Art

[0002] At present, with the rapid development of artificial intelligence technology, the application of artificial intelligence technology is becoming more and more extensive, and has been applied to many fields such as image analysis, speech recognition, text processing, intelligent recommendation, and security detection. In most cases, the more parameters a model has, the better the performance of the model in completing the task. However, in application scenarios where hardware computing and storage resources are limited, such as face recognition instruments, intelligent driving vision systems, electronic monitoring equipment, and other scenarios involving image recognition, the hardware equipment running the model is not enough to support the operation of models with a large number of parameters. For example, in the case of limited hardware computing resources, if there are a large number of non-zero parameters in the model, so that the complexity of the model exceeds the computing power of the hardware, the image recognition speed of the hardware will be significantly reduced, and the calculation and final image recognition results may be wrong due to hardware overload. In application scenarios such as intelligent driving or electronic monitoring, which require accurate image recognition results in a limited time, how to simplify the model without significantly reducing the prediction accuracy of the model, so that the image recognition model can achieve the expected image recognition accuracy when applied to the above scenarios, is a technical problem that needs to be solved at present. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an image recognition method, apparatus, device, machine-readable storage medium, and computer program product to solve the technical problem in the prior art of how to streamline the model without significantly reducing the prediction accuracy of the model, so that the image recognition model can give the expected recognition accuracy when applied to the above-mentioned scenarios.

[0004] In order to achieve the above-mentioned object, the first aspect of the present application provides an image recognition method, which is applied to an image recognition device, and the method comprises: Acquire an image of a target area, the image including an object to be identified; Identify the object type of the object to be identified by using an image recognition neural network model; Output image recognition results according to the object type; Among them, the image recognition neural network model is obtained through the following steps: Obtain available computing resources and available storage resources of the image recognition device; Based on available computing resources and available storage resources, determine the expected sparseness metric interval of the image recognition neural network model. The sparseness metric is the ratio of the non-zero parameters to the total number of parameters in the image recognition neural network model. Based on the training image set, determine the current sparsity measure of the pre-trained image recognition neural network model after the current round of training; When the current sparseness metric is greater than the upper bound of the expected sparseness metric, increase the weight of the first regularization term and the second regularization term of the loss function in the next round of training; When the current sparseness metric is less than the lower bound of the expected sparseness metric, the weight of the first regularization term and the weight of the second regularization term of the loss function in the next round of training are reduced; When the loss function converges or the number of training rounds reaches a preset number, the training of the pre-trained image recognition neural network model is terminated to obtain an image recognition neural network model.

[0005] In the embodiment of the present application, the first regularization term includes: the logarithm of the L1 norm plus a constant term, where the constant term is greater than or equal to 1.

[0006] In the embodiment of the present application, the loss function includes: ; in, is the classification loss function, is the weight of the first regularization term, is the weight of the second regularization term, is the parameter vector of the pre-trained image recognition neural network model, is the L1 norm of the parameter vector, is the square of the L2 norm of the parameter vector, and the square of the L2 norm is the second regular term.

[0007] In an embodiment of the present application, when the current sparse metric and the previous round sparse metric of the pre-trained image recognition neural network model in the previous round of training are respectively located on both sides of the sparse metric interval, the change in the first regularization term weight and the second regularization term weight is reduced; when the current sparse metric and the previous round sparse metric are located on the same side of the sparse metric interval, the change in the first regularization term weight and the second regularization term weight is increased.

[0008] In an embodiment of the present application, the step of obtaining an image recognition neural network model also includes: after completing the training of the pre-trained image recognition neural network model, obtaining a trained image recognition neural network model; and trimming the zero-value parameters in the trained image recognition neural network model to obtain an image recognition neural network model.

[0009] In an embodiment of the present application, the step of obtaining an image recognition neural network model further includes: after trimming the zero-value parameters in the trained image recognition neural network model, determining the image classification accuracy of the trimmed trained image recognition neural network model according to the first test image set; determining a parameter sorting table of each neural network layer according to the neural network layer structure of the trimmed trained image recognition neural network model; wherein, in each parameter sorting table, the parameters are sorted according to the absolute value; when the image classification accuracy is greater than the upper limit of the accuracy threshold interval, trimming the parameters with smaller absolute values ​​in each parameter sorting table in equal proportion to obtain the second trimmed An image recognition neural network model; wherein the trimming of zero-value parameters is included in the proportional calculation; the image classification accuracy of the image recognition neural network model after secondary trimming is determined according to the second test image set; when the image classification accuracy after secondary trimming is greater than the upper limit of the accuracy threshold interval, the parameters with smaller absolute values ​​in each parameter sorting table are cyclically trimmed in proportion, and the image classification accuracy after secondary trimming is determined until the image classification accuracy after secondary trimming falls into the accuracy threshold interval to obtain the image recognition neural network model after secondary trimming; according to the image recognition neural network model after secondary trimming, the image recognition neural network model is determined.

[0010] In an embodiment of the present application, the step of obtaining an image recognition neural network model also includes: after obtaining the image recognition neural network model after secondary trimming, training the image recognition neural network model after secondary trimming according to a third test image set until the image recognition neural network model after secondary trimming meets convergence conditions to obtain the image recognition neural network model; wherein the learning rate of training the image recognition neural network model after secondary trimming is lower than the learning rate of training the pre-trained image recognition neural network model.

[0011] In an embodiment of the present application, the pruned image recognition neural network model includes: a convolutional layer, a BN layer and a fully connected layer; according to the neural network layer structure, a parameter sorting table of each neural network layer is determined, including: determining a bias parameter sorting table of each convolutional layer; determining a scaling parameter sorting table of each BN layer; determining a fully connected layer parameter sorting table of the fully connected layer; when the image classification accuracy is greater than the upper limit of the accuracy threshold interval, the parameters with smaller absolute values ​​in each parameter sorting table are pruned in proportion to obtain a secondary pruned image recognition neural network model, including: pruned in proportion to each bias parameter sorting table, each scaling parameter sorting table and the fully connected layer parameter sorting table to obtain a secondary pruned image recognition neural network model.

[0012] In an embodiment of the present application, each scaling parameter of the BN layer corresponds to the input data of a channel and several front-layer neurons that generate the channel data; in the case of trimming the scaling parameters in the scaling parameter sorting table, the offset parameters corresponding to the scaling parameters and the parameters of the front-layer neurons corresponding to the scaling parameters are trimmed.

[0013] A second aspect of the present application provides an image recognition device, comprising: a memory configured to store instructions; and a processor configured to call instructions from the memory and implement the image recognition method provided according to any one of the above embodiments when executing the instructions.

[0014] A third aspect of the present application provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the image recognition method provided according to any one of the above embodiments.

[0015] A fourth aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the image recognition method provided according to any of the above embodiments.

[0016] Through the above technical scheme, the total number of parameters of the image recognition neural network model can be determined according to the storage resources of the image device and the needs of model establishment, and then the expected sparse measurement range of the image recognition neural network can be determined in combination with the computing resources of the image recognition device; then the pre-trained image recognition neural network model is trained, and the weights of the first regularization term and the second regularization term in the next round of training are adjusted according to the size relationship between the current sparse measurement of the model after each round of training and the expected sparse measurement, so as to adjust the size of the current sparsity after the next round of training, so that the sparse measurement of the image recognition neural network model finally obtained meets the expected sparse measurement value within the expected sparse measurement range; therefore, the image recognition neural network model finally obtained can run smoothly in the image recognition device without causing overload of the hardware of the image recognition device, and can ensure that the accuracy of the image recognition result meets the needs of the application scenario.

[0017] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings: Figure 1 A schematic diagram of a process flow of an image recognition method according to an embodiment of the present application is shown; Figure 2 Schematically shows Figure 1 A flowchart of a training method for an image recognition neural network model; Figure 3 A flow chart of a method for trimming a trained image recognition neural network model according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0020] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. In the embodiments of this application, some existing solutions in the industry such as certain software, components, and models may be mentioned, which should be considered as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0021] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), such directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0022] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0023] [First embodiment] Figure 1 The following schematically shows a flow chart of an image recognition method according to an embodiment of the present application. Figure 1As shown, an embodiment of the present application provides an image recognition method, which is applied to an image recognition device. The method may include the following steps: S102, acquiring an image of the target area, where the image includes an object to be identified; S104, identifying the object type of the object to be identified by using an image recognition neural network model; S106, outputting the image recognition result according to the object type; Among them, Figure 2 As shown, the image recognition neural network model is obtained through the following steps: S202, obtaining available computing resources and available storage resources of the image recognition device; S204, determining an expected sparse metric interval of the image recognition neural network model based on available computing resources and available storage resources, where the sparse metric is the ratio of non-zero parameters to total parameters in the image recognition neural network model; S206, determining the current sparseness metric of the pre-trained image recognition neural network model after the current round of training based on the training image set; S208, when the current sparse metric is greater than the upper bound of the expected sparse metric, increase the weight of the first regularization term and the weight of the second regularization term of the loss function in the next round of training; when the current sparse metric is less than the lower bound of the expected sparse metric, reduce the weight of the first regularization term and the weight of the second regularization term of the loss function in the next round of training; S210. When the loss function converges or the number of training rounds reaches a preset number, the training of the pre-trained image recognition neural network model is terminated to obtain an image recognition neural network model.

[0024] The image recognition method provided in the embodiment of the present application can apply the trained image recognition neural network model to the image recognition device to perform image recognition. The image recognition device may, for example, include an on-board processor, a built-in processor for electronic monitoring, etc. The computing resources of the above-mentioned processor are limited, and during the execution of intelligent driving and electronic monitoring functions, they will rely on the computing resources of the local processor. In the image recognition method provided in the embodiment of the present application, the expected sparse measurement interval of the image recognition neural network model is determined based on the computing resources and storage resources of the image recognition device, wherein, for example, the total number of parameters of the image recognition neural network model can be first determined based on the storage resources of the image device and the need for model establishment, and then the expected non-zero parameter amount is determined in combination with the computing resources of the image recognition device. The proportion of the total number of parameters, that is, the expected sparse measurement interval of the image recognition neural network is determined. Then, the pre-trained image recognition neural network model is trained, and the loss function of the model training has a first regularization term and a second regularization term. By adjusting the weights of the first regularization term and the second regularization term in the next round of training according to the size relationship between the current sparsity metric of the model after each round of training and the expected sparsity metric, the size of the current sparsity after the next round of training is adjusted, so that the sparse metric of the image recognition neural network model finally obtained meets the expected sparse metric value within the expected sparsity metric interval. Therefore, the image recognition neural network model finally obtained can run smoothly in the image recognition device without causing the hardware of the image recognition device to overload, thereby ensuring that the image recognition results will not be erroneous. Therefore, in application scenarios such as intelligent driving or electronic monitoring that require accurate image recognition results within a limited time, detection and recognition results can also be given in a timely manner while ensuring the expected recognition accuracy.

[0025] On the other hand, the image recognition method provided in the embodiment of the present application can also reduce the cost of image recognition equipment. For example, the total number of parameters and the expected sparse measurement interval of the image recognition neural network model are determined according to the expected model accuracy. Therefore, the image recognition neural network model obtained after training can reduce the proportion of non-zero parameters in the total number of parameters while meeting its model accuracy requirements, thereby reducing the amount of calculation required for the model operation, and thus reducing the processing performance requirements for the image recognition equipment, thereby providing a possibility for reducing the cost of the image recognition equipment.

[0026] In the above training process, the first regularization term is used to perform sparse optimization on small-value parameters, and the second regularization term focuses on sparse optimization on large-value parameters. The combined use of the first regularization term and the second regularization term can improve the overall sparsification effect of the model. When the current sparse metric is greater than the upper bound of the expected sparse metric, it means that the current non-zero parameter amount of the model is large. By increasing the weight of the first regularization term and the weight of the second regularization term, the sparse effect of the first regularization term and the second regularization term can be increased, so that the amount of non-zero parameters tends to be reduced in the next round of training; conversely, when the current sparse metric is less than the lower bound of the expected sparse metric, it is necessary to reduce the sparse influence of the first regularization term and the second regularization term, so that the training of the model gradually approaches a sparse metric value within the expected sparse metric interval as the number of iterations increases.

[0027] It is understandable that the training of the image recognition neural network model can be performed on a computer with sufficient computing and storage resources, or on a distributed computer network architecture. The image recognition neural network model obtained after the training is completed can be imported into the image recognition device for application.

[0028] In some embodiments of the present application, in the loss function during the training of the pre-trained image recognition neural network model, the first regularization term includes: the logarithm of the L1 norm plus a constant term, where the constant term is greater than or equal to 1.

[0029] The first regularization term in the embodiment of the present application is in the form of the logarithm of the L1 norm plus the constant term, so a larger sparsification force is applied when the parameter value is close to 0, and the sparsification force on the parameter is gradually reduced when the parameter value is far away from 0, so that the sparsification effect of the first regularization term is focused near the value of 0, and the sparsification of parameters with larger absolute values ​​is handed over to the second regularization term. The second regularization term can be, for example, the L2 norm, and its sparsification force for parameters with values ​​close to 0 is almost 0, while the sparsification force for large-value parameters far exceeds the first regularization term.

[0030] In some embodiments of the present application, the loss function during training of the pre-trained image recognition neural network model includes: ; (1) in, is the classification loss function, is the weight of the first regularization term, is the weight of the second regularization term, is the parameter vector of the pre-trained image recognition neural network model, is the L1 norm of the parameter vector, is the square of the L2 norm of the parameter vector, and the square of the L2 norm is the second regular term.

[0031] In some embodiments of the present application, when the current sparse metric and the previous round sparse metric of the pre-trained image recognition neural network model in the previous round of training are respectively located on both sides of the sparse metric interval, the change in the first regularization term weight and the second regularization term weight is reduced; when the current sparse metric and the previous round sparse metric are located on the same side of the sparse metric interval, the change in the first regularization term weight and the second regularization term weight is increased.

[0032] During the training of the pre-trained image recognition neural network model, the size of the first regularization item weight and the second regularization item weight is directly related to the change amplitude of the model sparse metric during the training process. In the process of adjusting the above weights, it may happen that the model sparse metric obtained in the previous round of training and the model sparse metric of the current round are on both sides of the expected sparse metric interval due to the excessive weight adjustment amplitude. Therefore, it is explained that the adjustment amplitude of the above weights, that is, the change amount, is too large, causing the sparse metric of the model to directly cross the range of the expected sparse metric interval during the two trainings. Therefore, it is chosen to reduce the change amount of the first regularization item weight and the second regularization item weight, so that the model can better approach the value within the expected sparse metric interval during training. On the other hand, if the model sparse metric obtained in the previous round of training and the model sparse metric of the current round are on the same side of the expected sparse metric interval, it means that the change amount of the above weights after the previous round of training is small, so the change amount can be appropriately increased so that the sparse metric of the model falls into the expected sparse metric interval as soon as possible.

[0033] In some embodiments of the present application, the step of obtaining the image recognition neural network model further includes: S302, after completing the training of the pre-trained image recognition neural network model, obtaining a trained image recognition neural network model; S304, trimming the zero-value parameters in the trained image recognition neural network model to obtain an image recognition neural network model.

[0034] By pruning the zero-valued parameters in the trained image recognition neural network model, the storage resources required to store the model can be reduced.

[0035] Specifically, in some embodiments of the present application, the step of obtaining the image recognition neural network model further includes: S306, after trimming the zero-value parameters in the trained image recognition neural network model, determining the image classification accuracy of the trimmed trained image recognition neural network model according to the first test image set; S308, determining a parameter sorting table for each neural network layer according to the trimmed neural network layer structure of the trained image recognition neural network model; wherein in each parameter sorting table, the parameters are sorted according to the absolute value; S310, when the image classification accuracy is greater than the upper limit of the accuracy threshold interval, the parameters with smaller absolute values ​​in each parameter sorting table are trimmed in equal proportion to obtain an image recognition neural network model after secondary trimming; wherein the trimming of zero-value parameters is included in the calculation of the equal proportion; S312, determining the classification accuracy of the secondary cropped image of the secondary cropped image recognition neural network model according to the second test image set; S314, when the image classification accuracy after secondary cropping is greater than the upper limit of the accuracy threshold interval, cyclically cropping the parameters with smaller absolute values ​​in each parameter sorting table in equal proportion, and determining the image classification accuracy after secondary cropping, until the image classification accuracy after secondary cropping falls within the accuracy threshold interval, so as to obtain a neural network model for image recognition after secondary cropping; S316. Determine the image recognition neural network model according to the image recognition neural network model after secondary trimming.

[0036] Based on the above-mentioned solution of trimming model parameters according to the parameter sorting table, the number of model parameters can be compressed as much as possible while ensuring that the accuracy of the model meets expectations, thereby reducing the storage space required to store the obtained image recognition neural network model.

[0037] It can be understood that the accuracy threshold interval in step S314 can be set according to the expected accuracy of the model. For example, if the expected accuracy of the model is 0.9, the accuracy threshold interval can be, for example, [0.9, 0.91].

[0038] Specifically, step S314 may include: When the image classification accuracy after secondary cropping is lower than the accuracy threshold range, the current round of secondary cropping is canceled, and secondary cropping is performed again based on the secondary cropped image recognition neural network model obtained by the previous round of secondary cropping; wherein the number of parameters for the re-secondary cropping is reduced.

[0039] In some embodiments of the present application, the step of obtaining the image recognition neural network model further includes: After obtaining the image recognition neural network model after secondary cropping, the image recognition neural network model after secondary cropping is trained according to the third test image set until the image recognition neural network model after secondary cropping meets the convergence conditions to obtain the image recognition neural network model; wherein the learning rate of training the image recognition neural network model after secondary cropping is lower than the learning rate of training the pre-trained image recognition neural network model.

[0040] The secondary cropped image recognition neural network model is trained based on the third test image set, and the model can be further fine-tuned under the existing accuracy of the secondary cropped image recognition neural network model, so that the accuracy of the model can be further improved.

[0041] As an example, the trimmed image recognition neural network model includes: convolutional layer, BN layer and fully connected layer; Step S308 may include: Determine the bias parameter sorting table of each convolutional layer; Determine the scaling parameter sorting table for each BN layer; Determine the fully connected layer parameter sorting table of the fully connected layer; When the image classification accuracy is greater than the upper limit of the accuracy threshold interval, the parameters with smaller absolute values ​​in the parameter sorting table are trimmed in equal proportion to obtain the image recognition neural network model after secondary trimming, including: Each bias parameter sorting table, each scaling parameter sorting table and the fully connected layer parameter sorting table are pruned in equal proportion to obtain a secondary pruned image recognition neural network model.

[0042] As mentioned above, for the BN layer, each scaling parameter corresponds to the input data of a channel and a number of front-layer neurons that generate the channel data. Based on this, in some embodiments of the present application, when trimming the scaling parameters in the scaling parameter sorting table, the offset parameters corresponding to the scaling parameters and the parameters of the front-layer neurons corresponding to the scaling parameters are trimmed.

[0043] The following is an exemplary description of the image recognition method provided in the embodiment of the present application in combination with the technical solutions provided in the above embodiments.

[0044] The image recognition device provided in the embodiment of the present application is, for example, a face recognition device.

[0045] First, based on the computing resources and storage resources of the face recognition device and the total number of parameters of the pre-trained convolutional neural network model (for example, 30 million), the expected parameter sparsity measure S=0.7 of the model adapted to the face recognition device is determined, and the expected parameter sparsity measure range is [0.699, 0.701].

[0046] Then, the facial images of the people to be identified are used as the training data set, and the Adam algorithm is used to iteratively train the parameters of the convolutional neural network model until the target classification loss function converges or the number of training rounds exceeds the preset number (for example, 200 rounds). The target classification loss function used in the training process is shown in formula (1): If the sparsity of the model parameters at the end of the current training round is greater than , then in the next round of training, , The value of increases by 1%; if the sparsity of the model parameters at the end of the current training round is less than , then in the next round of training, , The value of decreases by 1%.

[0047] For each convolutional layer of the trained convolutional neural network model, all bias parameters of the convolutional layer are sorted by absolute value, and a sorted list is generated for each convolutional layer. For the BN layer parameters of the trained convolutional neural network model, the BN layer parameters include scaling parameters and offset parameters. All scaling parameters of each BN layer are sorted by absolute value, and a sorted list is generated for each BN layer. For the fully connected layer parameters of the trained convolutional neural network model, they are sorted by absolute value, and a sorted list is generated for the fully connected layer.

[0048] Prune all zero-valued parameters of the trained convolutional neural network model.

[0049] Test the target classification accuracy of the trimmed model on the test dataset. If the accuracy is higher than the preset classification accuracy threshold , then the parameters in each of the aforementioned sorted lists are appropriately pruned according to their absolute values ​​from small to large, and the number of pruned parameters in different sorted lists (including the aforementioned pruned zero-value parameters) accounts for the same proportion of the total number of parameters in the sorted lists. Repeat the above method in this step until the target classification accuracy of the model is equal to the preset classification accuracy threshold .

[0050] The pruned model is fine-tuned on the training dataset with a smaller learning rate. The fine-tuned model is deployed on the face recognition device as the final face recognition model to perform specific face recognition tasks.

[0051] [Second embodiment] A second embodiment of the present application provides an image recognition method, which is applied to an image recognition device. The method may include the following steps: Acquire an image of a target area, the image including an object to be identified; Identify the object type of the object to be identified by using an image recognition neural network model; Output image recognition results according to the object type; The image recognition neural network model is obtained by a sparse and robust federated learning method, and the sparse and robust federated learning method is applied to a server, and the server communicates with a client. The sparse and robust federated learning method may include the following steps: Obtain available computing resources and available storage resources of the image recognition device; Based on available computing resources and available storage resources, determine the expected sparseness metric interval of the image recognition neural network model, where the sparseness metric is the ratio of non-zero parameters to total parameters in the image recognition neural network model; Initializing the parameters of the pre-trained image recognition neural network model according to the expected sparse measurement interval to obtain initialization parameters; Sending the initialization parameters to multiple target client nodes respectively, so that the client updates the local model according to the initialization parameters, the first regularization term and the second regularization term, and the local training image set of the client, thereby obtaining multiple local model update parameters; Receive multiple local model update parameters returned by the client. Each local model update parameter can be obtained by the following steps: Based on the local training image set of the client, determine the current sparseness metric of the local model after this round of training; When the current sparse metric is greater than the upper bound of the expected sparse metric, increase the weight of the first regular term and the second regular term of the loss function in the next round of training; when the current sparse metric is less than the lower bound of the expected sparse metric, reduce the weight of the first regular term and the second regular term of the loss function in the next round of training; When the local training rounds reach a preset number, the training of the local model is terminated to obtain the local model update parameters; Determining aggregate parameters of a pre-trained image recognition neural network model based on a plurality of local model update parameters; Determine whether the aggregation parameters of the pre-trained image recognition neural network model meet the preset termination conditions; When the aggregation parameters of the pre-trained image recognition neural network model meet the preset termination conditions, it is determined that the training of the pre-trained image recognition neural network model is completed to obtain the image recognition neural network model.

[0052] The image recognition method provided in the embodiment of the present application can apply the trained image recognition neural network model to the image recognition device to perform image recognition. The image recognition device may, for example, include an on-board processor, a built-in processor for electronic monitoring, etc. The computing resources of the above-mentioned processor are limited, and during the execution of intelligent driving and electronic monitoring functions, they will rely on the computing resources of the local processor. In the image recognition method provided in the embodiment of the present application, the expected sparse measurement interval of the image recognition neural network model is determined based on the computing resources and storage resources of the image recognition device, wherein, for example, the total number of parameters of the image recognition neural network model can be first determined based on the storage resources of the image device and the need for model establishment, and then the expected non-zero parameter amount is determined in combination with the computing resources of the image recognition device. The proportion of the total number of parameters, that is, the expected sparse measurement interval of the image recognition neural network is determined. Then, the pre-trained image recognition neural network model is trained, and the loss function of the model training has a first regularization term and a second regularization term. By adjusting the weights of the first regular term and the second regular term in the next round of training according to the size relationship between the current sparsity metric of the model after each round of training and the expected sparsity metric, the size of the current sparsity after the next round of training is adjusted, so that the sparse metric of the image recognition neural network model finally obtained meets the expected sparse metric value within the expected sparsity metric interval. Therefore, the image recognition neural network model finally obtained can run smoothly in the image recognition device without causing the hardware of the image recognition device to overload, thereby ensuring that the image recognition results will not be erroneous. Therefore, in application scenarios such as intelligent driving or electronic monitoring that require accurate image recognition results within a limited time, detection and recognition results can also be given in a timely manner while ensuring the expected recognition accuracy.

[0053] On the other hand, the image recognition method provided in the embodiment of the present application can also reduce the cost of image recognition equipment. For example, the total number of parameters and the expected sparse measurement interval of the image recognition neural network model are determined according to the expected model accuracy. Therefore, the image recognition neural network model obtained after training can reduce the proportion of non-zero parameters in the total number of parameters while meeting its model accuracy requirements, thereby reducing the amount of calculation required for the model operation, and thus reducing the processing performance requirements for the image recognition equipment, thereby providing a possibility for reducing the cost of the image recognition equipment.

[0054] In the embodiment of the present application, before model training, the server first initializes the parameters of the pre-trained image recognition neural network model to obtain the initialization parameters as For example, the pre-trained image recognition neural network model can be a deep neural network model. After obtaining the initialization parameters, the initialization parameters are sent to multiple target client nodes respectively. In one example, the client has nodes, Randomly select from client nodes The selected federated learning client nodes are ,in is the client node serial number, . And the federated learning client nodes are Send initialization parameters After receiving the initialization parameters, the client updates the local model according to the initialization parameters, the first regularization term, and the second regularization term, thereby obtaining multiple local model update parameters. The first regularization term may be, for example, a Log-modified regularization term of the L1 norm. For example, during the update, the pre-trained image recognition neural network model may be trained using a stochastic gradient descent algorithm using a local training image set of the client. In one example, in the first When the global model parameters are sent, the server node S Randomly select from client nodes client nodes, and pre-trained image recognition neural network model parameters Sent to the selected federated learning client node, starting the Round of model training and global model parameter update.

[0055] In an embodiment of the present application, the model on the client node is a local model, and the model on the server side is a global model. The client node submits multiple update parameters of local models to the server. The server node uses multiple parameters to aggregate and generate a global model, and then sends the aggregated updated global model parameters to the client node. The client node updates the local model with the newly received global model parameters, and repeats the cycle. After the client updates the local model and obtains multiple local model update parameters, the multiple local model update parameters are returned to the server. After receiving the multiple local model update parameters returned by the client, the server determines the aggregation parameters of the global model according to the multiple local model update parameters. For example, when determining the aggregation parameters, the number of samples and multiple local model update parameters contained in the local storage sample data sets of multiple client nodes can be obtained respectively. And the number of samples contained in the local storage sample data sets of each client node and the local model update parameters are multiplied to obtain multiple products. Finally, the multiple products are summed and then divided by the sum of the number of samples of multiple client nodes to obtain the aggregation parameters of the global model. Then determine whether the aggregation parameters meet the preset termination conditions. The preset termination condition refers to the condition for terminating the training of the global model set in advance. For example, the preset termination condition may be that the aggregate parameters of the global model converge, or the total rounds of training of the global model are greater than or equal to a threshold constant. When the aggregate parameters of the global model meet the preset termination condition, it is determined that the global model training is completed.

[0056] Through the above technical solution, the server initializes the parameters of the global model to obtain the initialization parameters. Then the initialization parameters are sent to multiple target client nodes respectively, so that the local model is updated by the client according to the initialization parameters, the first regularization term and the second regularization term, thereby obtaining multiple local model update parameters, and receiving multiple local model update parameters returned by the client. Then the aggregation parameters of the global model are determined according to the multiple local model update parameters. Finally, it is judged whether the aggregation parameters of the global model meet the preset termination conditions. When the aggregation parameters of the global model meet the preset termination conditions, it is determined that the global model training is completed, which can improve the sparsity of the model while improving the robustness of the model, and the regularization term weight can be automatically and dynamically adjusted as the number of training rounds increases, thereby enhancing the training effect of the model. Since the training process of the image recognition neural network model involves the reduction of non-zero parameters of the model, in order to better ensure the accuracy of the image recognition neural network model, the obtained image recognition neural network model can be integrated with the data knowledge of the training image sets of multiple clients through federated learning, thereby improving the generalization ability of the image recognition neural network model as a global model, and then helping to improve the accuracy of the image recognition neural network model in practical applications.

[0057] In the above training process, the first regularization term is used to perform sparse optimization on small-value parameters, and the second regularization term focuses on sparse optimization on large-value parameters. The combined use of the first regularization term and the second regularization term can improve the overall sparsification effect of the model. When the current sparse metric is greater than the upper bound of the expected sparse metric, it means that the current non-zero parameter amount of the model is large. By increasing the weight of the first regularization term and the weight of the second regularization term, the sparse effect of the first regularization term and the second regularization term can be increased, so that the amount of non-zero parameters tends to be reduced in the next round of training; conversely, when the current sparse metric is less than the lower bound of the expected sparse metric, it is necessary to reduce the sparse influence of the first regularization term and the second regularization term, so that the training of the model gradually approaches a sparse metric value within the expected sparse metric interval as the number of iterations increases.

[0058] It is understandable that the training of the image recognition neural network model can be performed on a computer with sufficient computing and storage resources, or on a distributed computer network architecture. The image recognition neural network model obtained after the training is completed can be imported into the image recognition device for application.

[0059] In the embodiment of the present application, the update parameters may satisfy formula (2): ; (2) in, is the client node serial number, is the number of model training rounds, For the Round, The updated parameters of the global model of the client nodes, For the Parameters of the wheel global model, is the local model parameter vector, is the learning rate constant, represents the novelty loss function exist Right gradient.

[0060] Specifically, the update parameters can satisfy the formula In the During the round of model training, each selected federated learning client node Using slave server nodes Received global model parameters Update the local model. For the Round, Update parameters of the global model for each client node. is the local model parameter vector, is the learning rate constant, represents the novelty loss function exist Right The new loss function refers to the function after adding the regularization term to the original loss function. By determining the update parameters of the global model, the aggregation parameters of the global model can be determined.

[0061] In the embodiment of the present application, the new loss function may satisfy the formula: ; (3) in, The new loss function after adding the regularization term to the original loss function of the global model, is the classification loss function, is the weight of the first regularization term, is the weight of the second regularization term, is the local model parameter vector, is the L1 norm of the parameter vector, is the square of the L2 norm of the parameter vector, and the square of the L2 norm is the second regular term.

[0062] In some embodiments of the present application, when the current sparse metric and the previous round sparse metric of the pre-trained image recognition neural network model in the previous round of training are respectively located on both sides of the sparse metric interval, the change in the first regularization term weight and the second regularization term weight is reduced; when the current sparse metric and the previous round sparse metric are located on the same side of the sparse metric interval, the change in the first regularization term weight and the second regularization term weight is increased.

[0063] During the training of the pre-trained image recognition neural network model, the size of the first regularization item weight and the second regularization item weight is directly related to the change amplitude of the model sparse metric during the training process. In the process of adjusting the above weights, it may happen that the model sparse metric obtained in the previous round of training and the model sparse metric of the current round are on both sides of the expected sparse metric interval due to the excessive weight adjustment amplitude. Therefore, it is explained that the adjustment amplitude of the above weights, that is, the change amount, is too large, causing the sparse metric of the model to directly cross the range of the expected sparse metric interval during the two trainings. Therefore, it is chosen to reduce the change amount of the first regularization item weight and the second regularization item weight, so that the model can better approach the value within the expected sparse metric interval during training. On the other hand, if the model sparse metric obtained in the previous round of training and the model sparse metric of the current round are on the same side of the expected sparse metric interval, it means that the change amount of the above weights after the previous round of training is small, so the change amount can be appropriately increased so that the sparse metric of the model falls into the expected sparse metric interval as soon as possible.

[0064] In an embodiment of the present application, determining whether the aggregation parameters of the global model meet a preset termination condition may include: Determine whether the aggregation parameters of the global model have converged; Alternatively, determine whether the total number of training rounds of the global model is greater than or equal to a threshold constant.

[0065] Specifically, the preset termination condition refers to a pre-set condition for terminating the training of the global model. The preset termination condition may be the convergence of the aggregation parameters of the global model, or the total number of training rounds of the global model being greater than or equal to a threshold constant. The threshold constant is a positive integer greater than 1. When the aggregation parameters of the global model meet the preset termination condition, it can be determined that the global model training is completed, thereby obtaining an image recognition neural network model.

[0066] In an embodiment of the present application, determining the aggregation parameter of the global model according to multiple local model update parameters may include: Respectively obtain the number of samples and multiple local model update parameters contained in the local storage sample data sets of multiple client nodes; Multiplying the number of samples contained in the locally stored sample data set of each client node and the local model update parameter to obtain a plurality of products; The multiple products are summed and divided by the sum of the number of samples of multiple client nodes to obtain the aggregate parameters of the global model.

[0067] Specifically, after the client updates the global model and obtains multiple local model update parameters, the multiple local model update parameters are returned to the server. After receiving the multiple local model update parameters returned by the client, the server determines the aggregation parameters of the global model based on the multiple local model update parameters. First, the server obtains the number of samples contained in the locally stored sample data sets of multiple client nodes and the multiple local model update parameters returned by the client. Then, the number of samples contained in the locally stored sample data sets of each client node and the local model update parameters are multiplied to obtain multiple products. Finally, the multiple products are summed and divided by the sum of the number of samples of multiple client nodes to obtain the aggregation parameters of the global model.

[0068] Specifically, the aggregation parameters can satisfy the formula . is the aggregation parameter of the global model. For The number of client nodes randomly selected from client nodes. For client nodes A locally stored sample dataset, For collection Contains the number of elements. Client nodes The number of elements in the locally stored sample dataset is multiplied by the update parameter to obtain Then multiply The products are summed and divided by the sum of the number of samples of all selected client nodes to obtain the aggregate parameters of the global model.

[0069] In the embodiment of the present application, the federated learning method may further include: When the aggregation parameters of the global model do not meet the preset termination conditions, the global model is retrained.

[0070] Specifically, when the aggregation parameters of the global model do not meet the preset termination conditions, multiple client nodes are randomly selected again, and initialization parameters are sent to the client nodes to perform model training until the aggregation parameters of the global model meet the preset termination conditions.

[0071] Through the above technical solution, the server initializes the parameters of the global model to obtain the initialization parameters. Then, the initialization parameters are sent to multiple target client nodes respectively, so that the local model is updated by the client according to the initialization parameters, the first regularization term and the second regularization term, thereby obtaining multiple local model update parameters, and receiving multiple local model update parameters returned by the client. Then, the aggregation parameters of the global model are determined based on the multiple local model update parameters. Finally, it is determined whether the aggregation parameters of the global model meet the preset termination conditions. When the aggregation parameters of the global model meet the preset termination conditions, it is determined that the global model training is completed.

[0072] After the trained global model is obtained, the trained image recognition neural network model provided in the first embodiment of the present application is obtained, and the global model can be trimmed with reference to the first embodiment of the present application. Thus, the number of parameters of the model can be compressed as much as possible while ensuring that the accuracy of the model meets expectations, thereby reducing the storage space required to store the obtained image recognition neural network model.

[0073] [Third embodiment] The present application also provides an image recognition device, which may include: a memory and a processor. The memory is configured to store instructions; and the processor is configured to call instructions from the memory and implement the image recognition method provided in any of the above embodiments when executing the instructions.

[0074] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the image recognition method provided by any of the above embodiments.

[0075] An embodiment of the present application also provides a computer program product, including a computer program, which implements the image recognition method provided according to any of the above embodiments when executed by a processor.

[0076] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0077] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0078] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0080] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0081] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0082] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0083] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0084] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An image recognition method, characterized in that: Applied to an image recognition device, the method comprises: Acquire an image of a target area, wherein the image includes an object to be identified; Identify the object type of the object to be identified by using an image recognition neural network model; Outputting an image recognition result according to the object type; The image recognition neural network model is obtained by the following steps: Acquiring available computing resources and available storage resources of the image recognition device; Determine an expected sparseness metric interval of the image recognition neural network model based on the available computing resources and the available storage resources, wherein the sparseness metric is a ratio of the amount of non-zero parameters to the total amount of parameters in the image recognition neural network model; Based on the training image set, determine the current sparsity measure of the pre-trained image recognition neural network model after the current round of training; When the current sparse metric is greater than the upper bound of the expected sparse metric, increase the first regularization term weight and the second regularization term weight of the loss function in the next round of training; When the current sparse metric is less than the lower bound of the expected sparse metric, reducing the first regularization term weight and the second regularization term weight of the loss function in the next round of training; When the loss function converges or the number of training rounds reaches a preset number, the training of the pre-trained image recognition neural network model is terminated to obtain the image recognition neural network model.

2. The method according to claim 1, characterized in that The first regularization term includes: The logarithm of the L1 norm plus a constant term, where the constant term is greater than or equal to 1.

3. The method according to claim 2, characterized in that The loss function includes: ； Among them, the is the classification loss function, is the weight of the first regularization term, is the weight of the second regularization term, is the parameter vector of the pre-trained image recognition neural network model, is the L1 norm of the parameter vector, is the square of the L2 norm of the parameter vector, and the square of the L2 norm is the second regularization term.

4. The method according to claim 1, characterized in that: When the current sparse metric and the last round sparse metric of the pre-trained image recognition neural network model in the last round of training are respectively located on both sides of the sparse metric interval, reducing the change amount of the first regularization term weight and the second regularization term weight; In a case where the current sparse metric and the previous round sparse metric are located on the same side of the sparse metric interval, the change amounts of the first regularization term weight and the second regularization term weight are increased.

5. The method according to claim 1, characterized in that The step of obtaining the image recognition neural network model also includes: After completing the training of the pre-trained image recognition neural network model, obtaining a trained image recognition neural network model; The zero-value parameters in the trained image recognition neural network model are trimmed to obtain the image recognition neural network model.

6. The method according to claim 5, characterized in that The step of obtaining the image recognition neural network model also includes: After trimming the zero-value parameters in the trained image recognition neural network model, determining the image classification accuracy of the trimmed trained image recognition neural network model according to the first test image set; Determine a parameter sorting table for each neural network layer according to the trimmed neural network layer structure of the trained image recognition neural network model; wherein, in each parameter sorting table, the parameters are sorted according to the absolute value; When the image classification accuracy is greater than the upper limit of the accuracy threshold interval, the parameters with smaller absolute values ​​in each parameter sorting table are trimmed in equal proportion to obtain an image recognition neural network model after secondary trimming; wherein the trimming of the zero-value parameters is included in the calculation of the equal proportion; Determining the secondary cropped image classification accuracy of the secondary cropped image recognition neural network model according to the second test image set; When the classification accuracy of the image after secondary cropping is greater than the upper limit of the accuracy threshold interval, the parameters with smaller absolute values ​​in each parameter sorting table are cyclically cropped in equal proportion, and the classification accuracy of the image after secondary cropping is determined until the classification accuracy of the image after secondary cropping falls within the accuracy threshold interval, so as to obtain a neural network model for image recognition after secondary cropping; The image recognition neural network model is determined according to the image recognition neural network model after secondary trimming.

7. The method according to claim 6, characterized in that The step of obtaining the image recognition neural network model also includes: After obtaining the secondary cropped image recognition neural network model, the secondary cropped image recognition neural network model is trained according to a third test image set until the secondary cropped image recognition neural network model meets a convergence condition to obtain the image recognition neural network model; wherein the learning rate of training the secondary cropped image recognition neural network model is lower than the learning rate of training the pre-trained image recognition neural network model.

8. The method according to claim 6, characterized in that The trimmed image recognition neural network model includes: a convolutional layer, a BN layer and a fully connected layer; Determining a parameter sorting table of each neural network layer according to the neural network layer structure includes: Determine a bias parameter sorting table for each of the convolutional layers; Determine a scaling parameter ranking table for each of the BN layers; Determine a fully connected layer parameter sorting table of the fully connected layer; When the image classification accuracy is greater than the upper limit of the accuracy threshold interval, the parameters with smaller absolute values ​​in each parameter sorting table are trimmed in equal proportion to obtain a secondary trimmed image recognition neural network model, including: Each of the bias parameter sorting tables, each of the scaling parameter sorting tables, and the fully connected layer parameter sorting table is pruned in equal proportion to obtain the image recognition neural network model after secondary pruning.

9. The method according to claim 8, characterized in that: Each of the scaling parameters of the BN layer corresponds to the input data of a channel and a number of front-layer neurons that generate the channel data; In the case of trimming the scaling parameters in the scaling parameter sorting table, the offset parameters corresponding to the scaling parameters and the parameters of the front layer neurons corresponding to the scaling parameters are trimmed.

10. An image recognition device, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instructions from the memory and implement the image recognition method according to any one of claims 1 to 9 when executing the instructions.

11. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions, which are used to enable a machine to execute the image recognition method according to any one of claims 1 to 9.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the image recognition method according to any one of claims 1 to 9.

Images